Saturday, 28 March 2015

Practical work with Powershell

One of the things I sometimes need to do is to be able to generate sets of reports with different parameters. As some of these reports can take a while to run this is an ideal opportunity to script the task and as I’m currently spending more time in PowerShell it seemed like the obvious tool.

For working with Excel I’ve been using ClosedXML although in the future I’m going to investigate EPPlus as there are claims that it’s better for larger files.

So the first thing to do is to demonstrate how easy it is to use ClosedXML with PowerShell to write to a file

Then it’s a case of getting the data out of the database and into a DataTable

One of the things I like about ClosedXML is the ability to pass in a datatable as an argument to the new sheet method and have it do the work automatically.

This results in the final script which can be used as base for further customisation

Sunday, 17 March 2013

Web.py, Lighttpd and FastCGI

 

Following the instructions from http://webpy.org/install and http://webpy.org/docs/0.3/tutorial I had setup Lighttpd.

However, when running the demo code the following error was received

 

2013-03-17 23:04:19: (mod_fastcgi.c.1490) released proc: pid: 8806 socket: unix:/var/tmp/lighttpd/fastcgi.socket-0 load: 0
2013-03-17 23:14:50: (mod_fastcgi.c.1732) connect failed: Connection refused on unix:/var/tmp/lighttpd/fastcgi.socket-0
2013-03-17 23:14:50: (mod_fastcgi.c.454) backend disabled for 1 seconds
2013-03-17 23:14:50: (mod_fastcgi.c.3002) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1

Checking the script showed that it was working correctly and no useful information was appearing in error.log

The following config option was then added to lighttpd.conf

server.breakagelog          = "/var/log/lighttpd/breakage.log"

This gives the much more useful

Traceback (most recent call last):
  File "/var/www/code.py", line 16, in <module>
    app.run()
  File "/usr/local/lib/python2.7/dist-packages/web.py-0.37-py2.7.egg/web/application.py", line 313, in run
    return wsgi.runwsgi(self.wsgifunc(*middleware))
  File "/usr/local/lib/python2.7/dist-packages/web.py-0.37-py2.7.egg/web/wsgi.py", line 35, in runwsgi
    return runfcgi(func, None)
  File "/usr/local/lib/python2.7/dist-packages/web.py-0.37-py2.7.egg/web/wsgi.py", line 16, in runfcgi
    import flup.server.fcgi as flups
ImportError: No module named flup.server.fcgi

This was unexpected as I had already used easy_install to install flup, but upon checking could find no evidence of it being installed

The latest egg was downloaded from https://pypi.python.org/pypi/flup and installed using easy_install.

The installation process was then checked by making sure flup could be imported.

Friday, 8 February 2013

Windows Event Log Permisions & SQL Server Command Proxies

The following problem outlined in this entry is a relatively common situation but when searching for a solution there was not a single definitive guide for what I needed. This post is based on information in the following articles.
Jag Padda's Blog - Event Log Permissions
Washington University - Understanding SDDL Syntax
Microsoft – KB323076
Window Security – WEVTUTIL
Database Journal - Proxy Accounts in SQL Server
The following was written and tested against SQL Server 2008R2 and Windows Server 2008R2.
Background
You are in the envious position of being both the DBA and Windows administrator for a company and are given the task of integrating a third party application into your estate. The application consists of a database, a middle tier component running as a service and a client which runs on the user computers. In addition to this the application has multiple utility programs responsible for importing/exporting data, initiating summarisation routines.
This means in addition to the core service there are SQL Server Agent jobs calling out to a variety of programs, batch files and scripts. These additional programs either access the database through the middle tier or connect directly to the database, where they connect directly to the database the connection string is managed with a UDL file. In an effort to improve security windows authentication will be used rather than storing credentials in a text file.
Configuring Proxies.
For programs called by SQL Server agent we are going to use a proxy account to limit the permissions needed (SQL Server Agent is already running under a separate Managed Service Account).
First create the cmdshell proxy account which is needed to allow a non system admin account to run xp_cmdshell
EXEC sp_xp_cmdshell_proxy_account 'Domain\User', 'PASSWORD;
This results in the creation of the credential ##xp_cmdshell_proxy_account## which can now be assigned to the Operating System (CmdExec) Proxy
EXEC msdb.dbo.sp_add_proxy @proxy_name=N'CommandShell',@credential_name=N'##xp_cmdshell_proxy_account##', @enabled=1
EXEC msdb.dbo.sp_grant_proxy_to_subsystem @proxy_name=N'CommandShell', @subsystem_id=3 

EXEC msdb.dbo.sp_grant_login_to_proxy @proxy_name=N'CommandShell', @login_name=N'DOMAIN\USER'



In this case I have named the proxy CommandShell so when calling sp_add_jobstep the parameter @proxy_name would be CommandShell
The user account used for the proxy was then removed from the users group in active directory and given access only to the directories necessary for implementation.
Finally the permissions at the database level were set to deny everything apart from those functions which were necessary.
Configuring Event Log
Some of the external programs being called through the Command Shell Proxy were attempting to write to the Application event log by calling EventLog.WriteEntry which was failing. Clearly granting many additional permissions to the proxy account would be defeating the purpose of the earlier security work so it was necessary to only allow the Application event log to be accessible.
Firstly identify the SID of the user account by using powershell

Import-Module ActiveDirectory
$user= New-Object System.Security.Principal.NTAccount("DOMAIN_NAME", "USER_NAME")
$sid= $user.Translate([System.Security.Principal.SecurityIdentifier])
$sid.Value


This will give you the security ID of the domain user (in this case S-1-5-21-96566445-2708948433-2759571698-1113)

The next step is to use wevtutil (Windows Event Utility) to get the ChannelAccess (CA) value for the Application log.

wevtutil gl Application

This gives the output

name: Application
enabled: true
type: Admin
owningPublisher:
isolation: Application
channelAccess: O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x7;;;SO)(A;;0x3;;;IU)(A;;0x3;;;SU)(A;;0x3;;;S-1-5-3)(A;;0x3;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573)
logging:
  logFileName: %SystemRoot%\System32\Winevt\Logs\Application.evtx
  retention: false
  autoBackup: false
  maxSize: 20971520
publishing:
  fileMax: 1


The important bit is the channel access value O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x7;;;SO)(A;;0x3;;;IU)(A;;0x3;;;SU)(A;;0x3;;;S-1-5-3)(A;;0x3;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573)

TokenMeaning
O:BAOwner: Built-in administrators
G:SYGroup: Local system
(A;;0x3;;;IU)Access Allowed, Read/Write for Interactive Users


In this case we want to add an extra block for our account

(A;;0x3;;;S-1-5-21-96566445-2708948433-2759571698-1113)

This will give read/write permission for the SID

To apply this you would execute

wevtutil sl Application /ca:O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x7;;;SO)(A;;0x3;;;IU)(A;;0x3;;;SU)(A;;0x3;;;S-1-5-3)(A;;0x3;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573)(A;;0x3;;;S-1-5-21-96566445-2708948433-2759571698-1113)

If you wanted to simplify it and make it for all Authorised Users it would be

wevtutil sl Application /ca:O:BAG:SYD:(A;;0xf0007;;;SY)(A;;0x7;;;BA)(A;;0x7;;;SO)(A;;0x3;;;IU)(A;;0x3;;;SU)(A;;0x3;;;S-1-5-3)(A;;0x3;;;S-1-5-33)(A;;0x1;;;S-1-5-32-573)(A;;0x3;;;AU)

Now under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\eventlog\Application you have the new key CustomerSD

image

Monday, 21 May 2012

Practical Unicode–Part 4

This post is how to automate the conversion done in part 3 and some of the things which caused problems.
Starting with the file in UTF-8 without a Byte Order Marker (BOM)
image
Using ICONV to covert the file from UTF-8 to UCS2 Little Endian with the command below results in the following useless file.
iconv -f UTF-8 -t UCS-2LE input.txt > output.txt
image
As you can see every character is represented by two bytes but without the BOM 0xFF 0xFE notepad++ tries to display it as ANSI and SQL Server gives the following error on bulk insert
Msg 4832, Level 16, State 1, Line 1
Bulk load: An unexpected end of file was encountered in the data file.

If we insert a BOM we then get the following giberish

image
This is caused by the CRLF not being handled correctly.
image
The CRLF is represented as 0x0D 0x0A 0x00 and this means notepad++ can’t interpret it properly.
If we change these bytes to 0x0D 0x00 0x0A 0x00 then the file displays correctly and also loads into SQL server correctly.
image
This can be done automatically using binmay and the following command
binmay -i input.txt -o output.txt -s "0D 0A 00" -r "0D 00 0A 00"
It is worth noting at this point there is not a BOM so the file displays as UCS-2 LE w/o BOM
image
You can then add the BOM using a program called FFFE_ADD
FFFE_add output.txt
This program is based on UTF-BOM-UTILS which is published under a BSD Licence, FFFE_ADD can be found at my BitBucket Repo
image

Monday, 14 May 2012

Practical Unicode Part 3

This post is dedicated to the following problem.
Data was being received from a system as UTF-8 without a Byte Order Marker (BOM) and contained accented characters, this data had to be loaded into SQL Server via BULK INSERT without any data corruption.
As I’m not planning on demonstrating this using customer data I’m going to set up the following example.
Create a table with three fields (as I want to show the column separator) with all three fields being NVARCHAR.
create table Test
      (FieldOne nvarchar(50)
      ,FieldTwo nvarchar(50)
      ,FieldThree nvarchar(50))

Then create a datafile which is tab delimited and saved as UTF-8 without a BOM

Symbols have been turned on to make it easier to read
image
Then bulk load the data
bulk insert Test From 'C:\Scratch\input.ansi.utf8.txt'
      with (codepage = 'ACP', FieldTerminator = '\t', RowTerminator='\n')

It can be seen in the screenshot below that the data was corrupted.

image
If the file is converted to UCS2 Little Endian (using Notepad++) and the same load process executed we see the following
1) We should really specify the datatype to remove the following message
Bulk load: DataFileType was incorrectly specified as char. DataFileType will be assumed to be widechar because the data file has a Unicode signature.

2) The data has loaded correctly.

image
The next post will be how to automate this process and some of the pitfalls.

Monday, 7 May 2012

Practical Unicode–Part 2

This is a follow on to Practical Unicode Part 1 and covers what Notepad++ means when it describes a file.
The hex editor used in this is HXD.

Ansi as UTF-8

This means that the file is UTF-8 encoded without a Byte Order Marker (BOM)
image
image

UTF-8

In this example I’ve added an extended character to show how ‘normal chars’ are a single byte while others are multiple bytes.
image
Here you can see the BOM 0xEF 0xBB 0xBF, then the bulk of the text being stored as a single byte and finally the final character being stored as three bytes 0xC2 0x81 0x42
image

UCS2 Little Endian

image
In this example you can see the following
  • The endianess of the word represented by the BOM 0xFF 0xFE.
  • Characters being stored as two bytes (16 bit word) e.g. U is stored as 0x55 0x00
  • The extended character at the end being stored as two words ( 8 bytes) 0x81 0x00 0x42 0x00
image
If we delete the BOM then you can see Notepad++ displays the encoding as UCS2 Little Endian without BOM
image
image

Monday, 30 April 2012

Practical Unicode–Part 1

This post is not intended to be an exhaustive essay on unicode, the standards and how it works; this is the result of working with multiple systems transferring data in various formats and trying to get it into SQL Server without corruption.
Environment Background
As previously mentioned this is based on my experience working with multiple source systems and the following tools in a Windows environment.
Overview of Unicode and Why We Have It
When we first started storing text (ignoring EBCDIC) the general convention was one byte (8 bits) represented a single character and the alphabet and common symbols were mapped to values 32 to 127 (see ASCII table), the values 128 to 255 were considered to be the extended character set. What this meant was that to represent different alphabets and common usages of accented characters we ended up with code pages, this meant that value x could represent character y in one codepage but character z in another, this was a pain with websites as unless the code page was declared you could not be confident that you were displaying the characters correctly.
As a result we now have unicode where it is possible for a character to be represented by multiple bytes (simplification: glyphs and combined characters not covered). As with all things standards based unicode isn’t just a monolithic entity, there are various flavours and things to note.
Common ‘Flavours’
UTF-8
This is pretty much the defacto standard for storing text in a manner which can be read by almost everyone. Where a character can be represented by ASCII it is held as a single byte, other characters can be 2, 3 or 4 bytes.
While there is only one correct way to read the bytes and thus no need for a Byte Order Marker (BOM) is is common to find the bytes 0xEF 0xBB 0xBF at the beginning of some files. This can be removed using the following binmay command
binmay –s “EF BB BF” –r “” 
UCS2 Little Endian
This is the only format SQL Server can use for input via BCP or Bulk Insert, any data exported using BCP where the are NVARCHAR fields will be in this format.
Like UTF-16 it stores a character as either one or two 16 bit words.
A UCS2 Little Endian file with have a two byte Byte Order Marker consisting of 0xFF 0xFE, if it were a UCS2 Big Endian file then the BOM would be 0xFE 0xFF
Both UTF-16 and UCS2 are variable length/space encodings.
UTF-32
In UTF-32 every character is stored as two 16 bit words, this means that it is not space efficient but is very simple to parse. At the moment very few systems support it and with Python you have to choose the option at compile time.