This leads on from the previous blog entry on probabilistic models. The platform I am working on incorporates catastrophe models that produce Event Loss Tables or ELTs. These are tables containing Events, Ground Up Loss, Ground Up Loss standard deviation, loss frequency and various other loss perspectives. These loss tables are sorted in order of increasing ground up loss giving a cumulative loss distribution. A function that fits this curve is called a cumulative distribution function or cdf. Pricing a Cat XL treaty is done by integrating the CDF. This can be made in a number of ways:
The simplest way is by burning cost. This means just summing up the events taht trigger the treaty. This works very well in low layers because there are naturally more low cost events than high cost events. For high layers there may be only a few events and the accuracy is not sufficient. In these cases we can try to fit a points to a function such as Paratoo or log polynomial and then integrate the resulting function over the Cat XL layer. There are a number of problems with this approach, for example the numerical regression may give unreasonable coefficients that give unexpected results. The method that we use the most is to to sampling each loss assuming a distribution like log normal and the standard deviation and applying a burning cost.
Event loss tables can become quite large but are still manageable compared to Year Loss Tables. For example it is possible to make a loss estimation of a current event by making a query based on event or combination of events in the event loss table. It is not possible to combine different ELTs of different event sets
Some cat models incorporate some information over when an event occurred. This makes it possible to model clustering of events much more accurately. This has quite an impact on pricing. For example inuring treaties can be priced much more accurately. Imagine with an ELT you need to assume an average number of events in a year. There is a big difference between having an average of 2 events per year and a year loss table that says on the first year there are 0 events on the second year there are 4 events. Both have a mean of 2 events but in the first year the treaty is not touched and in the second year it can depend on the number of reinstatements.
Year loss tables are event based which makes them easy to combine with other YLTs.
Here are the steps needed to create a YLT from an ELT
1. Decide on how long the YLT is to be based
2. Determine an average number of events that happen in a year
3. Use a Poison distribution with a random sample to determine how many events happen in a particular year
(The poison distribution provides a sequence of integer numbers )
4. Sample from the ELT randomly these events
5. Based on the Mean Loss and STD Loss make a sample with a distribution such as a Beta or log normal to determine an actual loss using another random variable
The random variables are created from seeds. For a given seed the same sequence of random numbers are generated each time the application is run. Mathematically it turns out that this is an acceptable way of sampling and it means that the resulting YLTs are not needed to be stored. This is advantageous because in order to get events in the high layers a long simulation period is needed which means the storage requirements of a YLT is much bigger than an ELT.
Now let's drill into the details. Step 3 sounds very easy, make a sample random sample of a Poisson distribution. The thing is that Poisson is expressed in a formula like frequency = Function( Mean, Standard Deviation). For example in Excel it looks like POISSON(A3,$B$1,TRUE). The problem is that we need to make random samples of the number of events with a given mean and sd. To do this we need to use something called Inversion theory. There are some statistical packages that have this function built in. For example in r it looks like
pois <- rpois(100, lambda=3)
pois[1:10]
[1] 1 2 3 2 2 2 3 3 6 2
In C# you need to find a library to do this. I found 2 that look interesting open source projects that have these functions
http://code.google.com/p/pzmath/source/checkout
http://www.alglib.net/
Friday, 27 January 2012
Monday, 16 January 2012
Introducing probablistic models
Recently I have been revisiting some mathematics. Often maths becomes very complex very quickly so here I enjoy keeping it simple.
Probabilistic models consist of 3 parts
1. A model that produces a random variable(s) X
2. A model that creates probabilities for the random variables
3. The sum of frequencies must add up to 1
If we take the example of flipping a coin: The coin has 2 faces and these correspond to the random variable X. The values of X can be give a value such as X=1 for heads and X=2 for tails. The model would be the probability of X=1 is 0.5 and the probability of X=2 is 0.5 and the sum of these probabilities sum to 1. Then you have a coin that may not be completely round or evenly balanced meaning it may behave differently to the model.
There are other types of models for example the number of people in a queue can be modeled with Poison. Mr Poison was a lawyer that studied the lengths of queues Research on the "Probability of Judgments in Criminal and Civil Matters" and build a model around this. This is a very interesting model when modeling the length of queues, It's also very useful when converting an Event loss table to a Year loss table. Here the problem is to sample the number of events that happen in a year given a mean and a standard deviation
Catastrophe probabilistic models can also be broken down into these 3 components.
1. A modeling that combines the Hazard, Vulnerability and Exposure to produce the random variable ground up loss
2. A model that assigns a frequency for each event
3. An event catalogue where the sum of frequencies adds up to the total length of time the catalogue is constructed for
A catastrophe model is more elaborate than the coin example and comprises of the Hazard (the earthquake or the windstorm), the vulnerability (how fragile it is) and en exposure (what buildings are at risk)
Probabilistic models consist of 3 parts
1. A model that produces a random variable(s) X
2. A model that creates probabilities for the random variables
3. The sum of frequencies must add up to 1
If we take the example of flipping a coin: The coin has 2 faces and these correspond to the random variable X. The values of X can be give a value such as X=1 for heads and X=2 for tails. The model would be the probability of X=1 is 0.5 and the probability of X=2 is 0.5 and the sum of these probabilities sum to 1. Then you have a coin that may not be completely round or evenly balanced meaning it may behave differently to the model.
There are other types of models for example the number of people in a queue can be modeled with Poison. Mr Poison was a lawyer that studied the lengths of queues Research on the "Probability of Judgments in Criminal and Civil Matters" and build a model around this. This is a very interesting model when modeling the length of queues, It's also very useful when converting an Event loss table to a Year loss table. Here the problem is to sample the number of events that happen in a year given a mean and a standard deviation
Catastrophe probabilistic models can also be broken down into these 3 components.
1. A modeling that combines the Hazard, Vulnerability and Exposure to produce the random variable ground up loss
2. A model that assigns a frequency for each event
3. An event catalogue where the sum of frequencies adds up to the total length of time the catalogue is constructed for
A catastrophe model is more elaborate than the coin example and comprises of the Hazard (the earthquake or the windstorm), the vulnerability (how fragile it is) and en exposure (what buildings are at risk)
Thursday, 5 January 2012
Experimenting with Powershell
Command line scripts have been around for a long time. Power shell adds a powerful fluent syntax that enables a richer and more expressive way of coding our batch files. Powershell comes with Windows 7 and above as well as Windows Server 2008 R2 and above
Here is how to start using it... The first time you start PowerShell (Under Administrative tools I selected Windows PowerShell Modules) you will get the following warning
To fix this do the following steps
1. Under Administrative tools select Server Manager
2. In the tree view select Server Manager/Features
3. Add Feature
4. Select Windows PowerShell
5. Next, Install
6. under Accessories in a directory caller PowerShell
PoweShell basics
Powershell respects admin rights. Certain commandlet information require elevated privileges. Meaning that if you need access to information that requires elevated privileges you need to start powershell as administrator.
Here is a simple example how powershell can be used. Start notepad then execute
get -process notepad
Stop-Process
This gets a list of processes named “notepad” and pipes them to a Stop-Process commandlet which has the effect of closing the notepad that you just opened.
There are other ways of getting to the same information, for example
The output from a commandlet can be formatted in various ways, here are some examples:
get-process notepad
Format-List *
get-process
Format-Table name, starttime
get-process
where-object { $_.starttime }
Format-Table name, starttime
For WMI calls the command “get-wmiobject win32_service” does not give much detail. But the following provides plenty of detail:
get-wmiobject win32_service
Format-list *
Here are some examples of how to start applications
Start-process -FilePath notepad
Start-process -FilePath notepad -WindowStyle minimized
Here are some examples with dir.
Give me all files containing the string ground (Very usefull since in Windows 7 the text search function is missing)
dir *.* -recurs
select-string "ground"
Format-List FileName
Here’s a more elaborate version looking for files with “pro” in the name
dir *.* -recurs
Where-Object {$_.Name -like "*pro*"}
select-string "ground"
Format-List FileName
dir *.* -recurs
Where-Object {$_.Name -like "*"}
select-string "ground"
Format-List FileName
Get-Unique
Filename : myfile.vcxproj
dir *.* -recurs
Where-Object {$_.Name -like "*vcx*"}
select-string "ground"
Format-List LineNumber, Path
Get-Unique
LineNumber : 64
Path : D:\path\myfile.vcxproj
Here’s an example of how to replace a string in a file
(Get-Content .\File.TXT)
ForEach-Object {$_ -REPLACE "'", """" }
Set-Content TEST2.TXT
Here are some examples of how to read parts of text files
Get-Content ".\file_test.txt"
select -First 10
Get-Content ".\file_test.txt"
select -Last 10
Get-Content ".\file_test.txt"
select -First 3
select -Last 1
Get-Content ".\file_test.txt"
select -Skip 10
Power Shell with SQL Server
I found the following link that describes how to get the SQL features of powershell working
http://msdn.microsoft.com/en-us/library/cc281962.aspx
Basically you need to run the following script to enable sql commands to work
After executing this scripts you can run commands like:
Invoke-Sqlcmd -Query "SELECT @@VERSION;"
Column1
-------
Microsoft SQL Server 2008 (RTM) - 10.0.1600.22 (Intel X86) ...
Here’s a more entertaining script that executes an arbitrary command on a list of SQL servers
Here’s a URL that describes how to make your own powershell commandlets
http://www.gangleri.net/2009/04/21/BuildingPowerShellCmdletsWithVisualStudio2008.aspx
Building PowerShell Cmdlets with Visual Studio 2008
1.Download Windows PowerShell Cmdlet templates from CodePlex
2.Run the VSI installer to install the project and item templates
3.Open Visual Studio 2008 and select File->New->Project
4.From the New Project dialog select the ‘PowerShellCmdlet’ template
5.Now right click on the project and select ‘Add New Item’. This will show the ‘Add New Item’ dialog. You can see that there are class templates for PSCmdlets, Cmdlets, SnapIns and XML helper files. For now select the Cmdlet template.
6.The ProcessRecord method performs the actual processing for the Cmdlet, in this example we will just call the WriteObject method to display the ‘Hello World’ message.
using System.Management.Automation;
namespace HelloPowerShell
{
[Cmdlet(VerbsCommon.Get, "HelloCmdlet")]
public class HelloCmdlet : Cmdlet
{
protected override void ProcessRecord()
{
WriteObject("Hello World!");
}
}
}
7.Again right click on the project and select add new item, this time select the ‘PowerShellCmdlet Help XML’. The template will automatically prefix ‘.dll-help.xml’ therefore you should just type ‘Get-Hello’ as the name.
8.Now add another new item and select the ‘PowerShellCmdlet SnapIn’ template. Call this ‘HelloSnapIn.cs’
9.By default the SnapIn template fills out some sample information using MyCmdlet as the name simply change this to refer to Get-HelloCmdlet.
10.You can build the project and get a dll
11.Now start Windows PowerShell and run with administrator privileges as you will be installing a Cmdlet.
12.If you are using a 64-bit version of windows issue the command:
Set-Alias installutil C:\Windows\Microsoft.NET\Framework64\v2.0.50727\InstallUtil.exe
If you have a 32-bit version of windows issues the command:
Set-Alias installutil C:\Windows\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe
13.Now install your Cmdlet with the command:
installutil HelloPowerShell.dll
14.You can verify that the Cmdlet has been installed with the command:
Get-PSSnapin -Registered
15.Add the snap-in to your shell with:
Add-PSSnapin HelloCmdlet
16.You can run the Cmdlet with the command Get-HelloCmdlet and you should see the message “Hello World!”
By the way in VS2010 the PowerShell templates are there by default
Here is how to start using it... The first time you start PowerShell (Under Administrative tools I selected Windows PowerShell Modules) you will get the following warning
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. All rights reserved.
WARNING: File C:\Windows\system32\WindowsPowerShell\v1.0\Modules\PSDiagnostics\PSDiagnostics.psm1 cannot be loaded
because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details.
PS C:\Users\nfindlater>
To fix this do the following steps
1. Under Administrative tools select Server Manager
2. In the tree view select Server Manager/Features
3. Add Feature
4. Select Windows PowerShell
5. Next, Install
6. under Accessories in a directory caller PowerShell
PoweShell basics
Powershell respects admin rights. Certain commandlet information require elevated privileges. Meaning that if you need access to information that requires elevated privileges you need to start powershell as administrator.
Here is a simple example how powershell can be used. Start notepad then execute
get -process notepad
Stop-Process
This gets a list of processes named “notepad” and pipes them to a Stop-Process commandlet which has the effect of closing the notepad that you just opened.
There are other ways of getting to the same information, for example
PS C:\Users\nfindlater> get-process | where { $_.ProcessName -eq 'notepad' }
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
47 2 1108 3732 61 0.20 3112 notepad
Here’s an example using the event log
PS C:\Users\nfindlater> get-eventlog -LogName application -Newest 3
Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
77896 Oct 04 15:14 Information Microsoft-Windows... 1 The description for Event
77895 Oct 04 15:14 Information Desktop Window Ma... 1073750827 The Desktop Window Manager
77894 Oct 04 15:14 Information Winlogon 1073745925 Windows license validated.The output from a commandlet can be formatted in various ways, here are some examples:
get-process notepad
Format-List *
get-process
Format-Table name, starttime
get-process
where-object { $_.starttime }
Format-Table name, starttime
For WMI calls the command “get-wmiobject win32_service” does not give much detail. But the following provides plenty of detail:
get-wmiobject win32_service
Format-list *
Here are some examples of how to start applications
Start-process -FilePath notepad
Start-process -FilePath notepad -WindowStyle minimized
PS C:\Users\nfindlater> Get-EventLog application | where-object { $_.message -match 'Desk' }
Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
77895 Oct 04 15:14 Information Desktop Window Ma... 1073750827 The Desktop Window Manager was unable to start ...
This is faster
C:\Users\nfindlater> Get-EventLog application -InstanceId 1001
PS C:\Users\nfindlater> Get-EventLog application -newest 1
Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
77897 Oct 04 16:40 Information MSSQLSERVER 1073760088 Database backed up. Database: ICW_20081001_tRI_...
Here are some more examples using the eventlog
PS C:\Users\nfindlater> Get-EventLog application -newest 1
format-list *
EventID : 18264
MachineName : CHBByyyy.zzz.gggg.net
Data : {88, 71, 0, 0...}
Index : 77897
Category : Backup
etc
Here are some examples with dir.
Give me all files containing the string ground (Very usefull since in Windows 7 the text search function is missing)
dir *.* -recurs
select-string "ground"
Format-List FileName
Here’s a more elaborate version looking for files with “pro” in the name
dir *.* -recurs
Where-Object {$_.Name -like "*pro*"}
select-string "ground"
Format-List FileName
dir *.* -recurs
Where-Object {$_.Name -like "*"}
select-string "ground"
Format-List FileName
Get-Unique
Filename : myfile.vcxproj
dir *.* -recurs
Where-Object {$_.Name -like "*vcx*"}
select-string "ground"
Format-List LineNumber, Path
Get-Unique
LineNumber : 64
Path : D:\path\myfile.vcxproj
Here’s an example of how to replace a string in a file
(Get-Content .\File.TXT)
ForEach-Object {$_ -REPLACE "'", """" }
Set-Content TEST2.TXT
Here are some examples of how to read parts of text files
Get-Content ".\file_test.txt"
select -First 10
Get-Content ".\file_test.txt"
select -Last 10
Get-Content ".\file_test.txt"
select -First 3
select -Last 1
Get-Content ".\file_test.txt"
select -Skip 10
Power Shell with SQL Server
I found the following link that describes how to get the SQL features of powershell working
http://msdn.microsoft.com/en-us/library/cc281962.aspx
Basically you need to run the following script to enable sql commands to work
#
# Add the SQL Server Provider.
#
$ErrorActionPreference = "Stop"
$sqlpsreg="HKLM:\SOFTWARE\Microsoft\PowerShell\1\ShellIds\Microsoft.SqlServer.Management.PowerShell.sqlps"
if (Get-ChildItem $sqlpsreg -ErrorAction "SilentlyContinue")
{
throw "SQL Server Provider for Windows PowerShell is not installed."
}
else
{
$item = Get-ItemProperty $sqlpsreg
$sqlpsPath = [System.IO.Path]::GetDirectoryName($item.Path)
}
#
# Set mandatory variables for the SQL Server provider
#
Set-Variable -scope Global -name SqlServerMaximumChildItems -Value 0
Set-Variable -scope Global -name SqlServerConnectionTimeout -Value 30
Set-Variable -scope Global -name SqlServerIncludeSystemObjects -Value $false
Set-Variable -scope Global -name SqlServerMaximumTabCompletion -Value 1000
#
# Load the snapins, type data, format data
#
Push-Location
cd $sqlpsPath
Add-PSSnapin SqlServerCmdletSnapin100
Add-PSSnapin SqlServerProviderSnapin100
Update-TypeData -PrependPath SQLProvider.Types.ps1xml
update-FormatData -prependpath SQLProvider.Format.ps1xml
Pop-LocationAfter executing this scripts you can run commands like:
Invoke-Sqlcmd -Query "SELECT @@VERSION;"
Column1
-------
Microsoft SQL Server 2008 (RTM) - 10.0.1600.22 (Intel X86) ...
Here’s a more entertaining script that executes an arbitrary command on a list of SQL servers
$servers=get-content "D:\Nigel\PowerShell\SqlServers.txt"
foreach($server in $servers)
{
# go to each server and execute the command in D:\Nigel\PowerShell\SQLCommand.txt
# Invoke-sqlcmd -ServerInstance $server -Database master -InputFile “D:\Nigel\PowerShell\SQLCommand.txt”;
Invoke-Sqlcmd -ServerInstance $server -Query "SELECT @@VERSION;"
}
Extending this to include input from a file:
$servers=get-content "D:\Nigel\PowerShell\SqlServers.txt"
foreach($server in $servers)
{
# go to each server and execute the command in D:\Nigel\PowerShell\SQLCommand.txt
# Invoke-sqlcmd -ServerInstance $server -Database master
Invoke-Sqlcmd -ServerInstance $server -InputFile “D:\Nigel\PowerShell\SQLCommand.txt”;
}
SQLCommand.txt
==============
SELECT @@VERSION;
SqlServers.txt
==============
CHMyServer1
CHMyServer2
Here’s a URL that describes how to make your own powershell commandlets
http://www.gangleri.net/2009/04/21/BuildingPowerShellCmdletsWithVisualStudio2008.aspx
Building PowerShell Cmdlets with Visual Studio 2008
1.Download Windows PowerShell Cmdlet templates from CodePlex
2.Run the VSI installer to install the project and item templates
3.Open Visual Studio 2008 and select File->New->Project
4.From the New Project dialog select the ‘PowerShellCmdlet’ template
5.Now right click on the project and select ‘Add New Item’. This will show the ‘Add New Item’ dialog. You can see that there are class templates for PSCmdlets, Cmdlets, SnapIns and XML helper files. For now select the Cmdlet template.
6.The ProcessRecord method performs the actual processing for the Cmdlet, in this example we will just call the WriteObject method to display the ‘Hello World’ message.
using System.Management.Automation;
namespace HelloPowerShell
{
[Cmdlet(VerbsCommon.Get, "HelloCmdlet")]
public class HelloCmdlet : Cmdlet
{
protected override void ProcessRecord()
{
WriteObject("Hello World!");
}
}
}
7.Again right click on the project and select add new item, this time select the ‘PowerShellCmdlet Help XML’. The template will automatically prefix ‘.dll-help.xml’ therefore you should just type ‘Get-Hello’ as the name.
using System.Collections.ObjectModel;
using System.ComponentModel;
using System.Management.Automation;
using System.Management.Automation.Runspaces;
namespace HelloPowerShell
{
[RunInstaller(true)]
public class HelloSnapIn : CustomPSSnapIn
{
private Collection _cmdlets;
///
/// Gets description of powershell snap-in.
///
public override string Description
{
get { return "A Description of HelloCmdlet"; }
}
///
/// Gets name of power shell snap-in
///
public override string Name
{
get { return "HelloCmdlet"; }
}
///
/// Gets name of the vendor
///
public override string Vendor
{
get { return ""; }
}
public override Collection Cmdlets
{
get
{
if (null == _cmdlets)
{
_cmdlets = new Collection();
_cmdlets.Add(new CmdletConfigurationEntry
("Get-HelloCmdlet", typeof(HelloCmdlet), "Get-HelloCmdlet.dll-Help.xml"));
}
return _cmdlets;
}
}
}
} 8.Now add another new item and select the ‘PowerShellCmdlet SnapIn’ template. Call this ‘HelloSnapIn.cs’
9.By default the SnapIn template fills out some sample information using MyCmdlet as the name simply change this to refer to Get-HelloCmdlet.
10.You can build the project and get a dll
11.Now start Windows PowerShell and run with administrator privileges as you will be installing a Cmdlet.
12.If you are using a 64-bit version of windows issue the command:
Set-Alias installutil C:\Windows\Microsoft.NET\Framework64\v2.0.50727\InstallUtil.exe
If you have a 32-bit version of windows issues the command:
Set-Alias installutil C:\Windows\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe
13.Now install your Cmdlet with the command:
installutil HelloPowerShell.dll
14.You can verify that the Cmdlet has been installed with the command:
Get-PSSnapin -Registered
15.Add the snap-in to your shell with:
Add-PSSnapin HelloCmdlet
16.You can run the Cmdlet with the command Get-HelloCmdlet and you should see the message “Hello World!”
By the way in VS2010 the PowerShell templates are there by default
Thursday, 29 December 2011
Moving an SSIS Package from a 32 bit machine to a 64 bit machine
I have an SSIS package that works on a 32 bit server but fails on a 64 bit machine with a number of errors, the most interesting being:
Surfing I found these sites...
http://social.msdn.microsoft.com/Forums/en/sqlintegrationservices/thread/289e29ad-26dc-4f90-bad4-ffb86c76e5f9
http://toddmcdermid.blogspot.com/2009/10/quick-reference-ssis-in-32-and-64-bits.html
It turns out that there are a number of drivers that are available in 32 bit but are not available in 64 bit. It is possible to run an SSIS package in 32 bit on a 64 bit machine. These URLs suggest
The first thing is to make the Visual Studio 2008 DTS package run in 32 bit mode in the designer. This can be done by:
DTExec /f "D:\MyPath\MySSISPackage.dtsx" /SET \package.Variables[Variable1].Value;"Value1" /SET \package.Variables[Variable2].Value;"Value2" /X86
This works well if the total length of the command line is less than 255 characters. In my case I had a lot of parameters.
To get arround this 255 caharacter limitation I think the best solution is to make a command line executable that is compiled in 32 bit. I thought about making a power shell command for this but decided it would be simpler to keep it to a command line and search the standard output for "Success" or "Failure".
To get this to work I included the following DLLs:
To know for sure that this is running in 32 bit the Console.WriteLine("SizeOf IntPtr is: {0}", IntPtr.Size); equals 4 for 32 bit and 8 for 64 bit.
I call this from within VB.NET in:
[Excel Destination [511]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC00F9304. There may be error messages posted before this with more information on why the AcquireConnection method call failed.
http://social.msdn.microsoft.com/Forums/en/sqlintegrationservices/thread/289e29ad-26dc-4f90-bad4-ffb86c76e5f9
http://toddmcdermid.blogspot.com/2009/10/quick-reference-ssis-in-32-and-64-bits.html
It turns out that there are a number of drivers that are available in 32 bit but are not available in 64 bit. It is possible to run an SSIS package in 32 bit on a 64 bit machine. These URLs suggest
The first thing is to make the Visual Studio 2008 DTS package run in 32 bit mode in the designer. This can be done by:
- Goto the properties of the SSIS project
- In the left tree view select Configuration Properties
- Set Run64BitRuntime = false
This enables you to debug the SSIS package. The next step is to execute the package in 32 bit. The easiest approach is to use
DTExec /f "D:\MyPath\MySSISPackage.dtsx" /SET \package.Variables[Variable1].Value;"Value1" /SET \package.Variables[Variable2].Value;"Value2" /X86
This works well if the total length of the command line is less than 255 characters. In my case I had a lot of parameters.
To get arround this 255 caharacter limitation I think the best solution is to make a command line executable that is compiled in 32 bit. I thought about making a power shell command for this but decided it would be simpler to keep it to a command line and search the standard output for "Success" or "Failure".
To get this to work I included the following DLLs:
D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SqlServer.ConnectionInfo.dll
D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SqlServer.Dts.Design.dllThe code looked like:
D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SqlServer.DTSPipelineWrap.dll
D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SQLServer.DTSRuntimeWrap.dll
D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SQLServer.ManagedDTS.dll
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using Microsoft.SqlServer.Dts.Runtime;
using System.IO;
using System.Threading;
namespace Ssis32BitExec
{
class Program
{
static void Main(string[] args)
{
if (args.Count() != 2)
{
Console.WriteLine("Ssis32BitExec ");
Console.WriteLine("===========================================");
Console.WriteLine("");
Console.WriteLine("Utility to execute an SSIS package in 32 bit mode thus enabling 32 bit drivers");
Console.WriteLine("that may not be available in 64 bit");
Console.WriteLine("");
Console.WriteLine("");
Console.WriteLine(" UNC path to .dtsx file (SSIS package)");
Console.WriteLine(" Path to parameter name and value pair file (This file");
Console.WriteLine(" contains lines like ParameterName;ParameterValue)");
Console.ReadLine();
Environment.Exit(0);
}
string packagePath = args[0];
string parameterValuePairsPath = args[1];
Console.WriteLine("SizeOf IntPtr is: {0}", IntPtr.Size);
if (!File.Exists(packagePath))
{
Console.WriteLine("Failed: The following file does not exist {0}", packagePath);
Environment.Exit(0);
}
if (!File.Exists(parameterValuePairsPath))
{
Console.WriteLine("Failed: The following file does not exist {0}", parameterValuePairsPath);
Environment.Exit(0);
}
XmlDocument packageDoc = new XmlDocument();
packageDoc.Load(packagePath);
Package package = new Package();
package.LoadFromXML(packageDoc.DocumentElement, null);
string[] lines= File.ReadAllLines(parameterValuePairsPath);
string[] Fields;
foreach(string line in lines)
{
Fields = line.Split(';');
if (Fields.Count() != 2)
{
Console.WriteLine("Failed: The following line is not correctly formated {0}", line);
Thread.CurrentThread.Abort();
}
if (package.Variables.Contains(Fields[0]))
{
package.Variables[Fields[0]].Value=Fields[1];
}
}
//This is a fudge factor that allows a badly constructed package to throw some errors and run through
//package.MaximumErrorCount = 2;
DTSExecResult result = package.Execute();
if (result == DTSExecResult.Success)
{
Console.WriteLine("Success");
Environment.Exit(0);
}
else
{
Console.WriteLine("Failed");
for (int i = 0; i < package.Errors.Count; i++) { Console.WriteLine(package.Errors[i].Description); } Environment.Exit(1); } } } } To know for sure that this is running in 32 bit the Console.WriteLine("SizeOf IntPtr is: {0}", IntPtr.Size); equals 4 for 32 bit and 8 for 64 bit.
I call this from within VB.NET in:
Dim TempFilePath As String = IO.Path.GetTempFileName
Dim Sw As New StreamWriter(TempFilePath)
Sw.WriteLine("Param1;{0}", Value1)
Sw.WriteLine("Param2;{0}", Value2)
Sw.Close()
Dim SSIS32BitProcess As New ProcessStartInfo()
SSIS32BitProcess.WorkingDirectory = Environment.CurrentDirectory
SSIS32BitProcess.FileName = "Ssis32BitExec.exe"
SSIS32BitProcess.UseShellExecute = False
SSIS32BitProcess.RedirectStandardOutput = True
SSIS32BitProcess.RedirectStandardError = True
SSIS32BitProcess.Arguments = packagePath + " " + TempFilePath
Dim Proc As Process = Process.Start(SSIS32BitProcess)
Dim Out As String = Proc.StandardOutput.ReadToEnd()
Proc.WaitForExit()
Dim Err As String = Proc.StandardError.ReadToEnd()
Proc.WaitForExit()
If Out.Contains("Success") Then
Return True
Else
Message = Out + vbCr + Err
Return False
End If
File.Delete(TempFilePath)
Return TrueMonday, 5 December 2011
How to version control databases using TFS and Visual Studio Database Projects
We have some databases that contain certain reference data. We would like to keep track of changes to both the schema and the reference data and store both in one central place. Another challenge was to allow everyone to develop on a local version of the database thus avoiding colliding and breaking changes. This was solved in the following way:
1. Open Visual Studio 2010
2. File/New Project
3. Go to the tree view of the New Project Dialog select
Database/SQL Server
4. In the list select SQL Server 2008 Database Project
- Choose an appropriate Solution Name
- The Name should be the same as the database you are putting into source control
5. [OK]
6. Right mouse click on the database project and select "Import Database Objects and Settings"
7. Click New Connection
8. Server Name = (local)
Database Name = the local database you are trying to version
[OK]
9. Select the following check boxs:
Script the collation only if it is different from the database collation
Import extended properties
10. [Start]
If you have multiple instances of SQL Server (eg 2005, 2008 etc) Then you need to create an Alias for the database instance that you are targeting
Aliases are created by:
1. Open Sql Server Configuration Manager
2. Click on the SQL Server Network Configuration and enable Named Pipes
3. Click on all versions of SQL Native Cient 10.0 Configuration
Add a new alias
- Alias Name: MyDB_LOCAL
- Pipe Name: \\localhost\pipe\MSSQL2008R2
- Protocol: Named Pipes
- Server: localhost\SQL2008R2
4. Repeat 3 for SQL Native Cient 10.0 Configuration (32bit)
The Database may be documented using extended properties
Extended properties are access by
1. Open the Microsoft SQL Server Management Studio ...
2. Right Mouse click and select properties
3. In the tree view select properties
4. Add the combination of Name and Value
Extended properties are available on the database level down to the table field level
After this you have imported the database schema into a Visual Studio Database project. The next step is to insert data. Depending on what you want to achieve there are several mechanisms for importing data.
The PostDeployment script is intended to be used for populating reference data. This can be found in the following way:
In the VS2010 database projects under the solution directory Scripts\PostDeployment\Script.PostDeployment.sql
In the file system under DatabaseProject\Scripts\Post-Deployment
There are several possibilities for inserting reference data:
1. If it's small you can use INSERT INTO eg
INSERT INTO [MyDatabase].[dbo].[DATABASE_VERSION] ([DATE],[MODEL_VERSION],[DATABASE_VERSION],[COMMENTS])
VALUES ('2011-01-07','4.1.0','1.0.0','A comment about what is new in this version')
GO
2. If it's big and in multiple tables any you are not looking to compare changes in reference data you can include a bakup of a database.
Then the script would look something like:
USE [master]
go
if (exists(select * from sys.databases where name=N'Testdata') )
begin
EXEC msdb.dbo.sp_delete_database_backuphistory @database_name = N'Testdata'
ALTER DATABASE [Testdata] SET SINGLE_USER WITH ROLLBACK IMMEDIATE
DROP DATABASE [Testdata]
end
go
-- Note that SolutionRoot is an environmental variable that can either be set in the control panel or via a batch file that opens the database project
RESTORE DATABASE [Testdata] FROM DISK = N'$(SolutionRoot)\MyDatabase\Scripts\Post-Deployment\Testdata.bak' WITH FILE = 1, MOVE N'Testdata' TO N'$(DefaultSqlFileLocation)\Testdata.mdf', MOVE N'Testdata_log' TO N'$(DefaultSqlFileLocation)\Testdata_1.ldf', NOUNLOAD, REPLACE, STATS = 10
GO
INSERT [MyDatabase].[dbo].[StudentList] (StFName,StLName,StEmail,OrderID)
SELECT StFName,StLName,StEmail,OrderID FROM [Testdata].[dbo].[StudentList]
3. Here is another method using BCP and a binary file.
This can be useful when you don't want the inconvenience of creating a backup etc.
To create the bcp file (adjust the table in the from-clause, and possibly the server after the -S param):
bcp "select field1,field2,field3 from MyDatabase.dbo.MyTable order by Id asc" queryout MyTable.bcp -S localhost -T -E -N
The script looks something like:
:setvar EstimatedRowCount 5000000
ALTER TABLE MyDatabase.dbo.MyTable NOCHECK CONSTRAINT ALL
bulk insert MyDatabase.dbo.MyTable from '$(SolutionRoot)\MyDatabase\Scripts\Post-Deployment\MyTable.bcp' with (DATAFILETYPE = 'widenative', order(ModelEventId asc), keepidentity, KEEPNULLS, ROWS_PER_BATCH = $(EstimatedRowCount))
ALTER TABLE MyDatabase.dbo.MyTable WITH CHECK CHECK CONSTRAINT ALL
4. Here is another method using BCP and a text file.
With this it is possible to compare versions of reference data directly out of TFS
To create the text file
i. Right mouse click on the database name and select All Tasks/Export Data
ii. Select Next and then in the Choose a source dialog select
- Database source : SQL Server Native Client 10.0
- Server Name : The Server Name
- Database : The database
[Next]
iii In the Choose a destination dialog choose:
- Destination : Flat File Destination
- File name : path to tablename.txt
- Locale : English (United states)
- Code Page 1252 (NASI Latin1)
- Unicode is not checked
- Format : Delimted
- Text qualifier :
- Column names in the first data row is not checked
Next
iv - Copy data from one or more tables or views
Next
v . Source taböle or view : the table to be exported
- Row delimiter : {CR}{LF}
- Column delimiter : Semicolon {;}
Next
vi - Run immediately
Finish
Finish
Then to import script looks like:
BULK INSERT MyTable FROM '$(TFSROOT)\DataBases\MyDatabase\Scripts\Post-Deployment\MyTable.txt' WITH (FIELDTERMINATOR = ';', ROWTERMINATOR = '\n' )
So now we have a Visual Studio 2010 Database project and data. Now suppose that you have a view that references another table in another database. You then create a sub database project within the database solution but when you want to deploy the database you get an error like
Error 2 SQL03006: View: [dbo].[MyView] has an unresolved reference to object [MyOtherDatabase].[dbo].[MyOtherTable]. D:\SOURCE\DataBases\MyDatabase\Schema Objects\Schemas\dbo\Views\MyView.view.sql 12 23 MyDatabase
Looking at the SQL
CREATE VIEW dbo.MyView
AS
SELECT dbo.MyTable.field1, MyOtherDatabase.dbo.MyOtherTable.Field
FROM dbo.MyTable INNER JOIN
MyOtherDatabase.dbo.MyOtherTable ON dbo.MyTable.Id = MyDatabase.dbo.MyTable.FKId
The problem is MyOtherDatabase.dbo.MyOtherTable.Field to a field in a table in another database. This is causing the deployment to fail. The solution to this can be found in an article on how to defining Cross-Database References at http://msdn.microsoft.com/en-us/library/bb386242.aspx
When adding the database reference you must define a database variable.
Fill out the Add Database Reference as follows:
- Database projects in the current solution: YourRefDB
- Database Reference Variables
- Name
- Uncheck Literal
Name $(IO_DB) Value YourRefDB
- Check Update the existing schema object definitions and scripts to use the database variables
- Check Suppress errors caused by unresolved references in the reference project
[OK]
Then you get to an opportunity to edit the modifications to the scripts
This resolves the compilation errors.
The next problem comes when we want to change the Deployment function from generating a script to deploying the script
Going to the solution view and looking at the properties of the database project and changing the Deploy action to "Create a deployment script (.sql) and deploy to database
Message 1 The deployment script was generated, but was not deployed. You can change the deploy action on the Deploy tab of the project properties. D:\SOURCE\DataBases\DBProj\DBName\sql\debug\DbName.sql 0 0 DBName
The solution to this is:
1 Right mouse click on the database project and select properties
2 Goto the Deployment tab
3 Edit Target connection
Server :(local)
Database name :MyDatabase
After this the deployment of the database project works…
I found some other curious effects. If you try and open this database project on a server where you have uninstalled SQL2008 R2 and installed SQL2008 SP3 you will get the following error
D:\SOURCE\DataBases\MyDBProject\DBName\DBName.dbproj : error : Could not load file or assembly 'Microsoft.SqlServer.Management.SqlParser, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
This problem is caused because the uninstall process removed some assemblies needed by Visual Studio to correctly interpret the database project. There are 2 ways to solve this. Either re-install VS2010 or run the following commands
Open a command line and goto: D:\Software\VS2010\DVD\WCU\SMO
Then execute
msiexec /i SQLSysClrTypes_amd64_enu.msi
msiexec /i SharedManagementObjects_amd64_enu.msi
Next change dir to D:\Software\VS2010\DVD\WCU\DAC and execute:
msiexec /i DACFramework_enu.msi
msiexec /i DACProjectSystemSetup_enu.msi
msiexec /i TSqlLanguageService_enu.msi
Another interesting thing happens when the MS SQL2008 Server has been setup with different paths.
To fix this problem go to the database project and then open the following solution folder path Schema Objects/Database Level Objects/Storage/Files. Here you will find 2 files with names like:
MyDatabase.sqlfile.sql
MyDatabase_log.sqlfile.sql
Within these files you have
ALTER DATABASE [$(DatabaseName)]
ADD FILE (NAME = [MyDatabaseName], FILENAME = 'G:\Data\MyDatabaseName.mdf', FILEGROWTH = 1024 KB) TO FILEGROUP [PRIMARY];
To make the database project run on any setup you need to modify the path using the environmental variables DefaultDataPath and DefaultLogPath. For example:
ALTER DATABASE [$(DatabaseName)]
ADD FILE (NAME = [$(DatabaseName)], FILENAME = '$(DefaultDataPath)$(DatabaseName).mdf', FILEGROWTH = 1024 KB) TO FILEGROUP [PRIMARY];
By the way if you want to use environmental variables within a Query in MS SQL Server Manager you need to:
1. Goto the Query drop down menu
2. Select SQLCMD Mode
3. Cut and paist script....
Then you can use commands like
:setvar somevariable somevalue
1. Open Visual Studio 2010
2. File/New Project
3. Go to the tree view of the New Project Dialog select
Database/SQL Server
4. In the list select SQL Server 2008 Database Project
- Choose an appropriate Solution Name
- The Name should be the same as the database you are putting into source control
5. [OK]
6. Right mouse click on the database project and select "Import Database Objects and Settings"
7. Click New Connection
8. Server Name = (local)
Database Name = the local database you are trying to version
[OK]
9. Select the following check boxs:
Script the collation only if it is different from the database collation
Import extended properties
10. [Start]
If you have multiple instances of SQL Server (eg 2005, 2008 etc) Then you need to create an Alias for the database instance that you are targeting
Aliases are created by:
1. Open Sql Server Configuration Manager
2. Click on the SQL Server Network Configuration and enable Named Pipes
3. Click on all versions of SQL Native Cient 10.0 Configuration
Add a new alias
- Alias Name: MyDB_LOCAL
- Pipe Name: \\localhost\pipe\MSSQL2008R2
- Protocol: Named Pipes
- Server: localhost\SQL2008R2
4. Repeat 3 for SQL Native Cient 10.0 Configuration (32bit)
The Database may be documented using extended properties
Extended properties are access by
1. Open the Microsoft SQL Server Management Studio ...
2. Right Mouse click and select properties
3. In the tree view select properties
4. Add the combination of Name and Value
Extended properties are available on the database level down to the table field level
After this you have imported the database schema into a Visual Studio Database project. The next step is to insert data. Depending on what you want to achieve there are several mechanisms for importing data.
The PostDeployment script is intended to be used for populating reference data. This can be found in the following way:
In the VS2010 database projects under the solution directory Scripts\PostDeployment\Script.PostDeployment.sql
In the file system under DatabaseProject\Scripts\Post-Deployment
There are several possibilities for inserting reference data:
1. If it's small you can use INSERT INTO eg
INSERT INTO [MyDatabase].[dbo].[DATABASE_VERSION] ([DATE],[MODEL_VERSION],[DATABASE_VERSION],[COMMENTS])
VALUES ('2011-01-07','4.1.0','1.0.0','A comment about what is new in this version')
GO
2. If it's big and in multiple tables any you are not looking to compare changes in reference data you can include a bakup of a database.
Then the script would look something like:
USE [master]
go
if (exists(select * from sys.databases where name=N'Testdata') )
begin
EXEC msdb.dbo.sp_delete_database_backuphistory @database_name = N'Testdata'
ALTER DATABASE [Testdata] SET SINGLE_USER WITH ROLLBACK IMMEDIATE
DROP DATABASE [Testdata]
end
go
-- Note that SolutionRoot is an environmental variable that can either be set in the control panel or via a batch file that opens the database project
RESTORE DATABASE [Testdata] FROM DISK = N'$(SolutionRoot)\MyDatabase\Scripts\Post-Deployment\Testdata.bak' WITH FILE = 1, MOVE N'Testdata' TO N'$(DefaultSqlFileLocation)\Testdata.mdf', MOVE N'Testdata_log' TO N'$(DefaultSqlFileLocation)\Testdata_1.ldf', NOUNLOAD, REPLACE, STATS = 10
GO
INSERT [MyDatabase].[dbo].[StudentList] (StFName,StLName,StEmail,OrderID)
SELECT StFName,StLName,StEmail,OrderID FROM [Testdata].[dbo].[StudentList]
3. Here is another method using BCP and a binary file.
This can be useful when you don't want the inconvenience of creating a backup etc.
To create the bcp file (adjust the table in the from-clause, and possibly the server after the -S param):
bcp "select field1,field2,field3 from MyDatabase.dbo.MyTable order by Id asc" queryout MyTable.bcp -S localhost -T -E -N
The script looks something like:
:setvar EstimatedRowCount 5000000
ALTER TABLE MyDatabase.dbo.MyTable NOCHECK CONSTRAINT ALL
bulk insert MyDatabase.dbo.MyTable from '$(SolutionRoot)\MyDatabase\Scripts\Post-Deployment\MyTable.bcp' with (DATAFILETYPE = 'widenative', order(ModelEventId asc), keepidentity, KEEPNULLS, ROWS_PER_BATCH = $(EstimatedRowCount))
ALTER TABLE MyDatabase.dbo.MyTable WITH CHECK CHECK CONSTRAINT ALL
4. Here is another method using BCP and a text file.
With this it is possible to compare versions of reference data directly out of TFS
To create the text file
i. Right mouse click on the database name and select All Tasks/Export Data
ii. Select Next and then in the Choose a source dialog select
- Database source : SQL Server Native Client 10.0
- Server Name : The Server Name
- Database : The database
[Next]
iii In the Choose a destination dialog choose:
- Destination : Flat File Destination
- File name : path to tablename.txt
- Locale : English (United states)
- Code Page 1252 (NASI Latin1)
- Unicode is not checked
- Format : Delimted
- Text qualifier :
- Column names in the first data row is not checked
Next
iv - Copy data from one or more tables or views
Next
v . Source taböle or view : the table to be exported
- Row delimiter : {CR}{LF}
- Column delimiter : Semicolon {;}
Next
vi - Run immediately
Finish
Finish
Then to import script looks like:
BULK INSERT MyTable FROM '$(TFSROOT)\DataBases\MyDatabase\Scripts\Post-Deployment\MyTable.txt' WITH (FIELDTERMINATOR = ';', ROWTERMINATOR = '\n' )
So now we have a Visual Studio 2010 Database project and data. Now suppose that you have a view that references another table in another database. You then create a sub database project within the database solution but when you want to deploy the database you get an error like
Error 2 SQL03006: View: [dbo].[MyView] has an unresolved reference to object [MyOtherDatabase].[dbo].[MyOtherTable]. D:\SOURCE\DataBases\MyDatabase\Schema Objects\Schemas\dbo\Views\MyView.view.sql 12 23 MyDatabase
Looking at the SQL
CREATE VIEW dbo.MyView
AS
SELECT dbo.MyTable.field1, MyOtherDatabase.dbo.MyOtherTable.Field
FROM dbo.MyTable INNER JOIN
MyOtherDatabase.dbo.MyOtherTable ON dbo.MyTable.Id = MyDatabase.dbo.MyTable.FKId
The problem is MyOtherDatabase.dbo.MyOtherTable.Field to a field in a table in another database. This is causing the deployment to fail. The solution to this can be found in an article on how to defining Cross-Database References at http://msdn.microsoft.com/en-us/library/bb386242.aspx
When adding the database reference you must define a database variable.
Fill out the Add Database Reference as follows:
- Database projects in the current solution: YourRefDB
- Database Reference Variables
- Name
- Uncheck Literal
Name $(IO_DB) Value YourRefDB
- Check Update the existing schema object definitions and scripts to use the database variables
- Check Suppress errors caused by unresolved references in the reference project
[OK]
Then you get to an opportunity to edit the modifications to the scripts
This resolves the compilation errors.
The next problem comes when we want to change the Deployment function from generating a script to deploying the script
Going to the solution view and looking at the properties of the database project and changing the Deploy action to "Create a deployment script (.sql) and deploy to database
Message 1 The deployment script was generated, but was not deployed. You can change the deploy action on the Deploy tab of the project properties. D:\SOURCE\DataBases\DBProj\DBName\sql\debug\DbName.sql 0 0 DBName
The solution to this is:
1 Right mouse click on the database project and select properties
2 Goto the Deployment tab
3 Edit Target connection
Server :(local)
Database name :MyDatabase
After this the deployment of the database project works…
I found some other curious effects. If you try and open this database project on a server where you have uninstalled SQL2008 R2 and installed SQL2008 SP3 you will get the following error
D:\SOURCE\DataBases\MyDBProject\DBName\DBName.dbproj : error : Could not load file or assembly 'Microsoft.SqlServer.Management.SqlParser, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
This problem is caused because the uninstall process removed some assemblies needed by Visual Studio to correctly interpret the database project. There are 2 ways to solve this. Either re-install VS2010 or run the following commands
Open a command line and goto: D:\Software\VS2010\DVD\WCU\SMO
Then execute
msiexec /i SQLSysClrTypes_amd64_enu.msi
msiexec /i SharedManagementObjects_amd64_enu.msi
Next change dir to D:\Software\VS2010\DVD\WCU\DAC and execute:
msiexec /i DACFramework_enu.msi
msiexec /i DACProjectSystemSetup_enu.msi
msiexec /i TSqlLanguageService_enu.msi
Another interesting thing happens when the MS SQL2008 Server has been setup with different paths.
To fix this problem go to the database project and then open the following solution folder path Schema Objects/Database Level Objects/Storage/Files. Here you will find 2 files with names like:
MyDatabase.sqlfile.sql
MyDatabase_log.sqlfile.sql
Within these files you have
ALTER DATABASE [$(DatabaseName)]
ADD FILE (NAME = [MyDatabaseName], FILENAME = 'G:\Data\MyDatabaseName.mdf', FILEGROWTH = 1024 KB) TO FILEGROUP [PRIMARY];
To make the database project run on any setup you need to modify the path using the environmental variables DefaultDataPath and DefaultLogPath. For example:
ALTER DATABASE [$(DatabaseName)]
ADD FILE (NAME = [$(DatabaseName)], FILENAME = '$(DefaultDataPath)$(DatabaseName).mdf', FILEGROWTH = 1024 KB) TO FILEGROUP [PRIMARY];
By the way if you want to use environmental variables within a Query in MS SQL Server Manager you need to:
1. Goto the Query drop down menu
2. Select SQLCMD Mode
3. Cut and paist script....
Then you can use commands like
:setvar somevariable somevalue
Friday, 7 October 2011
Cross platform applications with HTML5/JavaScript and how this ties in with SOLID OO principles
I have just started to think about writing applications that are cross platform impendent ie that can run on devices like iPad. Here are some notes about this subject
Over the last few years I have made extensive use of SOLID. OO programmin principles as I developed C# applications in .Net.
Where
SRP: The Single Responsibility Principle A class should have one, and only one, reason to change.
OCP: The Open Closed Principle You should be able to extend a classes behavior, without modifying it.
LSP: The Liskov Substitution Principle Derived classes must be substitutable for their base classes.
ISP: The Interface Segregation Principle Make fine grained interfaces that are client specific.
DIP: The Dependency Inversion Principle Depend on abstractions, not on concretions.
My first reaction was to reapply these principles in JavaScript. So I started to think of interfaces, dependency injection containers, mocking frameworks and Test Driven development. It turns out that there is a fundamental difference between C# and JavaScript in the sence that c# is class based and JavaScript is prototypical. So it is difficult to apply the notion of interfaces and abstraction in the world of JavaScript.
So the question is what principles are relevant to a Prototypical language over a Class based OO language. As far as I can see :
S Applicable
O Not Applicable
L Applicable
I Not Applicable
D Not Applicable
One principle that applys to both programming paradigms is DRY which stands for Don't Repeat Yourself
JavaScript Object Oriented Programming include classes, inheritance, and scope that can be used encapsulate, support namespaces, and avoid collisions. Actually JavaScript does not have a class entity BUT it implements the pattern of classes. The difference is in the inheritance model. In other object oriented languages, class is an actual data type that represents the blueprint for creating objects. In JavaScript, although we can use Functions to simulate an object blueprint, they are just in fact objects themselves. These objects are then used as models (aka prototypes) for other objects. Applying the concept of prototypal inheritance allows us to create “subclasses”, or objects that inherit the properties of another object. This becomes particularly useful when we want to use the methods of another object with some slight modifications.
The next thing I was thinking about is that after programming a while in Xaml I preffer to use the MVVM model over the MVC. This is in MVC the Controller is tightly coupled to the view. In practical terms this means that every time you need to change a control on the view the controller needs to change things like how this effects all the other controls. Where as in MVVM the View is completely separate from the VM and thus a change to a control results in minimal update to the ViewModel. It turns out that there is a JavaScript library called KnockOut.js that makes the MVVM pattern possible
Here are some examples of JavaScript. It can be seen that JavaScript is far from just a begineers language. JavaScript's prototypal OOP is flexible and it does not limit the programmer to things like static typing.
Here is an example of how to create your own foreach function:
Array.prototype.foreach.function(callback)
{
for (var i=0; i= companrand:}
"=" : function(comporand){return function(e){return e= companrand:}
}
}
a.set(op[">="](5).foreach(alert);
This is called Curring after Mr Curry.
btw If you uise this in DOS you will need toescape the > to ^>
----
Here is a way tobiuld up the arrays:
Number.protype.inputModule
{
UpTo : function(Upeer,step)
{
var a[]
for(var i = this.valueof(Upper); jreturn a;
}
}
(1).Upto(20,2).....
This is called High Order Programming
Here's a link to some further examples: http://www.w3schools.com/js/
The next version of Visual Studio will have a lot more support for JavaScript development, so this should reduce the barrier to trying this out...
Over the last few years I have made extensive use of SOLID. OO programmin principles as I developed C# applications in .Net.
Where
SRP: The Single Responsibility Principle A class should have one, and only one, reason to change.
OCP: The Open Closed Principle You should be able to extend a classes behavior, without modifying it.
LSP: The Liskov Substitution Principle Derived classes must be substitutable for their base classes.
ISP: The Interface Segregation Principle Make fine grained interfaces that are client specific.
DIP: The Dependency Inversion Principle Depend on abstractions, not on concretions.
My first reaction was to reapply these principles in JavaScript. So I started to think of interfaces, dependency injection containers, mocking frameworks and Test Driven development. It turns out that there is a fundamental difference between C# and JavaScript in the sence that c# is class based and JavaScript is prototypical. So it is difficult to apply the notion of interfaces and abstraction in the world of JavaScript.
So the question is what principles are relevant to a Prototypical language over a Class based OO language. As far as I can see :
S Applicable
O Not Applicable
L Applicable
I Not Applicable
D Not Applicable
One principle that applys to both programming paradigms is DRY which stands for Don't Repeat Yourself
JavaScript Object Oriented Programming include classes, inheritance, and scope that can be used encapsulate, support namespaces, and avoid collisions. Actually JavaScript does not have a class entity BUT it implements the pattern of classes. The difference is in the inheritance model. In other object oriented languages, class is an actual data type that represents the blueprint for creating objects. In JavaScript, although we can use Functions to simulate an object blueprint, they are just in fact objects themselves. These objects are then used as models (aka prototypes) for other objects. Applying the concept of prototypal inheritance allows us to create “subclasses”, or objects that inherit the properties of another object. This becomes particularly useful when we want to use the methods of another object with some slight modifications.
The next thing I was thinking about is that after programming a while in Xaml I preffer to use the MVVM model over the MVC. This is in MVC the Controller is tightly coupled to the view. In practical terms this means that every time you need to change a control on the view the controller needs to change things like how this effects all the other controls. Where as in MVVM the View is completely separate from the VM and thus a change to a control results in minimal update to the ViewModel. It turns out that there is a JavaScript library called KnockOut.js that makes the MVVM pattern possible
Here are some examples of JavaScript. It can be seen that JavaScript is far from just a begineers language. JavaScript's prototypal OOP is flexible and it does not limit the programmer to things like static typing.
Here is an example of how to create your own foreach function:
Array.prototype.foreach.function(callback)
{
for (var i=0; i
"=" : function(comporand){return function(e){return e= companrand:}
}
}
a.set(op[">="](5).foreach(alert);
This is called Curring after Mr Curry.
btw If you uise this in DOS you will need toescape the > to ^>
----
Here is a way tobiuld up the arrays:
Number.protype.inputModule
{
UpTo : function(Upeer,step)
{
var a[]
for(var i = this.valueof(Upper); j
}
}
(1).Upto(20,2).....
This is called High Order Programming
Here's a link to some further examples: http://www.w3schools.com/js/
The next version of Visual Studio will have a lot more support for JavaScript development, so this should reduce the barrier to trying this out...
Tuesday, 27 September 2011
Condensed notes from the Build about where Microsoft technology is going over the next year
Here is a summary of what I learned at the Build. I went to a lot of trouble this year to attend this conference which was not easy considering that the agenda was empty up to the last minute. But it was certainly worth while because Microsoft reveled where it is going with its technology which included a unified user experience from mobile devices through to large television sized devices.
CNN “Microsoft unveils a radically redesigned Windows 8”
This technology will only be available in about a year from now, probably industry will adopt this in perhaps 2 years from now. This conference is important because no strategy would be complete without an idea of where Windows is going because it is our main operating system. It is necessary because our software engineers need to gather the skills needed to take advantage of this technology and it is necessary in order to make informed decisions over if and how to integrate other technology such as iPhone and iPad.
Windows 8 Metro Style
We can expect a shift towards touch base applications over the coming years. Microsoft is of the opinion that all screens will be touched enabled and the ones that are not will feel antiquated when used. To make use of the Windows 8 metro style the user will have to adapt by learning new gestures and become familiar with the concept of having a less cluttered desktop using active Tiles. This means a “Re imagine your application” when making an application in Metro style. It was emphasized that there are a number of applications where the chrome style is simply the most appropriate meaning that this style of UI will continued to be supported. The difference between metro and chrome will go along the lines of the difference between a DOS prompt and Windows ie an expert system that requires training to a intuitive system that requires minimal training. Chrome seems to be based on the philosophy that “Less is more”.
Performance and energy saving has become a key factor in the design of Windows 8. A cold start up takes about 20 seconds. To make this possible the number of running processes has been minimized and that tasks that are not currently in view are suspended. This has an impact on the design and development of applications because they only have a few seconds to persist there state before the task is suspended. It is now possible to boot a Windows 8 client directly from a USB memory stick. Also the first bleeding edge problems with VS are coming to light eg
http://blog.galasoft.ch/archive/2011/09/25/quick-tip-killing-a-metro-style-app-in-windows-8.aspx?utm_source=twitterfeed&utm_medium=twitter&utm_campaign=Feed%3A+galasoft+%28Laurent+Bugnion+%28GalaSoft%29%29
There are some features that may take more time to be adopted by the industry. Such as location aware applications that can make use of local devices or the touch sensor that perform a kind of electronic hand shake and setup a link between the 2 devices for collaboration or simply to exchange an electronic business card.
The Windows 8 user experience is based on a fast and fluid immersive and full screen user experience. Applications communicate with each other through “charms” that are implemented via shared contracts. The same intuitive interface will be used across all Windows devices from mobile devices through desk tops and televisions.
The result is a non cluttered canvas with controls. Commands that are frequently used go on the canvas, all others are revealed through the edge gestures. The metro style desktop consists of Tiles that provide simple text or images via pre defined templates and come in 2 sizes. Secondary tiles are created by pinning the content of the application and have the same capabilities as the main tile. They provide a deep link within the application. Live tiles keep people connected to your app and male it more likely the application is put on the first page.
Notifications appear via a toaster icon that appears for a short period of time and user must opt onto it. They use Windows Push Notification Service WNS and can update any time your application is running
CNN “Microsoft unveils a radically redesigned Windows 8”
This technology will only be available in about a year from now, probably industry will adopt this in perhaps 2 years from now. This conference is important because no strategy would be complete without an idea of where Windows is going because it is our main operating system. It is necessary because our software engineers need to gather the skills needed to take advantage of this technology and it is necessary in order to make informed decisions over if and how to integrate other technology such as iPhone and iPad.
Windows 8 Metro Style
We can expect a shift towards touch base applications over the coming years. Microsoft is of the opinion that all screens will be touched enabled and the ones that are not will feel antiquated when used. To make use of the Windows 8 metro style the user will have to adapt by learning new gestures and become familiar with the concept of having a less cluttered desktop using active Tiles. This means a “Re imagine your application” when making an application in Metro style. It was emphasized that there are a number of applications where the chrome style is simply the most appropriate meaning that this style of UI will continued to be supported. The difference between metro and chrome will go along the lines of the difference between a DOS prompt and Windows ie an expert system that requires training to a intuitive system that requires minimal training. Chrome seems to be based on the philosophy that “Less is more”.
Performance and energy saving has become a key factor in the design of Windows 8. A cold start up takes about 20 seconds. To make this possible the number of running processes has been minimized and that tasks that are not currently in view are suspended. This has an impact on the design and development of applications because they only have a few seconds to persist there state before the task is suspended. It is now possible to boot a Windows 8 client directly from a USB memory stick. Also the first bleeding edge problems with VS are coming to light eg
http://blog.galasoft.ch/archive/2011/09/25/quick-tip-killing-a-metro-style-app-in-windows-8.aspx?utm_source=twitterfeed&utm_medium=twitter&utm_campaign=Feed%3A+galasoft+%28Laurent+Bugnion+%28GalaSoft%29%29
There are some features that may take more time to be adopted by the industry. Such as location aware applications that can make use of local devices or the touch sensor that perform a kind of electronic hand shake and setup a link between the 2 devices for collaboration or simply to exchange an electronic business card.
The Windows 8 user experience is based on a fast and fluid immersive and full screen user experience. Applications communicate with each other through “charms” that are implemented via shared contracts. The same intuitive interface will be used across all Windows devices from mobile devices through desk tops and televisions.
The result is a non cluttered canvas with controls. Commands that are frequently used go on the canvas, all others are revealed through the edge gestures. The metro style desktop consists of Tiles that provide simple text or images via pre defined templates and come in 2 sizes. Secondary tiles are created by pinning the content of the application and have the same capabilities as the main tile. They provide a deep link within the application. Live tiles keep people connected to your app and male it more likely the application is put on the first page.
Notifications appear via a toaster icon that appears for a short period of time and user must opt onto it. They use Windows Push Notification Service WNS and can update any time your application is running
There is a new layer that sits directly on the Windows Kernal called WinRT. This is a clean API that has no duplication of Runtime API and all obsolete or inapplicable APIs have been removed. Metro style apps for c# and VB.net can interact directly with WinRT APIs or via .Net for Metro style apps via the CLS and Win32 APIs.
To publish a metro app there is a pipeline of checks that guarantee that an application quality. This consists of Pre Processing, Security tests, Technical Compliance, Content compliance, Signing and publishing.
x86, 64, and ARM processors will all be supported. This has a big positive implication that all mobile devices running on ARM processors will support these applications.
No solid time line was given. Some guesses included perhaps it being available about this time next year. The next milestone is Beta, the RC, then RTM and then GA
Windows 8 Server
I did not attend many Windows 8 server sessions. But a lot of progress has been made on performance. I also believe that the HyperV virtualization has been improved by lessons learned in Azure. This is a way to make low cost hard disks available in a SAN. This can perform very fast because it is possible to make use of more than 1 network card through teaming. It is also possible to change a system drive on the fly.
.Net 4.5
In the key note it is really sure that Silverlight is definitely not dead. Microsoft has been working on making it possible to program the UI in languages programmers might choose. This includes the HTML5/Java script as well as C++. Microsoft is in the process of making a metro style implementation of office. From what I could determine this is going to be made using HTML5 for the view part of the presentation layer and Java script as the model art of the view model. All this sits on WinRT APIs which sits directly on Window Kernel Services. At first site this has been misinterpreted last year as a move away from Silverlight. This is not the case because the idea is to allow the programmer to express themselves in any appropriate programming language and this was demonstrated by including demonstrations in C and C++. I asked some experts whether the HTML5/JavaScript implementation would use the MVVM pattern. The answer was that this question had been asked by a lot of people and that it should be possible but there are no examples for this yet.
Entity framework EF 4.5 now includes Enums, SQL Server and Azure features such as spatial functions
In .Net 4 data manipulation was made by first starting with the data and then setting up the computation. TPL Dataflow allows the computation to be first setup and then the data
Other Parallel Computing Additions
Combinators Task.WhenAll, Task.WhenAny
Timer integration Task.Delay(TimeSpan), CancellationTokenSource.CancelAfter(TimeSpan)
Task schedulingConcurrentExclusiveSchedulerPair
Fine-grained control DenyChildAttach, HideScheduler, LazyCancellation, EnumerablePartitionerOptions
ThreadLocal
PERFORMANCE (“it’s just faster”)
The garbage collector has been improved. A multi-cored JIT with pre-fetch options makes ASP.Net start 35% faster.
.Net 4.5 is an in place upgrade from .Net 4.0
VS2011
Firstly VS2011 can open VS2010 project without changing the format.
There is a 3D editor that has the capability of breaking down each transformation that is made to a 3D object. It is not designed for creating these objects but can manipulate them. Improvements have been made on the IDE experience for C++ programmers. This includes unit tests, and color intellisence pickers etc.
Visual Studio 11 is the tool for Windows 8 and supports VB.NET, C#, C++ & HTML5/JS. There is a new designer Available today built on shared architecture with Expression Blend. When shelving changes information over which windows are open are saved together with the shelf set and are re established when opening the shelve set the windows are reopened in their original position. There is also a document well that avoids each time a file is clicked upon that a new document window is created. There is a XAML editor with IntelliSense. It is now possible to use C# code directly in Jscript and have changes in the generated HTML fed back into the code behind files.
There is a static code analysis feature that allows you to look and replace chunks of repeated code.
PowerPoint templates enable the mocking of an interactive UI.
Asynchronous programming is necessary for creating responsive fast and fluent applications. Here are some new features coming with .Net 4.5
Asynchronous programming models
Windows Runtime: IAsyncOperation
.NET Framework: Task
JavaScript: Promises
All are objects representing “ongoing operations”
All use callbacks to signal completion of operation
Challenge: Callbacks turn your code inside out
Insight: Automatic transformation to callbacks is possible
Asynchronous methods automatically transform normal code into a callback state machine eg
public async Task
var client = new HttpClient();
var response = await client.GetAsync(url);
var text = response.Content.ReadAsString();
return XElement.Parse(text);
}
.Net 5.0 aka Project Roslyn
The .Net 5.0 JIT will be written in C#. This means that it will be possible to call APIs to the pipeline used to compile code. This pipeline comprises of 4 steps
1. Parser
2. Symbols / Meta data export
3. Binder
4. Emitter
Here’s an example
ScriptEngine engine = new ScriptEngine();
Session session = Session.Create();
Engine.Execute(“using System;, session);
Engine.Execute(“for(int I = 0; I < 10; i++) Console.WriteLine(i*i);”,session); Var f = (func
For (int i=0; i<10 i++) Console.WriteLine(f(i)); This might look strange but it could be used in allowing users to use a macro style interface to your application. In my case I doubt that I would use this but it is very interesting. I could imagine that this will allow mixing of languages that could be useful in creating fast code. There is also a command line Roslyn C# Compiler that runs in a similar style as for F# and includes intellisence and code tips. To create a reference to another assembly “er” is used, eg: Er “PresentationCore” Here’s an example of translating c# into VB Public VB.SytaxNode Covert( CS.SyntaxTree syntaxTree, IDictionary
Bool convertStrings =false)
{
Var text = syntaxTree.Text;
Var mode = syntaxTree.Root;
Var vbText = Convert(text, node, identifierMap, convertStrings);
Return VB.SyntaxTree.ParseCompilationUnit(vbText).Root;
}
There is also a feature for .Net 3.5 on demand that I believe means that we don’t need to install the .Net 3.5 framework when certain obsolete functions are required.
.Net 5 will be a fresh installation
HPC
I have met up with some HPC / Azure MVPs and discussed some of the problems that I encountered when making my cat model in the cloud. The problem was that I was not able to transfer a 60 GB VM into the cloud including a local SQL Server. The suggestion was to use SQL Azure and transfer data via blob storage. In the next couple of months there will be a release of HPC Server that runs of a standard worker role meaning that we don’t need to construct and maintain our own VM. For very large quantities of data we could consider speeding up the queries by using TriadLinq aka Linq to HPC. This works by producing a sealed data block that is replicated between multiple nodes and is then queried upon in parallel. In addition to the Windows server based HPC Scheduler there is an Azure based HPC Scheduler. These are useful because it means that we don’t need to write our own schedulers.
There was a demo of a PC with 4 water cooled graphics cards that had 2500 times the processing power of a Cray. The HPC team uses these machines for numerical tasks. Currently it takes a crack programmer to correctly program such an algorithm, the hope is that this will become abstracted away at some point in the future.
Azure
Azure is a very cheap, elastic and quickly configurable source of compute power and one that we really should take advantage of. I saw some presentations on debugging and branch caching.
ALM Application Life Cycle Management
Attendees of Build where given a golden ticket to an evaluation version of the SaaS version of Team Foundation Server 2010. This contains a many of Application Lifecyle Management features that are all a part of professional software engineering. The Application life cycle is based on 2 parts . The Development cycle where a product backlog is worked upon during sprints to create a working software asset. The Operations which consists of an Ops back log and monitoring. When a bug is discovered a corresponding requirement is made that is sent back to the development backlog. In addition to the existing ALM functions a Code review function was demonstrated in the context of a development team with inline or side by side code comparison
Currently both TFS Server and TFS SaaS are capable of:
Work items, Source Control and Build
Agile Product/Project Management
Test Case Management
Heterogeneous Development
TFS SaaS has the advantage when:
Near Zero setup and administration
Collaborate with anyone from anywhere
TFS Server has the advantage when
Virtual Test Lab Management
SharePoint Integration
Data Warehouse and Reporting
Subscribe to:
Posts (Atom)
