Nigel Findlater

Using Event Loss Tables and Year Loss Tables

2012-01-27T07:28:00.000-08:00

This leads on from the previous blog entry on probabilistic models. The platform I am working on incorporates catastrophe models that produce Event Loss Tables or ELTs. These are tables containing Events, Ground Up Loss, Ground Up Loss standard deviation, loss frequency and various other loss perspectives. These loss tables are sorted in order of increasing ground up loss giving a cumulative loss distribution. A function that fits this curve is called a cumulative distribution function or cdf. Pricing a Cat XL treaty is done by integrating the CDF. This can be made in a number of ways:

The simplest way is by burning cost. This means just summing up the events taht trigger the treaty. This works very well in low layers because there are naturally more low cost events than high cost events. For high layers there may be only a few events and the accuracy is not sufficient. In these cases we can try to fit a points to a function such as Paratoo or log polynomial and then integrate the resulting function over the Cat XL layer. There are a number of problems with this approach, for example the numerical regression may give unreasonable coefficients that give unexpected results. The method that we use the most is to to sampling each loss assuming a distribution like log normal and the standard deviation and applying a burning cost.

Event loss tables can become quite large but are still manageable compared to Year Loss Tables. For example it is possible to make a loss estimation of a current event by making a query based on event or combination of events in the event loss table. It is not possible to combine different ELTs of different event sets

Some cat models incorporate some information over when an event occurred. This makes it possible to model clustering of events much more accurately. This has quite an impact on pricing. For example inuring treaties can be priced much more accurately. Imagine with an ELT you need to assume an average number of events in a year. There is a big difference between having an average of 2 events per year and a year loss table that says on the first year there are 0 events on the second year there are 4 events. Both have a mean of 2 events but in the first year the treaty is not touched and in the second year it can depend on the number of reinstatements.

Year loss tables are event based which makes them easy to combine with other YLTs.

Here are the steps needed to create a YLT from an ELT

1. Decide on how long the YLT is to be based
2. Determine an average number of events that happen in a year
3. Use a Poison distribution with a random sample to determine how many events happen in a particular year
(The poison distribution provides a sequence of integer numbers )
4. Sample from the ELT randomly these events
5. Based on the Mean Loss and STD Loss make a sample with a distribution such as a Beta or log normal to determine an actual loss using another random variable

The random variables are created from seeds. For a given seed the same sequence of random numbers are generated each time the application is run. Mathematically it turns out that this is an acceptable way of sampling and it means that the resulting YLTs are not needed to be stored. This is advantageous because in order to get events in the high layers a long simulation period is needed which means the storage requirements of a YLT is much bigger than an ELT.

Now let's drill into the details. Step 3 sounds very easy, make a sample random sample of a Poisson distribution. The thing is that Poisson is expressed in a formula like frequency = Function( Mean, Standard Deviation). For example in Excel it looks like POISSON(A3,$B$1,TRUE). The problem is that we need to make random samples of the number of events with a given mean and sd. To do this we need to use something called Inversion theory. There are some statistical packages that have this function built in. For example in r it looks like

pois <- rpois(100, lambda=3)

pois[1:10]

[1] 1 2 3 2 2 2 3 3 6 2

In C# you need to find a library to do this. I found 2 that look interesting open source projects that have these functions

http://code.google.com/p/pzmath/source/checkout
http://www.alglib.net/

Introducing probablistic models

2012-01-16T07:05:00.000-08:00

Recently I have been revisiting some mathematics. Often maths becomes very complex very quickly so here I enjoy keeping it simple.

Probabilistic models consist of 3 parts
1. A model that produces a random variable(s) X
2. A model that creates probabilities for the random variables
3. The sum of frequencies must add up to 1

If we take the example of flipping a coin: The coin has 2 faces and these correspond to the random variable X. The values of X can be give a value such as X=1 for heads and X=2 for tails. The model would be the probability of X=1 is 0.5 and the probability of X=2 is 0.5 and the sum of these probabilities sum to 1. Then you have a coin that may not be completely round or evenly balanced meaning it may behave differently to the model.

There are other types of models for example the number of people in a queue can be modeled with Poison. Mr Poison was a lawyer that studied the lengths of queues Research on the "Probability of Judgments in Criminal and Civil Matters" and build a model around this. This is a very interesting model when modeling the length of queues, It's also very useful when converting an Event loss table to a Year loss table. Here the problem is to sample the number of events that happen in a year given a mean and a standard deviation

Catastrophe probabilistic models can also be broken down into these 3 components.
1. A modeling that combines the Hazard, Vulnerability and Exposure to produce the random variable ground up loss
2. A model that assigns a frequency for each event
3. An event catalogue where the sum of frequencies adds up to the total length of time the catalogue is constructed for

A catastrophe model is more elaborate than the coin example and comprises of the Hazard (the earthquake or the windstorm), the vulnerability (how fragile it is) and en exposure (what buildings are at risk)

Experimenting with Powershell

2012-01-05T05:28:00.000-08:00

Command line scripts have been around for a long time. Power shell adds a powerful fluent syntax that enables a richer and more expressive way of coding our batch files. Powershell comes with Windows 7 and above as well as Windows Server 2008 R2 and above

Here is how to start using it... The first time you start PowerShell (Under Administrative tools I selected Windows PowerShell Modules) you will get the following warning


Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. All rights reserved.
WARNING: File C:\Windows\system32\WindowsPowerShell\v1.0\Modules\PSDiagnostics\PSDiagnostics.psm1 cannot be loaded

because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details.

PS C:\Users\nfindlater>

To fix this do the following steps

1. Under Administrative tools select Server Manager
2. In the tree view select Server Manager/Features
3. Add Feature
4. Select Windows PowerShell
5. Next, Install
6. under Accessories in a directory caller PowerShell

PoweShell basics

Powershell respects admin rights. Certain commandlet information require elevated privileges. Meaning that if you need access to information that requires elevated privileges you need to start powershell as administrator.

Here is a simple example how powershell can be used. Start notepad then execute

get -process notepad
Stop-Process

This gets a list of processes named “notepad” and pipes them to a Stop-Process commandlet which has the effect of closing the notepad that you just opened.

There are other ways of getting to the same information, for example


PS C:\Users\nfindlater> get-process | where { $_.ProcessName -eq 'notepad' }
Handles NPM(K) PM(K) WS(K) VM(M) CPU(s) Id ProcessName
------- ------ ----- ----- ----- ------ -- -----------
47 2 1108 3732 61 0.20 3112 notepad
Here’s an example using the event log
PS C:\Users\nfindlater> get-eventlog -LogName application -Newest 3

Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
77896 Oct 04 15:14 Information Microsoft-Windows... 1 The description for Event
77895 Oct 04 15:14 Information Desktop Window Ma... 1073750827 The Desktop Window Manager
77894 Oct 04 15:14 Information Winlogon 1073745925 Windows license validated.

The output from a commandlet can be formatted in various ways, here are some examples:

get-process notepad
Format-List *

get-process
Format-Table name, starttime

get-process
where-object { $_.starttime }
Format-Table name, starttime
For WMI calls the command “get-wmiobject win32_service” does not give much detail. But the following provides plenty of detail:

get-wmiobject win32_service
Format-list *

Here are some examples of how to start applications

Start-process -FilePath notepad
Start-process -FilePath notepad -WindowStyle minimized


PS C:\Users\nfindlater> Get-EventLog application | where-object { $_.message -match 'Desk' }
Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
77895 Oct 04 15:14 Information Desktop Window Ma... 1073750827 The Desktop Window Manager was unable to start ...
This is faster
C:\Users\nfindlater> Get-EventLog application -InstanceId 1001

PS C:\Users\nfindlater> Get-EventLog application -newest 1
Index Time EntryType Source InstanceID Message
----- ---- --------- ------ ---------- -------
77897 Oct 04 16:40 Information MSSQLSERVER 1073760088 Database backed up. Database: ICW_20081001_tRI_...
Here are some more examples using the eventlog


PS C:\Users\nfindlater> Get-EventLog application -newest 1 
format-list *

EventID : 18264
MachineName : CHBByyyy.zzz.gggg.net
Data : {88, 71, 0, 0...}
Index : 77897
Category : Backup
etc

Here are some examples with dir.

Give me all files containing the string ground (Very usefull since in Windows 7 the text search function is missing)

dir *.* -recurs
select-string "ground"
Format-List FileName

Here’s a more elaborate version looking for files with “pro” in the name

dir *.* -recurs
Where-Object {$_.Name -like "*pro*"}
select-string "ground"
Format-List FileName

dir *.* -recurs
Where-Object {$_.Name -like "*"}
select-string "ground"
Format-List FileName
Get-Unique

Filename : myfile.vcxproj

dir *.* -recurs
Where-Object {$_.Name -like "*vcx*"}
select-string "ground"
Format-List LineNumber, Path
Get-Unique

LineNumber : 64
Path : D:\path\myfile.vcxproj

Here’s an example of how to replace a string in a file

(Get-Content .\File.TXT)
ForEach-Object {$_ -REPLACE "'", """" }
Set-Content TEST2.TXT

Here are some examples of how to read parts of text files

Get-Content ".\file_test.txt"
select -First 10

Get-Content ".\file_test.txt"
select -Last 10

Get-Content ".\file_test.txt"
select -First 3
select -Last 1

Get-Content ".\file_test.txt"
select -Skip 10

Power Shell with SQL Server

I found the following link that describes how to get the SQL features of powershell working

http://msdn.microsoft.com/en-us/library/cc281962.aspx

Basically you need to run the following script to enable sql commands to work

#
# Add the SQL Server Provider.
#
$ErrorActionPreference = "Stop"
$sqlpsreg="HKLM:\SOFTWARE\Microsoft\PowerShell\1\ShellIds\Microsoft.SqlServer.Management.PowerShell.sqlps"
if (Get-ChildItem $sqlpsreg -ErrorAction "SilentlyContinue")
{
throw "SQL Server Provider for Windows PowerShell is not installed."
}
else
{
$item = Get-ItemProperty $sqlpsreg
$sqlpsPath = [System.IO.Path]::GetDirectoryName($item.Path)
}

#
# Set mandatory variables for the SQL Server provider
#
Set-Variable -scope Global -name SqlServerMaximumChildItems -Value 0
Set-Variable -scope Global -name SqlServerConnectionTimeout -Value 30
Set-Variable -scope Global -name SqlServerIncludeSystemObjects -Value $false
Set-Variable -scope Global -name SqlServerMaximumTabCompletion -Value 1000
#
# Load the snapins, type data, format data
#
Push-Location
cd $sqlpsPath
Add-PSSnapin SqlServerCmdletSnapin100
Add-PSSnapin SqlServerProviderSnapin100
Update-TypeData -PrependPath SQLProvider.Types.ps1xml 
update-FormatData -prependpath SQLProvider.Format.ps1xml 
Pop-Location

After executing this scripts you can run commands like:

Invoke-Sqlcmd -Query "SELECT @@VERSION;"

Column1
-------
Microsoft SQL Server 2008 (RTM) - 10.0.1600.22 (Intel X86) ...

Here’s a more entertaining script that executes an arbitrary command on a list of SQL servers


$servers=get-content "D:\Nigel\PowerShell\SqlServers.txt"
foreach($server in $servers)
{
# go to each server and execute the command in D:\Nigel\PowerShell\SQLCommand.txt
# Invoke-sqlcmd -ServerInstance $server -Database master -InputFile “D:\Nigel\PowerShell\SQLCommand.txt”; 
Invoke-Sqlcmd -ServerInstance $server -Query "SELECT @@VERSION;"
} 
Extending this to include input from a file:
$servers=get-content "D:\Nigel\PowerShell\SqlServers.txt"
foreach($server in $servers)
{
# go to each server and execute the command in D:\Nigel\PowerShell\SQLCommand.txt
# Invoke-sqlcmd -ServerInstance $server -Database master 
Invoke-Sqlcmd -ServerInstance $server -InputFile “D:\Nigel\PowerShell\SQLCommand.txt”; 
} 

SQLCommand.txt
==============
SELECT @@VERSION;

SqlServers.txt
==============
CHMyServer1
CHMyServer2

Here’s a URL that describes how to make your own powershell commandlets

http://www.gangleri.net/2009/04/21/BuildingPowerShellCmdletsWithVisualStudio2008.aspx
Building PowerShell Cmdlets with Visual Studio 2008

1.Download Windows PowerShell Cmdlet templates from CodePlex
2.Run the VSI installer to install the project and item templates
3.Open Visual Studio 2008 and select File->New->Project
4.From the New Project dialog select the ‘PowerShellCmdlet’ template
5.Now right click on the project and select ‘Add New Item’. This will show the ‘Add New Item’ dialog. You can see that there are class templates for PSCmdlets, Cmdlets, SnapIns and XML helper files. For now select the Cmdlet template.
6.The ProcessRecord method performs the actual processing for the Cmdlet, in this example we will just call the WriteObject method to display the ‘Hello World’ message.

using System.Management.Automation;
namespace HelloPowerShell
{
[Cmdlet(VerbsCommon.Get, "HelloCmdlet")]
public class HelloCmdlet : Cmdlet
{
protected override void ProcessRecord()
{
WriteObject("Hello World!");
}
}
}

7.Again right click on the project and select add new item, this time select the ‘PowerShellCmdlet Help XML’. The template will automatically prefix ‘.dll-help.xml’ therefore you should just type ‘Get-Hello’ as the name.


using System.Collections.ObjectModel;
using System.ComponentModel;
using System.Management.Automation;
using System.Management.Automation.Runspaces;
namespace HelloPowerShell
{
[RunInstaller(true)]
public class HelloSnapIn : CustomPSSnapIn
{
private Collection _cmdlets;

/// 
/// Gets description of powershell snap-in.
/// 

public override string Description
{
get { return "A Description of HelloCmdlet"; }
}
/// 
/// Gets name of power shell snap-in
/// 

public override string Name
{
get { return "HelloCmdlet"; }
}
/// 
/// Gets name of the vendor
/// 

public override string Vendor
{
get { return ""; }
}
public override Collection Cmdlets
{
get
{
if (null == _cmdlets)
{
_cmdlets = new Collection();
_cmdlets.Add(new CmdletConfigurationEntry
("Get-HelloCmdlet", typeof(HelloCmdlet), "Get-HelloCmdlet.dll-Help.xml"));
}
return _cmdlets;
}
}
}
}

8.Now add another new item and select the ‘PowerShellCmdlet SnapIn’ template. Call this ‘HelloSnapIn.cs’

9.By default the SnapIn template fills out some sample information using MyCmdlet as the name simply change this to refer to Get-HelloCmdlet.

10.You can build the project and get a dll

11.Now start Windows PowerShell and run with administrator privileges as you will be installing a Cmdlet.

12.If you are using a 64-bit version of windows issue the command:
Set-Alias installutil C:\Windows\Microsoft.NET\Framework64\v2.0.50727\InstallUtil.exe

If you have a 32-bit version of windows issues the command:
Set-Alias installutil C:\Windows\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe

13.Now install your Cmdlet with the command:
installutil HelloPowerShell.dll

14.You can verify that the Cmdlet has been installed with the command:
Get-PSSnapin -Registered

15.Add the snap-in to your shell with:
Add-PSSnapin HelloCmdlet

16.You can run the Cmdlet with the command Get-HelloCmdlet and you should see the message “Hello World!”

By the way in VS2010 the PowerShell templates are there by default

Moving an SSIS Package from a 32 bit machine to a 64 bit machine

2011-12-29T01:35:00.000-08:00

I have an SSIS package that works on a 32 bit server but fails on a 64 bit machine with a number of errors, the most interesting being:

[Excel Destination [511]] Error: SSIS Error Code DTS_E_CANNOTACQUIRECONNECTIONFROMCONNECTIONMANAGER. The AcquireConnection method call to the connection manager "Excel Connection Manager" failed with error code 0xC00F9304. There may be error messages posted before this with more information on why the AcquireConnection method call failed.

Surfing I found these sites...

http://social.msdn.microsoft.com/Forums/en/sqlintegrationservices/thread/289e29ad-26dc-4f90-bad4-ffb86c76e5f9

http://toddmcdermid.blogspot.com/2009/10/quick-reference-ssis-in-32-and-64-bits.html

It turns out that there are a number of drivers that are available in 32 bit but are not available in 64 bit. It is possible to run an SSIS package in 32 bit on a 64 bit machine. These URLs suggest

The first thing is to make the Visual Studio 2008 DTS package run in 32 bit mode in the designer. This can be done by:

Goto the properties of the SSIS project
In the left tree view select Configuration Properties
Set Run64BitRuntime = false

This enables you to debug the SSIS package. The next step is to execute the package in 32 bit. The easiest approach is to use

DTExec /f "D:\MyPath\MySSISPackage.dtsx" /SET \package.Variables[Variable1].Value;"Value1" /SET \package.Variables[Variable2].Value;"Value2" /X86

This works well if the total length of the command line is less than 255 characters. In my case I had a lot of parameters.

To get arround this 255 caharacter limitation I think the best solution is to make a command line executable that is compiled in 32 bit. I thought about making a power shell command for this but decided it would be simpler to keep it to a command line and search the standard output for "Success" or "Failure".

To get this to work I included the following DLLs:

D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SqlServer.ConnectionInfo.dll

D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SqlServer.Dts.Design.dll

D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SqlServer.DTSPipelineWrap.dll

D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SQLServer.DTSRuntimeWrap.dll

D:\Program Files (x86)\Microsoft SQL Server\100\SDK\Assemblies\Microsoft.SQLServer.ManagedDTS.dll

The code looked like:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Xml;
using Microsoft.SqlServer.Dts.Runtime;
using System.IO;
using System.Threading;

namespace Ssis32BitExec
{
class Program
{
static void Main(string[] args)
{

if (args.Count() != 2)
{
Console.WriteLine("Ssis32BitExec ");
Console.WriteLine("===========================================");
Console.WriteLine("");
Console.WriteLine("Utility to execute an SSIS package in 32 bit mode thus enabling 32 bit drivers");
Console.WriteLine("that may not be available in 64 bit");
Console.WriteLine("");
Console.WriteLine("");

Console.WriteLine(" UNC path to .dtsx file (SSIS package)");
Console.WriteLine(" Path to parameter name and value pair file (This file");
Console.WriteLine(" contains lines like ParameterName;ParameterValue)");

Console.ReadLine();
Environment.Exit(0);
}

string packagePath = args[0];
string parameterValuePairsPath = args[1];

Console.WriteLine("SizeOf IntPtr is: {0}", IntPtr.Size);

if (!File.Exists(packagePath))
{
Console.WriteLine("Failed: The following file does not exist {0}", packagePath);
Environment.Exit(0);
}

if (!File.Exists(parameterValuePairsPath))
{
Console.WriteLine("Failed: The following file does not exist {0}", parameterValuePairsPath);
Environment.Exit(0);
}


XmlDocument packageDoc = new XmlDocument();
packageDoc.Load(packagePath);

Package package = new Package();
package.LoadFromXML(packageDoc.DocumentElement, null);


string[] lines= File.ReadAllLines(parameterValuePairsPath);
string[] Fields;
foreach(string line in lines)
{
Fields = line.Split(';');
if (Fields.Count() != 2)
{
Console.WriteLine("Failed: The following line is not correctly formated {0}", line);
Thread.CurrentThread.Abort();
}
if (package.Variables.Contains(Fields[0]))
{
package.Variables[Fields[0]].Value=Fields[1];
}
}

//This is a fudge factor that allows a badly constructed package to throw some errors and run through
//package.MaximumErrorCount = 2;


DTSExecResult result = package.Execute();

if (result == DTSExecResult.Success)
{
Console.WriteLine("Success");
Environment.Exit(0);

}
else
{

Console.WriteLine("Failed");
for (int i = 0; i < package.Errors.Count; i++) { Console.WriteLine(package.Errors[i].Description); } Environment.Exit(1); } } } }

To know for sure that this is running in 32 bit the Console.WriteLine("SizeOf IntPtr is: {0}", IntPtr.Size); equals 4 for 32 bit and 8 for 64 bit.

I call this from within VB.NET in:


Dim TempFilePath As String = IO.Path.GetTempFileName
Dim Sw As New StreamWriter(TempFilePath)
Sw.WriteLine("Param1;{0}", Value1)
Sw.WriteLine("Param2;{0}", Value2)
Sw.Close()

Dim SSIS32BitProcess As New ProcessStartInfo()
SSIS32BitProcess.WorkingDirectory = Environment.CurrentDirectory
SSIS32BitProcess.FileName = "Ssis32BitExec.exe"
SSIS32BitProcess.UseShellExecute = False
SSIS32BitProcess.RedirectStandardOutput = True
SSIS32BitProcess.RedirectStandardError = True
SSIS32BitProcess.Arguments = packagePath + " " + TempFilePath

Dim Proc As Process = Process.Start(SSIS32BitProcess)

Dim Out As String = Proc.StandardOutput.ReadToEnd()
Proc.WaitForExit()

Dim Err As String = Proc.StandardError.ReadToEnd()
Proc.WaitForExit()

If Out.Contains("Success") Then
Return True
Else
Message = Out + vbCr + Err
Return False
End If
File.Delete(TempFilePath)
Return True

How to version control databases using TFS and Visual Studio Database Projects

2011-12-05T02:30:00.000-08:00

We have some databases that contain certain reference data. We would like to keep track of changes to both the schema and the reference data and store both in one central place. Another challenge was to allow everyone to develop on a local version of the database thus avoiding colliding and breaking changes. This was solved in the following way:

1. Open Visual Studio 2010
2. File/New Project
3. Go to the tree view of the New Project Dialog select
Database/SQL Server
4. In the list select SQL Server 2008 Database Project
- Choose an appropriate Solution Name
- The Name should be the same as the database you are putting into source control
5. [OK]
6. Right mouse click on the database project and select "Import Database Objects and Settings"
7. Click New Connection
8. Server Name = (local)
Database Name = the local database you are trying to version
[OK]
9. Select the following check boxs:
Script the collation only if it is different from the database collation
Import extended properties
10. [Start]

If you have multiple instances of SQL Server (eg 2005, 2008 etc) Then you need to create an Alias for the database instance that you are targeting

Aliases are created by:
1. Open Sql Server Configuration Manager
2. Click on the SQL Server Network Configuration and enable Named Pipes
3. Click on all versions of SQL Native Cient 10.0 Configuration
Add a new alias
- Alias Name: MyDB_LOCAL
- Pipe Name: \\localhost\pipe\MSSQL2008R2
- Protocol: Named Pipes
- Server: localhost\SQL2008R2
4. Repeat 3 for SQL Native Cient 10.0 Configuration (32bit)

The Database may be documented using extended properties

Extended properties are access by
1. Open the Microsoft SQL Server Management Studio ...
2. Right Mouse click and select properties
3. In the tree view select properties
4. Add the combination of Name and Value
Extended properties are available on the database level down to the table field level

After this you have imported the database schema into a Visual Studio Database project. The next step is to insert data. Depending on what you want to achieve there are several mechanisms for importing data.

The PostDeployment script is intended to be used for populating reference data. This can be found in the following way:
In the VS2010 database projects under the solution directory Scripts\PostDeployment\Script.PostDeployment.sql
In the file system under DatabaseProject\Scripts\Post-Deployment

There are several possibilities for inserting reference data:

1. If it's small you can use INSERT INTO eg
INSERT INTO [MyDatabase].[dbo].[DATABASE_VERSION] ([DATE],[MODEL_VERSION],[DATABASE_VERSION],[COMMENTS])
VALUES ('2011-01-07','4.1.0','1.0.0','A comment about what is new in this version')
GO

2. If it's big and in multiple tables any you are not looking to compare changes in reference data you can include a bakup of a database.

Then the script would look something like:

USE [master]
go

if (exists(select * from sys.databases where name=N'Testdata') )
begin
EXEC msdb.dbo.sp_delete_database_backuphistory @database_name = N'Testdata'

ALTER DATABASE [Testdata] SET SINGLE_USER WITH ROLLBACK IMMEDIATE

DROP DATABASE [Testdata]
end
go

-- Note that SolutionRoot is an environmental variable that can either be set in the control panel or via a batch file that opens the database project

RESTORE DATABASE [Testdata] FROM DISK = N'$(SolutionRoot)\MyDatabase\Scripts\Post-Deployment\Testdata.bak' WITH FILE = 1, MOVE N'Testdata' TO N'$(DefaultSqlFileLocation)\Testdata.mdf', MOVE N'Testdata_log' TO N'$(DefaultSqlFileLocation)\Testdata_1.ldf', NOUNLOAD, REPLACE, STATS = 10
GO

INSERT [MyDatabase].[dbo].[StudentList] (StFName,StLName,StEmail,OrderID)
SELECT StFName,StLName,StEmail,OrderID FROM [Testdata].[dbo].[StudentList]

3. Here is another method using BCP and a binary file.
This can be useful when you don't want the inconvenience of creating a backup etc.

To create the bcp file (adjust the table in the from-clause, and possibly the server after the -S param):
bcp "select field1,field2,field3 from MyDatabase.dbo.MyTable order by Id asc" queryout MyTable.bcp -S localhost -T -E -N

The script looks something like:

:setvar EstimatedRowCount 5000000

ALTER TABLE MyDatabase.dbo.MyTable NOCHECK CONSTRAINT ALL

bulk insert MyDatabase.dbo.MyTable from '$(SolutionRoot)\MyDatabase\Scripts\Post-Deployment\MyTable.bcp' with (DATAFILETYPE = 'widenative', order(ModelEventId asc), keepidentity, KEEPNULLS, ROWS_PER_BATCH = $(EstimatedRowCount))

ALTER TABLE MyDatabase.dbo.MyTable WITH CHECK CHECK CONSTRAINT ALL

4. Here is another method using BCP and a text file.
With this it is possible to compare versions of reference data directly out of TFS
To create the text file

i. Right mouse click on the database name and select All Tasks/Export Data
ii. Select Next and then in the Choose a source dialog select
- Database source : SQL Server Native Client 10.0
- Server Name : The Server Name
- Database : The database
[Next]
iii In the Choose a destination dialog choose:
- Destination : Flat File Destination
- File name : path to tablename.txt
- Locale : English (United states)
- Code Page 1252 (NASI Latin1)
- Unicode is not checked
- Format : Delimted
- Text qualifier :
- Column names in the first data row is not checked
Next
iv - Copy data from one or more tables or views
Next
v . Source taböle or view : the table to be exported
- Row delimiter : {CR}{LF}
- Column delimiter : Semicolon {;}
Next
vi - Run immediately
Finish
Finish

Then to import script looks like:

BULK INSERT MyTable FROM '$(TFSROOT)\DataBases\MyDatabase\Scripts\Post-Deployment\MyTable.txt' WITH (FIELDTERMINATOR = ';', ROWTERMINATOR = '\n' )

So now we have a Visual Studio 2010 Database project and data. Now suppose that you have a view that references another table in another database. You then create a sub database project within the database solution but when you want to deploy the database you get an error like

Error 2 SQL03006: View: [dbo].[MyView] has an unresolved reference to object [MyOtherDatabase].[dbo].[MyOtherTable]. D:\SOURCE\DataBases\MyDatabase\Schema Objects\Schemas\dbo\Views\MyView.view.sql 12 23 MyDatabase

Looking at the SQL
CREATE VIEW dbo.MyView
AS
SELECT dbo.MyTable.field1, MyOtherDatabase.dbo.MyOtherTable.Field
FROM dbo.MyTable INNER JOIN
MyOtherDatabase.dbo.MyOtherTable ON dbo.MyTable.Id = MyDatabase.dbo.MyTable.FKId

The problem is MyOtherDatabase.dbo.MyOtherTable.Field to a field in a table in another database. This is causing the deployment to fail. The solution to this can be found in an article on how to defining Cross-Database References at http://msdn.microsoft.com/en-us/library/bb386242.aspx
When adding the database reference you must define a database variable.
Fill out the Add Database Reference as follows:
- Database projects in the current solution: YourRefDB
- Database Reference Variables
- Name
- Uncheck Literal
Name $(IO_DB) Value YourRefDB
- Check Update the existing schema object definitions and scripts to use the database variables
- Check Suppress errors caused by unresolved references in the reference project
[OK]
Then you get to an opportunity to edit the modifications to the scripts
This resolves the compilation errors.

The next problem comes when we want to change the Deployment function from generating a script to deploying the script
Going to the solution view and looking at the properties of the database project and changing the Deploy action to "Create a deployment script (.sql) and deploy to database

Message 1 The deployment script was generated, but was not deployed. You can change the deploy action on the Deploy tab of the project properties. D:\SOURCE\DataBases\DBProj\DBName\sql\debug\DbName.sql 0 0 DBName

The solution to this is:
1 Right mouse click on the database project and select properties
2 Goto the Deployment tab
3 Edit Target connection
Server :(local)
Database name :MyDatabase

After this the deployment of the database project works…

I found some other curious effects. If you try and open this database project on a server where you have uninstalled SQL2008 R2 and installed SQL2008 SP3 you will get the following error

D:\SOURCE\DataBases\MyDBProject\DBName\DBName.dbproj : error : Could not load file or assembly 'Microsoft.SqlServer.Management.SqlParser, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.

This problem is caused because the uninstall process removed some assemblies needed by Visual Studio to correctly interpret the database project. There are 2 ways to solve this. Either re-install VS2010 or run the following commands

Open a command line and goto: D:\Software\VS2010\DVD\WCU\SMO
Then execute
msiexec /i SQLSysClrTypes_amd64_enu.msi
msiexec /i SharedManagementObjects_amd64_enu.msi

Next change dir to D:\Software\VS2010\DVD\WCU\DAC and execute:
msiexec /i DACFramework_enu.msi
msiexec /i DACProjectSystemSetup_enu.msi
msiexec /i TSqlLanguageService_enu.msi

Another interesting thing happens when the MS SQL2008 Server has been setup with different paths.
To fix this problem go to the database project and then open the following solution folder path Schema Objects/Database Level Objects/Storage/Files. Here you will find 2 files with names like:

MyDatabase.sqlfile.sql
MyDatabase_log.sqlfile.sql

Within these files you have

ALTER DATABASE [$(DatabaseName)]
ADD FILE (NAME = [MyDatabaseName], FILENAME = 'G:\Data\MyDatabaseName.mdf', FILEGROWTH = 1024 KB) TO FILEGROUP [PRIMARY];

To make the database project run on any setup you need to modify the path using the environmental variables DefaultDataPath and DefaultLogPath. For example:

ALTER DATABASE [$(DatabaseName)]
ADD FILE (NAME = [$(DatabaseName)], FILENAME = '$(DefaultDataPath)$(DatabaseName).mdf', FILEGROWTH = 1024 KB) TO FILEGROUP [PRIMARY];

By the way if you want to use environmental variables within a Query in MS SQL Server Manager you need to:

1. Goto the Query drop down menu
2. Select SQLCMD Mode
3. Cut and paist script....

Then you can use commands like

:setvar somevariable somevalue

Cross platform applications with HTML5/JavaScript and how this ties in with SOLID OO principles

2011-10-07T07:30:00.000-07:00

I have just started to think about writing applications that are cross platform impendent ie that can run on devices like iPad. Here are some notes about this subject
Over the last few years I have made extensive use of SOLID. OO programmin principles as I developed C# applications in .Net.

Where
SRP: The Single Responsibility Principle A class should have one, and only one, reason to change.
OCP: The Open Closed Principle You should be able to extend a classes behavior, without modifying it.
LSP: The Liskov Substitution Principle Derived classes must be substitutable for their base classes.
ISP: The Interface Segregation Principle Make fine grained interfaces that are client specific.
DIP: The Dependency Inversion Principle Depend on abstractions, not on concretions.

My first reaction was to reapply these principles in JavaScript. So I started to think of interfaces, dependency injection containers, mocking frameworks and Test Driven development. It turns out that there is a fundamental difference between C# and JavaScript in the sence that c# is class based and JavaScript is prototypical. So it is difficult to apply the notion of interfaces and abstraction in the world of JavaScript.

So the question is what principles are relevant to a Prototypical language over a Class based OO language. As far as I can see :

S Applicable
O Not Applicable
L Applicable
I Not Applicable
D Not Applicable

One principle that applys to both programming paradigms is DRY which stands for Don't Repeat Yourself

JavaScript Object Oriented Programming include classes, inheritance, and scope that can be used encapsulate, support namespaces, and avoid collisions. Actually JavaScript does not have a class entity BUT it implements the pattern of classes. The difference is in the inheritance model. In other object oriented languages, class is an actual data type that represents the blueprint for creating objects. In JavaScript, although we can use Functions to simulate an object blueprint, they are just in fact objects themselves. These objects are then used as models (aka prototypes) for other objects. Applying the concept of prototypal inheritance allows us to create “subclasses”, or objects that inherit the properties of another object. This becomes particularly useful when we want to use the methods of another object with some slight modifications.

The next thing I was thinking about is that after programming a while in Xaml I preffer to use the MVVM model over the MVC. This is in MVC the Controller is tightly coupled to the view. In practical terms this means that every time you need to change a control on the view the controller needs to change things like how this effects all the other controls. Where as in MVVM the View is completely separate from the VM and thus a change to a control results in minimal update to the ViewModel. It turns out that there is a JavaScript library called KnockOut.js that makes the MVVM pattern possible

Here are some examples of JavaScript. It can be seen that JavaScript is far from just a begineers language. JavaScript's prototypal OOP is flexible and it does not limit the programmer to things like static typing.

Here is an example of how to create your own foreach function:

Array.prototype.foreach.function(callback)
{
for (var i=0; i= companrand:}
"=" : function(comporand){return function(e){return e= companrand:}
}
}

a.set(op[">="](5).foreach(alert);

This is called Curring after Mr Curry.
btw If you uise this in DOS you will need toescape the > to ^>

----
Here is a way tobiuld up the arrays:
Number.protype.inputModule
{
UpTo : function(Upeer,step)
{
var a[]
for(var i = this.valueof(Upper); j return a;
}
}

(1).Upto(20,2).....

This is called High Order Programming

Here's a link to some further examples: http://www.w3schools.com/js/

The next version of Visual Studio will have a lot more support for JavaScript development, so this should reduce the barrier to trying this out...

Condensed notes from the Build about where Microsoft technology is going over the next year

2011-09-27T00:28:00.000-07:00

Here is a summary of what I learned at the Build. I went to a lot of trouble this year to attend this conference which was not easy considering that the agenda was empty up to the last minute. But it was certainly worth while because Microsoft reveled where it is going with its technology which included a unified user experience from mobile devices through to large television sized devices.

CNN “Microsoft unveils a radically redesigned Windows 8”

This technology will only be available in about a year from now, probably industry will adopt this in perhaps 2 years from now. This conference is important because no strategy would be complete without an idea of where Windows is going because it is our main operating system. It is necessary because our software engineers need to gather the skills needed to take advantage of this technology and it is necessary in order to make informed decisions over if and how to integrate other technology such as iPhone and iPad.

Windows 8 Metro Style

We can expect a shift towards touch base applications over the coming years. Microsoft is of the opinion that all screens will be touched enabled and the ones that are not will feel antiquated when used. To make use of the Windows 8 metro style the user will have to adapt by learning new gestures and become familiar with the concept of having a less cluttered desktop using active Tiles. This means a “Re imagine your application” when making an application in Metro style. It was emphasized that there are a number of applications where the chrome style is simply the most appropriate meaning that this style of UI will continued to be supported. The difference between metro and chrome will go along the lines of the difference between a DOS prompt and Windows ie an expert system that requires training to a intuitive system that requires minimal training. Chrome seems to be based on the philosophy that “Less is more”.

Performance and energy saving has become a key factor in the design of Windows 8. A cold start up takes about 20 seconds. To make this possible the number of running processes has been minimized and that tasks that are not currently in view are suspended. This has an impact on the design and development of applications because they only have a few seconds to persist there state before the task is suspended. It is now possible to boot a Windows 8 client directly from a USB memory stick. Also the first bleeding edge problems with VS are coming to light eg

http://blog.galasoft.ch/archive/2011/09/25/quick-tip-killing-a-metro-style-app-in-windows-8.aspx?utm_source=twitterfeed&utm_medium=twitter&utm_campaign=Feed%3A+galasoft+%28Laurent+Bugnion+%28GalaSoft%29%29

There are some features that may take more time to be adopted by the industry. Such as location aware applications that can make use of local devices or the touch sensor that perform a kind of electronic hand shake and setup a link between the 2 devices for collaboration or simply to exchange an electronic business card.

The Windows 8 user experience is based on a fast and fluid immersive and full screen user experience. Applications communicate with each other through “charms” that are implemented via shared contracts. The same intuitive interface will be used across all Windows devices from mobile devices through desk tops and televisions.

The result is a non cluttered canvas with controls. Commands that are frequently used go on the canvas, all others are revealed through the edge gestures. The metro style desktop consists of Tiles that provide simple text or images via pre defined templates and come in 2 sizes. Secondary tiles are created by pinning the content of the application and have the same capabilities as the main tile. They provide a deep link within the application. Live tiles keep people connected to your app and male it more likely the application is put on the first page.

Notifications appear via a toaster icon that appears for a short period of time and user must opt onto it. They use Windows Push Notification Service WNS and can update any time your application is running

There is a new layer that sits directly on the Windows Kernal called WinRT. This is a clean API that has no duplication of Runtime API and all obsolete or inapplicable APIs have been removed. Metro style apps for c# and VB.net can interact directly with WinRT APIs or via .Net for Metro style apps via the CLS and Win32 APIs.

To publish a metro app there is a pipeline of checks that guarantee that an application quality. This consists of Pre Processing, Security tests, Technical Compliance, Content compliance, Signing and publishing.

x86, 64, and ARM processors will all be supported. This has a big positive implication that all mobile devices running on ARM processors will support these applications.

No solid time line was given. Some guesses included perhaps it being available about this time next year. The next milestone is Beta, the RC, then RTM and then GA

Windows 8 Server

I did not attend many Windows 8 server sessions. But a lot of progress has been made on performance. I also believe that the HyperV virtualization has been improved by lessons learned in Azure. This is a way to make low cost hard disks available in a SAN. This can perform very fast because it is possible to make use of more than 1 network card through teaming. It is also possible to change a system drive on the fly.
.Net 4.5

In the key note it is really sure that Silverlight is definitely not dead. Microsoft has been working on making it possible to program the UI in languages programmers might choose. This includes the HTML5/Java script as well as C++. Microsoft is in the process of making a metro style implementation of office. From what I could determine this is going to be made using HTML5 for the view part of the presentation layer and Java script as the model art of the view model. All this sits on WinRT APIs which sits directly on Window Kernel Services. At first site this has been misinterpreted last year as a move away from Silverlight. This is not the case because the idea is to allow the programmer to express themselves in any appropriate programming language and this was demonstrated by including demonstrations in C and C++. I asked some experts whether the HTML5/JavaScript implementation would use the MVVM pattern. The answer was that this question had been asked by a lot of people and that it should be possible but there are no examples for this yet.

Entity framework EF 4.5 now includes Enums, SQL Server and Azure features such as spatial functions
In .Net 4 data manipulation was made by first starting with the data and then setting up the computation. TPL Dataflow allows the computation to be first setup and then the data

Other Parallel Computing Additions

Combinators Task.WhenAll, Task.WhenAny
Timer integration Task.Delay(TimeSpan), CancellationTokenSource.CancelAfter(TimeSpan)
Task schedulingConcurrentExclusiveSchedulerPair
Fine-grained control DenyChildAttach, HideScheduler, LazyCancellation, EnumerablePartitionerOptions
ThreadLocal.Values
PERFORMANCE (“it’s just faster”)
The garbage collector has been improved. A multi-cored JIT with pre-fetch options makes ASP.Net start 35% faster.

.Net 4.5 is an in place upgrade from .Net 4.0

VS2011

Firstly VS2011 can open VS2010 project without changing the format.

There is a 3D editor that has the capability of breaking down each transformation that is made to a 3D object. It is not designed for creating these objects but can manipulate them. Improvements have been made on the IDE experience for C++ programmers. This includes unit tests, and color intellisence pickers etc.

Visual Studio 11 is the tool for Windows 8 and supports VB.NET, C#, C++ & HTML5/JS. There is a new designer Available today built on shared architecture with Expression Blend. When shelving changes information over which windows are open are saved together with the shelf set and are re established when opening the shelve set the windows are reopened in their original position. There is also a document well that avoids each time a file is clicked upon that a new document window is created. There is a XAML editor with IntelliSense. It is now possible to use C# code directly in Jscript and have changes in the generated HTML fed back into the code behind files.

There is a static code analysis feature that allows you to look and replace chunks of repeated code.
PowerPoint templates enable the mocking of an interactive UI.

Asynchronous programming is necessary for creating responsive fast and fluent applications. Here are some new features coming with .Net 4.5

Asynchronous programming models
Windows Runtime: IAsyncOperation
.NET Framework: Task
JavaScript: Promises
All are objects representing “ongoing operations”
All use callbacks to signal completion of operation
Challenge: Callbacks turn your code inside out
Insight: Automatic transformation to callbacks is possible

Asynchronous methods automatically transform normal code into a callback state machine eg

public async Task GetXmlAsync(string url) {
var client = new HttpClient();
var response = await client.GetAsync(url);
var text = response.Content.ReadAsString();
return XElement.Parse(text);
}

.Net 5.0 aka Project Roslyn

The .Net 5.0 JIT will be written in C#. This means that it will be possible to call APIs to the pipeline used to compile code. This pipeline comprises of 4 steps

1. Parser
2. Symbols / Meta data export
3. Binder
4. Emitter

Here’s an example

ScriptEngine engine = new ScriptEngine();
Session session = Session.Create();
Engine.Execute(“using System;, session);
Engine.Execute(“for(int I = 0; I < 10; i++) Console.WriteLine(i*i);”,session); Var f = (funcengine.Execute(“new Func(Sqr)”);
For (int i=0; i<10 i++) Console.WriteLine(f(i)); This might look strange but it could be used in allowing users to use a macro style interface to your application. In my case I doubt that I would use this but it is very interesting. I could imagine that this will allow mixing of languages that could be useful in creating fast code. There is also a command line Roslyn C# Compiler that runs in a similar style as for F# and includes intellisence and code tips. To create a reference to another assembly “er” is used, eg: Er “PresentationCore” Here’s an example of translating c# into VB Public VB.SytaxNode Covert( CS.SyntaxTree syntaxTree, IDictionary identifierMap = null,
Bool convertStrings =false)
{
Var text = syntaxTree.Text;
Var mode = syntaxTree.Root;
Var vbText = Convert(text, node, identifierMap, convertStrings);
Return VB.SyntaxTree.ParseCompilationUnit(vbText).Root;
}

There is also a feature for .Net 3.5 on demand that I believe means that we don’t need to install the .Net 3.5 framework when certain obsolete functions are required.

.Net 5 will be a fresh installation

HPC

I have met up with some HPC / Azure MVPs and discussed some of the problems that I encountered when making my cat model in the cloud. The problem was that I was not able to transfer a 60 GB VM into the cloud including a local SQL Server. The suggestion was to use SQL Azure and transfer data via blob storage. In the next couple of months there will be a release of HPC Server that runs of a standard worker role meaning that we don’t need to construct and maintain our own VM. For very large quantities of data we could consider speeding up the queries by using TriadLinq aka Linq to HPC. This works by producing a sealed data block that is replicated between multiple nodes and is then queried upon in parallel. In addition to the Windows server based HPC Scheduler there is an Azure based HPC Scheduler. These are useful because it means that we don’t need to write our own schedulers.

There was a demo of a PC with 4 water cooled graphics cards that had 2500 times the processing power of a Cray. The HPC team uses these machines for numerical tasks. Currently it takes a crack programmer to correctly program such an algorithm, the hope is that this will become abstracted away at some point in the future.

Azure

Azure is a very cheap, elastic and quickly configurable source of compute power and one that we really should take advantage of. I saw some presentations on debugging and branch caching.

ALM Application Life Cycle Management

Attendees of Build where given a golden ticket to an evaluation version of the SaaS version of Team Foundation Server 2010. This contains a many of Application Lifecyle Management features that are all a part of professional software engineering. The Application life cycle is based on 2 parts . The Development cycle where a product backlog is worked upon during sprints to create a working software asset. The Operations which consists of an Ops back log and monitoring. When a bug is discovered a corresponding requirement is made that is sent back to the development backlog. In addition to the existing ALM functions a Code review function was demonstrated in the context of a development team with inline or side by side code comparison

Currently both TFS Server and TFS SaaS are capable of:
Work items, Source Control and Build
Agile Product/Project Management
Test Case Management
Heterogeneous Development

TFS SaaS has the advantage when:
Near Zero setup and administration
Collaborate with anyone from anywhere

TFS Server has the advantage when
Virtual Test Lab Management
SharePoint Integration
Data Warehouse and Reporting

User experience in an L39 Jet

2011-09-22T14:57:00.000-07:00

This has got to be the ultimate user experience. I am sitting behind the pilot in this Russian designed L39 Jet. If anyone would like to try it is quite expensive but is something you will never forget. Here are some links:

http://www.incredible-adventures.com/l-39-flights.html

http://www.mach1aviation.com/varied/l-39-safety-video.htm

My pilot was Dave Riggs who is a hollywood stunt pilot for films like XXX, Iron Man, Lord of War. We flew together with David LaFaille and did some combat flying and some low level flight avoiding radar in canion, by low level I mean we flew 4m of the ground. My stomach had a problem digesting this new user experience so we finished the day with some formation flying.

First Impression of the Build

2011-09-15T06:19:00.000-07:00

Just before the build started I had a chance to talk to a user experience MVP and to a company that has made some compelling applications for Health care and emergency services. This was quite fortunate because the majority of this conference is all about a revamp of the way that the user experiences Windows. Basically over the coming years there will be a shift towards touch based applications. The vision is that all screens will be touched enabled and the ones that are not will feel antiquated when used.

It was very interesting talking to Christian Moser before the conference. He is writing a book on User Experience where controls are represented by a kind of design pattern. For a control to be successful it’s use must be intuitive which means that it’s use should be common knowledge. This means that you should think twice before inventing a new way of interaction. In the context of Windows 8 this is going to be interesting because there are a new gestures that have been invented. Also the concept of having a less cluttered desktop with active tiles has an impact on the design of applications. I heard a lot the buzz word of “Re imagine your application”. There are a lot of sessions around this Metro style. It was also emphasized that for some applications the chrome style is simply the most appropriate UI and that this will be continued to be supported. While I was talking to Christian he mentioned that in his book he will be describing some aspects of expert users. Before the build the example on an expert user was the airline booking system which is a command line interface that requires a lot of training but is super efficient. I could imagine that the difference between metro and chrome would go down the same lines. Chrome seems to be based on the philosophy that “Less is more”, according to my friends at BlackMarble most users really appreciate something like this.

In the key note it is really sure that Silverlight is definitely not dead. In fact MS has been working on making it possible to program the UI in languages programmers might choose. This includes the HTML5/Java script as well as C++. Performance and energy saving has become a key factor in the design of Windows 8.

Before the conference I was talking to some friends at Black Marble about how an application can determine where it is. For example in a hospital application your device should know what other devices are nearby. This can be done in several ways. One way is to use the SID of the wireless network. This works quite well but there can be problems of leakage of signal from one zone to another. Another method that could be used is GPS repeaters. This is essentially like setting up a GPS satellite within your building. This could be particularly interesting for emergency services in conjunction with some mapping on a Surface 2 system to keep track of where there fire fighters are etc.

Part of the Windows 8 slate included a touch sensor. The concept is that two people are using an application and what to collaborate. So they physically touch there slates and an electronic handshake occurs that enables further communication over Bluetooth etc. This also works with sensors which I believe work by induction and need no battery. Another example is a business card that when touched opens a web browser to a website. Another example starts an application that if not present will install it first then run it.

There is really a lot going on. I have met up with some HPC / Azure MVPs and discussed some of the problems that I encounted when making my cat model in the cload. Basically my problem was that I was not able to transfer a 60 GB VM into the cloud including a local SQL Server. The suggestion was to use SQL Azure and transfer data via blob storage. Also to think about TriadLinq which is a way to spread out queries onto multiple VMs. The thing is Azure is simply way cheaper than using in house servers. So I will attend some sessions that will look at these in a little more detail. On the Slate PC is a pre beta version of Windows 8 and Visual Studio 2011. I have also got an evaluation version of the SaaS version of Team Foundation Server 2010 which contains a lot of Application Lifecyle Management features that are all a part of professional software engineering.

Some notes on the application of Cloud computing

2011-07-20T22:45:00.000-07:00

I am just about to go on holiday. So before I forget here are ideas about the application of Cloud Computing to Catastrophe Modeling.

Cloud computing makes sense for Cat simulations for various reasons. The costs are very reasonable, for example all my tests came to about 82 CHF compared to the fixed cost of a VM in a datacenter that mounts up to several thousand franks. Microsoft can achieve this by making the administration of the hundreds of thousands of VMs as automated as possible thus getting an economy of scale that we don't have in smaller data centers. Creating the cloud VMs took a few minutes as opposed to weeks via our internal/external processes. I could imagine that WMWare will improve the provisioning of virtual machines and that one day this may be as simple as filling out a web site. But we don't have such a system yet and if we did we would have a much more limited pool of servers and therefore higher costs than if the whole thing was in the cloud. One last aspect was that surprisingly the end to end time to process the bench mark cat model in the cloud took less time than on our on-premis servers.

There are some different possibilities how number crunching processes can be implemented. I have seen solutions that can call an executable, if this executable needs something to be installed on the machine it is possible to configure setup tasks that install software as the VM is being instantiated. In the context of my modeling platform I think I would implement the job submission in the same way as in the prototype I described earlier. By this I mean I would transfer the data to be simulated as a blob and once completed add a reference to this data in a Queue that the work roles pole for work. The difference would be instead of poling the results queue I would expose an on premise WCF service over http and call this from the cload using claims based tokens. I would use Queues for status information because they have an intrinsic order, but I would send this information via one way calls to the on premise web services. Since security is made using claims based security the role of the firewall changes slightly. The reason is that cloud apps need to connect via http or https to internal services and that the services and not the firewall will carry out authentication. So there is a shift of responsibility from the firewall the services

From my experience of uploading custom VMs into the cloud I am not sure how well HPC with burst into Azure works in practice. I will follow up on this at the Build conference.

Thinking about how Cloud computing can be integrated into an organization there are 3 aspects to consider Network, Storage and Compute.

A means of synchronizing data between office and mobiles devices brings substantial benefits. Some years ago I tried a CTP of Microsoft Mesh which enabled the synchronization of data between devices. Apple recently released a similar cloud service. There was a demo at the PDC which showed a business man losing his PC but because all his configuration and working data is continuously in synch it was possible to take a new PC and carry on where he left off. Since the CTP I have seen a version of Mesh in Hotmail and Office 360. Although it is difficult to come up with a single business case where this is useful it does add to productivity because it improves the functionality of the environment that we work in.

On the network idea there are datacenters around the world making a global presence much easier to maintain. On the other hand there is latency in getting data to and from the cloud. This makes the SaaS Team Foundation Server offering very interesting.

I think cloud computing will bring about a shift from relational databases to a more object orientated data storage model. Relational databases are not intrinsically more performant. At present I believe the performance of properly dimensioned on premise sql servers out-perform the cloud sql servers. This will probably change SQL Azure developes and as technologies like TriadLinq become available enabling the distribution of the workload needed in carrying out database queries. The reason is that a central sql server is a bottle neck, where as in a cloud the compute needed to carry out queries is by design made for scalability.

With this in mind I think it's worthwhile to encourage the development of web based apps or apps that can be deployed from the web because these fit easily into the Cloud. We would need to manage the scope of the broad set of mobile devices that may or may not be supported in our environment. In particular Rapid Application Development in the context of Microsoft orientated companies which would favor Windows mobile devices even though these devices have not been on the market long enough to really take a large market share. To develop iPad and iPhone applications means needing an Apple workstation learning C and a whole new API.

When looking at data It is difficult to categorize data into security groups. The reason is that depending on the application the data either be sensitive or non sensitive depending on it's context. It is therefore more practical to make security groups for applications

Parallel.For seems to have a large overhead compared to a simpler implementation

2011-06-09T09:02:00.001-07:00

I implemented the following alternative to the parallel for using threads:

        static void MyParallelFor(int fromInclusive, int toExclusive, Action<int> body)
        {
            int numProcs = Environment.ProcessorCount;
            using (CountdownEvent ce = new CountdownEvent(numProcs))
            {
              int rangeSize = (toExclusive - fromInclusive) / numProcs;
              for (int p = 0; p< numProcs; p++)
              {
                  int start = rangeSize * p;
                 //int end = start + rangeSize;
                 int end = p == numProcs-1 ? toExclusive : start + rangeSize;
                 ThreadPool.QueueUserWorkItem(delegate
                   {
                      for (int i = start; i < end; i++) body(i);
                      ce.Signal();
                    });
              }
              ce.Wait();
            }
        }

I then tested this on the benchmark earthquake model as described in my last blog entry. Since the signature of the parallel for and myParallelFor are identical the comparison can be made by commenting out the implementation that is not under test.

            for (i = 1; i <= ns; i++) // Outer loop going through lots of lines of exposure
            {
                loss[i] = 0;

                    //Parallel.For(1, nr,
                    MyParallelFor(1, nr, // Inner loop going through many events
                         (jj) =>
                         {
                            // Some maths to determine the mean damage ratio (see previous blog entry for details)
                            localsum[jj] = mdr * value[jj];
                         }

                         });
                 loss[i] = localsum.Sum();
            }

I then ran the above code on a physical machine with ProLiant DL580 G5 12 Cores 2.66 GHz Intel Xeon(R) X7460 with 8188 MB RAM

Parallel.For took about 114 seconds with all 12 CPUs running evenly at about 70%.
MyParallelFor took about 71 seconds with 11 CPUs at 30% and 1 CPU at 80%,

I found this difference quite surprising as it seems that the parallel.for implementation in System.Threading.Tasks seems to have a very large overhead. Not only does it take longer but the cpu’s are working harder. Using reflector I could see that the implementation of private static ParallelLoopResult ForWorker was quite extensive. When I have some time I will add some buffering in the localsum to check whether the problem is caused by cache invalidation.

Results of runing a simplified earthquake model in the cloud

2011-05-27T11:03:00.001-07:00

Here are some preliminary results of how long it takes to upload and run 10000 iterations of a cat model under various worker role configurations

Extra Large with 8 cores took 26s
Large with 4 cores took 34s
Medium with 2 cores took 49s
Small with 1 core took 82s

For comparison here are some run times on a D20 desktop

VB.NET on a D20 compiled with VS2010 running on 1 thread took 104s
F90 on a D20 compiled with Intel running on 1 thread took 162s
C++ on a D20 compiled with Intel with optimization on 1 thread took 58 s

The time needed to upload exposure and events took about 100s Within the cloud it took about 1s to download from blob storage

The initial flat file text files had the following sizes and number of points

Exposure file 7.3MB 385560 Points
Event Catalogue 4.8MB 194510 Events

To reduce the run times we reduced the number of events from 194510 to 10000. So this means the actual run times are 20 times longer.

I have not received the bill for these tests but I think it is probably quite cheap. I think there is a lot to be said for cloud computing because I only had to worry about my code, the infrastructure was completely abstracted away. In fact setting up a new node takes about 8 minutes with a very friendly billing terms that you pay only for the time that the role is active.

Numerical calculations in Azure continued

2011-05-27T06:04:00.001-07:00

After some help from Steve Spencer I figured out the best way to debug the WorkerRole was not to do the whole thing in unit tests but to pressed the play button on the WorkerRole and use a unit test to feed it with data. It turned out the problem was within the blob storage code. I still have some work to refactor the worker role but I want to get some results so I am going to leave it as it is for the moment:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.Diagnostics;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;
using AzureHelper;
using System.Configuration;
using System.IO;
using System.Text;
using CatModel;
using Contracts;
using Entities;
using Infrastructure;

namespace WorkerRole1
{
    public class WorkerRole : RoleEntryPoint
    {
        private String localPath;
        private String calcsimExePath;

private EQModel eQModel { get; set; }

        private DateTime Start { get; set; }
        private DateTime Finish { get; set; }
        private TimeSpan Stopwatch { get; set; }
        public ILog Log { set; get; }

        private AzQueueHandler jobInputQueue;
        private AzQueueHandler jobOutputQueue;
        private AzQueueHandler jobLogQueue;

private AzBlobHandler jobInputBlog;

        private AzBlobHandler dataStoreExposureX;
        private AzBlobHandler dataStoreExposureY;
        private AzBlobHandler dataStoreExposureV;

        private AzBlobHandler dataStoreCatalogSm;
        private AzBlobHandler dataStoreCatalogSx;
        private AzBlobHandler dataStoreCatalogSy;
        private AzBlobHandler dataStoreCatalogSd;

        private AzBlobHandler jobLossBlob;
        private AzBlobHandler jobLogBlob;
        private AzBlobHandler jobOutputBlog;

        private int idleCount;
        private int idleMax;
        private int idleSleepTime;

        public override bool OnStart()
        {
            // Set the maximum number of concurrent connections
            ServicePointManager.DefaultConnectionLimit = 12;

// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.

return base.OnStart();
}

        private void Init()
        {
            AppDomain.CurrentDomain.UnhandledException += new UnhandledExceptionEventHandler(CurrentDomain_UnhandledException);
            idleCount = 0; // how many times the instance have been idle
            idleMax = 10;   // After 10 times (5 minutes = 30 secs *10) being idle, then die
            idleSleepTime = 30 * 1000; // sleep for this number of seconds between each queue poll

var accountName = ConfigurationManager.AppSettings["AzureAccountName"];
var accountKey = ConfigurationManager.AppSettings["AzureAccountKey"];

            jobInputQueue = new AzQueueHandler(ConfigurationManager.AppSettings["JobInputQueue"], accountName, accountKey);
            jobOutputQueue = new AzQueueHandler(ConfigurationManager.AppSettings["JobOutputQueue"], accountName, accountKey);
            jobLogQueue = new AzQueueHandler(ConfigurationManager.AppSettings["JobLogQueue"], accountName, accountKey);
            jobInputBlog = new AzBlobHandler(ConfigurationManager.AppSettings["JobInputBlob"], accountName, accountKey);

            jobLogBlob = new AzBlobHandler(ConfigurationManager.AppSettings["jobLogBlob"], accountName, accountKey);
            dataStoreExposureX = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreExposureX"], accountName, accountKey);
            dataStoreExposureY = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreExposureY"], accountName, accountKey);
            dataStoreExposureV = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreExposureV"], accountName, accountKey);

            dataStoreCatalogSm = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreCatalogSm"], accountName, accountKey);
            dataStoreCatalogSx = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreCatalogSx"], accountName, accountKey);
            dataStoreCatalogSy = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreCatalogSy"], accountName, accountKey);
            dataStoreCatalogSd = new AzBlobHandler(ConfigurationManager.AppSettings["dataStoreCatalogSd"], accountName, accountKey);

jobLossBlob = new AzBlobHandler(ConfigurationManager.AppSettings["JobLossBlob"], accountName, accountKey);
jobOutputBlog = new AzBlobHandler(ConfigurationManager.AppSettings["JobOutputBlob"], accountName, accountKey);

eQModel = new EQModel();
this.Log = new Log("WorkerRole");

localPath = Environment.CurrentDirectory;

            // pull EXE file from blob storage to local file system
            calcsimExePath = System.IO.Path.Combine(localPath, "Startup\\calcsim.exe");
        }

        public override void Run()
        {
            // This is a sample worker implementation. Replace with your logic.
            Trace.WriteLine("WorkerRole1 entry point called", "Information");
            Init();

TraceInfo("AzJobHost::Run() Azure Instance ID {0}, DeploymentId {1}", RoleEnvironment.CurrentRoleInstance.Id, RoleEnvironment.DeploymentId);

bool more = true;
//Stopwatch swWrkRoleLifetime = Stopwatch.StartNew();

            // msg pump loop
            string id = ""; string popId = "";
            while (more)
            {
                id = ""; popId = "";
                string msg = jobInputQueue.GetMessage(ref id, ref popId, false);

                if (msg == null)
                {
                    idleCount++;
                    if (idleCount >= idleMax)
                        more = false;
                    else Thread.Sleep(idleSleepTime);
                }
                else
                {
                    ProcessMsg(id, popId, msg);
                }
            }

            //swWrkRoleLifetime.Stop();
            //TraceInfo("AzJobHost::Exit(). Execution time {0}", swWrkRoleLifetime.Elapsed);
        }

        private bool ProcessMsg(string id, string popId, string msg)
        {
            bool rc = true;

//Stopwatch sw = Stopwatch.StartNew();
TraceInfo("AzJobHost::ProcessMsg( '{0}', '{1}') - Azure Instance Id: {2}", id, msg, RoleEnvironment.CurrentRoleInstance.Id);

            List<double> x = new List<double>();
            List<double> y = new List<double>();
            List<double> v = new List<double>();
            List<double> sm = new List<double>();
            List<double> sx = new List<double>();
            List<double> sy = new List<double>();
            List<double> sd = new List<double>();

            Start = DateTime.Now;
            dataStoreExposureX.RecieveDataFromStorage<double>(msg, out x);
            dataStoreExposureY.RecieveDataFromStorage<double>(msg, out y);
            dataStoreExposureV.RecieveDataFromStorage<double>(msg, out v);
            Finish = DateTime.Now;
            Stopwatch = Finish.Subtract(Start);
            this.Log.LogMessage(String.Format("Time to Upload Exposure to model {0} milliseconds", Stopwatch.TotalMilliseconds), Stopwatch.TotalMilliseconds);

            Start = DateTime.Now;
            dataStoreCatalogSm.RecieveDataFromStorage<double>(msg, out sm);
            dataStoreCatalogSx.RecieveDataFromStorage<double>(msg, out sx);
            dataStoreCatalogSy.RecieveDataFromStorage<double>(msg, out sy);
            dataStoreCatalogSd.RecieveDataFromStorage<double>(msg, out sd);

            Finish = DateTime.Now;
            Stopwatch = Finish.Subtract(Start);
            this.Log.LogMessage(String.Format("Time to Upload Catalog to model {0} milliseconds", Stopwatch.TotalMilliseconds), Stopwatch.TotalMilliseconds);

List<double> losses = new List<double>();

            Start = DateTime.Now;
            eQModel.RunModel(x, y, v, sm, sx, sy, sd, out losses);
            Finish = DateTime.Now;
            Stopwatch = Finish.Subtract(Start);
            this.Log.LogMessage(String.Format("Time to Run Japan Earthquake {0} milliseconds", Stopwatch.TotalMilliseconds), Stopwatch.TotalMilliseconds);

this.Log.LogMessage("=== Appending Model Log to Server Log ====");

List<LogEvent> ModelLogs = eQModel.log.GetLogs();
this.Log.Add(ModelLogs);

jobLossBlob.SendDataToStorage<double>(msg, losses);
jobLogBlob.SendDataToStorage<LogEvent>(msg, this.Log.GetLogs());

jobOutputQueue.PutMessage(msg);

            jobInputQueue.DeleteMessage(id, popId);
            return rc;
        }

        private void TraceInfo(string format, params object[] args)
        {
            string msg = string.Format(format, args);
            Trace.WriteLine(msg, "Information");
        }

        private void RoleEnvironmentChanging(object sender, RoleEnvironmentChangingEventArgs e)
        {
            // If a configuration setting is changing
            if (e.Changes.Any(change => change is RoleEnvironmentConfigurationSettingChange))
            {
                // Set e.Cancel to true to restart this role instance
                e.Cancel = true;
            }
        }

        void CurrentDomain_UnhandledException(object sender, UnhandledExceptionEventArgs e)
        {
            TraceInfo("AzJobHost::UnhandledException: " + (Exception)e.ExceptionObject);
            RoleEnvironment.RequestRecycle();
        }

}
}

Numerical calculations in Azure

2011-05-26T12:15:00.001-07:00

Here are the steps that I took in looking at Azure and getting a feel for how we could use this. Recently we have made a number of bench mark tests with a simplified Japan earthquake model. This seems a natural place to start. The idea would be to utilize the compute power of the cloud to make earthquake simulations. Here some pseudo code as to what I intend to do

1.    The client loads exposure into cloud storage
2.    The client loads earthquake catalog into cloud storage
3.    The client adds a reference to the exposure and earthquake catalogue data to a queue
4.    The worker node listens on the input queue
5.    The worker node dequeues a reference to data for download
6.    The worker node downloads data from cloud storage
7.    The worker node processes the data
8.    The worker node saves loss data using a reference
9.    The worker node adds a reference to the loss data to a queue
10.    The client listens on the output queue
11.    The client dequeues the reference to result data
12.    The client downloads results and logs

Looking at this sequence the first thing I want to do is to be able to make CRUD operations on Queues and blobs. I found some good examples how this could be done, but I wanted to extend these examples to be more generic. Here's a handy generic function that sends lists of serializable objects to and from blob storage.

        public void RecieveDataFromStorage<T>(string key, out List<T> data)
        {
                CloudBlobContainer container = _blobClient.GetContainerReference(_blobContainer.Name);
                CloudBlob blob = container.GetBlobReference( key);
                byte[] bdata = blob.DownloadByteArray();

                MemoryStream f = new MemoryStream();
                BinaryFormatter sf = new BinaryFormatter();
                f.Write(bdata, 0, bdata.Length);
                f.Flush();
                List<T> target = new List<T>();
                f.Position = 0;

target = (List<T>)sf.Deserialize(f);
data = target;

return;
}

        public void SendDataToStorage<T>(string key, List<T> data)
        {
                CloudBlobContainer container = _blobClient.GetContainerReference(_blobContainer.Name);
                CloudBlob blob = container.GetBlobReference(key);

                MemoryStream f = new MemoryStream();
                BinaryFormatter sf = new BinaryFormatter();
                sf.Serialize(f, data);
                f.Position = 0;
                byte[] bdata = f.ToArray();

                blob.UploadByteArray(bdata);
            return;
        }

Where T can be any serializable entity. With these functions we have an easy way to transfer data too and from the cloud storage.

Next step is that we want to do some calculations. The simplified japan earthquake algorithm is single threaded. In order to make use of various number of processors we need to make this multi core enabled. Normally when making an algorithm multicore enabled you need to first really understand what is the algorithm trying to do. Then the next step is to think still in the problem domain how to split the work, fore example by data or by process. Then this is reviewed in the context of the hardware and technology available. Finally you would code applying patterns that do things like take account of cache invalidation etc. In our case we just want to get a feeling for what various CPU configurations can bring. Therefore I just made the inner loop parallel. The algorithm falls under the cataegory of "embarrasingly parrallizable code" because the loops are allmost completely independent from each other, only the line highlighted in bold has some cross dependancies. The trick is how to sum the losses. Below is the code before and after…

            for (i = 1; i <= ns; i++)
            {
                loss[i] = 0;

                if ((sm[i] > 3))
                {
                    dkrit = 0.5 * sm[i] * sm[i] * sm[i];

                    for (j = 1; j <= nr; j++)
                    {
                        rr[j] = 0.001 * Math.Sqrt((sx[i] - rx[j]) * (sx[i] - rx[j]) + (sy[i] - ry[j]) * (sy[i] - ry[j]));
                        rr[j] = Math.Sqrt(sd[i] * sd[i] + rr[j] * rr[j]);

                        if ((rr[j] < dkrit))
                        {
                            rlog[j] = Math.Log(rr[j]);
                            mmi[j] = 981 * Math.Exp(c1 + c2 * sm[i] + c3 * sm[i] * sm[i] + c4 * rlog[j] + c5 * rr[j]);
                            if ((mmi[j] > 50))
                            {
                                mmi[j] = 3.66 * Math.Log10(mmi[j]) - 1.66;
                                lmmi = Math.Log(mmi[j]);
                                mdr[j] = 0.01 * Math.Exp(v1 * lmmi * lmmi + v2 * lmmi + v3);
                                loss[i] = loss[i] + mdr[j] * value[j]; // <----------
                            }
                        }
                    }
                }

                   Parallel.For(1, nr,
                        (jj) => {
                                    //rr[jj] = 0.001 * Math.Sqrt((sx[i] - rx[jj]) * (sx[i] - rx[jj]) + (sy[i] - ry[jj]) * (sy[i] - ry[jj]));
                                    double rr = 0.001 * Math.Sqrt((sx[i] - rx[jj]) * (sx[i] - rx[jj]) + (sy[i] - ry[jj]) * (sy[i] - ry[jj]));
                                    // rr[jj] = Math.Sqrt(sd[i] * sd[i] + rr[jj] * rr[jj]);
                                    rr = Math.Sqrt(sd[i] * sd[i] + rr * rr);;

                                    if ((rr < dkrit))
                                    {
                                        //rlog[jj] = Math.Log(rr[jj]);
                                        double rlog = Math.Log(rr);
                                        //mmi[jj] = 981 * Math.Exp(c1 + c2 * sm[i] + c3 * sm[i] * sm[i] + c4 * rlog[jj] + c5 * rr[jj]);
                                        double mmi = 981 * Math.Exp(c1 + c2 * sm[i] + c3 * sm[i] * sm[i] + c4 * rlog + c5 * rr);
                                        if ((mmi > 50))
                                        {
                                            //mmi[jj] = 3.66 * Math.Log10(mmi[jj]) - 1.66;
                                            mmi = 3.66 * Math.Log10(mmi) - 1.66;
                                            lmmi = Math.Log(mmi);
                                            //mdr[jj] = 0.01 * Math.Exp(v1 * lmmi * lmmi + v2 * lmmi + v3);
                                            double mdr = 0.01 * Math.Exp(v1 * lmmi * lmmi + v2 * lmmi + v3);
                                            localsum[jj] = mdr * value[jj];
                                        }
                                }

});
loss[i] = localsum.Sum();

There is a lot of scope for further optimization for example we could add a buffer into localsum to account for cache invalidation. Here's an example of what I did with the base class.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Contracts;
using Entities;

namespace Contracts
{
    public abstract class JobProcessorBase : IJobProcessor
    {
        private DateTime Start { get; set; }
        private DateTime Finish { get; set; }
        private TimeSpan Stopwatch { get; set; }
        public List<double> LossList { get; set; }

public ILog Log { set; get; }

public StorageType StorageType { get; private set; }

public void ProcessJob()
{

string key = "";
TimedAction(() => key = DequeueAndDeleteKey(), "GetExposure");

            this.StorageType = StorageType.None;
            if (key.ToLower().IndexOf("blob") == 0)
            {
                this.StorageType = StorageType.Blob;
            }

            TimedAction(() => GetExposure(key), "GetExposure");
            TimedAction(() => GetEventCatalogue(key), "GetEventCatalogue");
            TimedAction(() => RunModel(), "RunModel");
            TimedAction(() => SaveLosses(key), "SaveLosses");
            TimedAction(() => SaveLog(key), "SaveLog");

}

        private void TimedAction(Action ToDo, string MethodName)
        {
            Start = DateTime.Now;
            ToDo.Invoke();
            Finish = DateTime.Now;
            Stopwatch = Finish.Subtract(Start);
            this.Log.LogMessage(String.Format("Time to {0} {1} milliseconds",MethodName, Stopwatch.TotalMilliseconds), Stopwatch.TotalMilliseconds);
        }

        public abstract string DequeueAndDeleteKey();
        public abstract void GetExposure(string key);
        public abstract void GetEventCatalogue(string key);
        public abstract void RunModel();
        public abstract void SaveLosses(string key);
        public abstract void SaveLog(string key);
    }
}

Next I started to implement the base classes that implements a template method pattern that times each individual step. In the true spirit of test driven development I build slowly an end to end test that loaded the data ran the model and downloaded the results.

        [TestMethod]
        public void RunBlobModel()
        {
            BlobJobDispatcher blobJobDispatcher = new BlobJobDispatcher();
            string key = "";

// Load data into Cload
blobJobDispatcher.CreateJob(out key);

            // Pretend to be the Worker Role
            LoadData loadData = new LoadData();
            JobProcessor jobProcessor = new JobProcessor(loadData);
            jobProcessor.ProcessJob();

            // Recieve results back from the cload
            blobJobDispatcher.RecieveResults(out key);
        }

Here's how the client implementation looked like:

namespace Client
{
public class BlobJobDispatcher : JobDispatcherBase
{

        private IJobProcessor JobProcessor;
        private LoadData loadData { get; set; }
        private IDataStore dataStoreExposureX { get; set; }
        private IDataStore dataStoreExposureY { get; set; }
        private IDataStore dataStoreExposureV { get; set; }
        private QueueStorage dataStoreExposureQueue { get; set; }

        private IDataStore dataStoreCatalogSm { get; set; }
        private IDataStore dataStoreCatalogSx { get; set; }
        private IDataStore dataStoreCatalogSy { get; set; }
        private IDataStore dataStoreCatalogSd { get; set; }
        private QueueStorage dataStoreCatalogQueue { get; set; }

private QueueStorage jobQueue { get; set; }

private QueueStorage resultsQueue { get; set; }

private IDataStore dataStoreLosses { get; set; }
private IDataStore dataStoreLog { get; set; }

        public BlobJobDispatcher()
        {
            this.ClientLog = new Log("BlobJobDispatcher");
            loadData = new LoadData();
            string storageConnectionString = ConfigurationManager.ConnectionStrings["Storage"].ConnectionString;
            dataStoreExposureX = new BlobStorage(storageConnectionString, "exposurex", false);
            dataStoreExposureY = new BlobStorage(storageConnectionString, "exposurey", false);
            dataStoreExposureV = new BlobStorage(storageConnectionString, "exposurev", false);
            dataStoreExposureQueue = new QueueStorage(storageConnectionString, "exposurequeue", true);

            dataStoreCatalogSm = new BlobStorage(storageConnectionString, "catalogsm", false);
            dataStoreCatalogSx = new BlobStorage(storageConnectionString, "catalogsx", false);
            dataStoreCatalogSy = new BlobStorage(storageConnectionString, "catalogsy", false);
            dataStoreCatalogSd = new BlobStorage(storageConnectionString, "catalogsd", false);
            dataStoreCatalogQueue = new QueueStorage(storageConnectionString, "catalogqueue", true);

jobQueue = new QueueStorage(storageConnectionString, "jobqueue", true);

            resultsQueue = new QueueStorage(storageConnectionString, "resultsqueue", true);
            dataStoreLosses = new BlobStorage(storageConnectionString, "catalogsd", false);
            dataStoreLog = new BlobStorage(storageConnectionString, "log", false);
        }

#region JobDispatcherBase Members

        public override void ReadExposure()
        {
            loadData.ReadExposure();
        }

        public override void ReadEventCatalogue()
        {
            loadData.ReadEventCatalogue();
        }

        public override void SendExposureToCload(string key)
        {
            this.dataStoreExposureX.SendDataToStorage<double>(key, this.loadData.rxList);
            this.dataStoreExposureY.SendDataToStorage<double>(key, this.loadData.ryList);
            this.dataStoreExposureV.SendDataToStorage<double>(key, this.loadData.valueList);
            this.dataStoreExposureQueue.SendDataToStorage(key);
        }

        public override void SendEventCatalogueToCload(string key)
        {
            dataStoreCatalogSm.SendDataToStorage<double>(key, this.loadData.smList);
            dataStoreCatalogSx.SendDataToStorage<double>(key, this.loadData.sxList);
            dataStoreCatalogSy.SendDataToStorage<double>(key, this.loadData.syList);
            dataStoreCatalogSd.SendDataToStorage<double>(key, this.loadData.sdList);
            this.dataStoreCatalogQueue.SendDataToStorage(key);
        }

        public override void SubmitJobToCload(string key)
        {
            jobQueue.SendDataToStorage(key);
        }

        public override string WaitForKey()
        {
            string recievedKey = "";
            while (recievedKey == "")
            {
                resultsQueue.RecieveDataFromStorage(out recievedKey);
                Thread.Sleep(1000);
            }
            return recievedKey;
        }

        public override List<double> GetLossListFromCloud(string key)
        {
            List<double> recievedLosses = new List<double>();
            dataStoreLosses.RecieveDataFromStorage<double>(key,out recievedLosses);
            return recievedLosses;
        }

        public override List<LogEvent> GetLogsFromCloud(string key)
        {
            List<LogEvent> recievedLogs = new List<LogEvent>();
            dataStoreLog.RecieveDataFromStorage<LogEvent>(key, out recievedLogs);
            return recievedLogs;
        }

        public override void PersistLogs()
        {
            this.ClientLog.Save("C:\\Azure\\PartnerRe\\CloudInfra\\Data\\FlatFileLogg.txt",false);
        }

        public override void PersistLossList()
        {
            StreamWriter FileOut = new StreamWriter("C:\\Azure\\PartnerRe\\CloudInfra\\Data\\LossFile.txt");
            foreach (double l in LossList)
            {
                FileOut.WriteLine(string.Format("{0}", l));
            }
            FileOut.Close();
        }
        #endregion

}
}

Here's how the server side implementation looked like

namespace Server
{
    public class
        JobProcessor : JobProcessorBase
    {
        public LoadData loadData {get; set;}

private EQModel eQModel { get; set; }

        private IDataStore dataStoreExposureX { get; set; }
        private IDataStore dataStoreExposureY { get; set; }
        private IDataStore dataStoreExposureV { get; set; }
        private QueueStorage dataStoreExposureQueue { get; set; }

private QueueStorage jobQueue { get; set; }

private QueueStorage resultsQueue { get; set; }

private IDataStore dataStoreLosses { get; set; }
private IDataStore dataStoreLog { get; set; }

        public JobProcessor(LoadData LoadData)
        {
            this.loadData = LoadData;
            Log = new Log("FlatFileJobProcessor");
            eQModel = new EQModel();

            string storageConnectionString = ConfigurationManager.ConnectionStrings["Storage"].ConnectionString;
            dataStoreExposureX = new BlobStorage(storageConnectionString, "exposurex", false);
            dataStoreExposureY = new BlobStorage(storageConnectionString, "exposurey", false);
            dataStoreExposureV = new BlobStorage(storageConnectionString, "exposurev", false);
            dataStoreExposureQueue = new QueueStorage(storageConnectionString, "exposurequeue", false);

jobQueue = new QueueStorage(storageConnectionString, "jobqueue", false);

            resultsQueue = new QueueStorage(storageConnectionString, "resultsqueue", false);
            dataStoreLosses = new BlobStorage(storageConnectionString, "joblossblob", false);
            dataStoreLog = new BlobStorage(storageConnectionString, "log", false);

}

        public override string DequeueAndDeleteKey()
        {
            string recievedKey = "";
            while (recievedKey == "")
            {
                jobQueue.RecieveDataFromStorage(out recievedKey);
                if (recievedKey == "")
                {
                    Thread.Sleep(1000);
                }
            }
            return recievedKey;
        }

        public override void GetExposure(string key)
        {
            switch ( this.StorageType)
            {
                case StorageType.None:
                    break;
                case StorageType.Blob:
                    List<double> recievedx = new List<double>();
                    List<double> recievedy = new List<double>();
                    List<double> recievedv = new List<double>();

                    dataStoreExposureX.RecieveDataFromStorage<double>(key, out recievedx);
                    dataStoreExposureY.RecieveDataFromStorage<double>(key, out recievedy);
                    dataStoreExposureV.RecieveDataFromStorage<double>(key, out recievedv);

                    this.loadData.rxList = recievedx;
                    this.loadData.ryList = recievedy;
                    this.loadData.valueList = recievedv;

                    break;
                case StorageType.SQLAzure:
                    break;
                case StorageType.Table:
                    break;
            }
            return;
        }

        public override void GetEventCatalogue(string key)
        {
            switch (this.StorageType)
            {
                case StorageType.None:
                    break;
                case StorageType.Blob:

                    List<double> recievedsm = new List<double>();
                    List<double> recievedsx = new List<double>();
                    List<double> recievedsy = new List<double>();
                    List<double> recievedsd = new List<double>();

                    dataStoreCatalogSm.RecieveDataFromStorage<double>(key, out recievedsm);
                    dataStoreCatalogSx.RecieveDataFromStorage<double>(key, out recievedsx);
                    dataStoreCatalogSy.RecieveDataFromStorage<double>(key, out recievedsy);
                    dataStoreCatalogSd.RecieveDataFromStorage<double>(key, out recievedsd);

                    this.loadData.sxList = recievedsx;
                    this.loadData.syList = recievedsy;
                    this.loadData.smList= recievedsm;
                    this.loadData.sdList = recievedsd;
                    break;
                case StorageType.SQLAzure:
                    break;
                case StorageType.Table:
                    break;
            }
            return;
        }

        public override void RunModel()
        {
            this.Log.LogMessage("Start earthquake model");

List<double> result = new List<double>();

            eQModel.RunModel(loadData.rxList,
                             loadData.ryList,
                             loadData.valueList,
                             loadData.smList,
                             loadData.sxList,
                             loadData.syList,
                             loadData.sdList,
                             out result);
            this.LossList = result;

this.Log.LogMessage("=== Appending Model Log to Server Log ====");

List<LogEvent> ModelLogs = eQModel.log.GetLogs();
this.Log.Add(ModelLogs);

}

        public override void SaveLosses(string key)
        {
            switch (this.StorageType)
            {
                case StorageType.None:
                    break;
                case StorageType.Blob:
                    this.dataStoreLosses.SendDataToStorage<double>(key,this.LossList);
                    this.resultsQueue.SendDataToStorage(key);
                    break;
                case StorageType.SQLAzure:
                    break;
                case StorageType.Table:
                    break;
            }
            return;
        }

        public override void SaveLog(string key)
        {
            switch (this.StorageType)
            {
                case StorageType.None:
                    break;
                case StorageType.Blob:
                    this.dataStoreLog.SendDataToStorage<LogEvent>(key,this.Log.GetLogs());
                    break;
                case StorageType.SQLAzure:
                    break;
                case StorageType.Table:
                    break;
            }
            return;
        }
    }
}

And the unit test worked. My next step was implement a worker role that executes the ProcessJob method and to change the unit test to look like

        [TestMethod]
        public void RunBlobModel()
        {
            BlobJobDispatcher blobJobDispatcher = new BlobJobDispatcher();
            string key = "";
            blobJobDispatcher.CreateJob(out key);
            //LoadData loadData = new LoadData();
            //JobProcessor jobProcessor = new JobProcessor(loadData);
            //jobProcessor.ProcessJob();

blobJobDispatcher.RecieveResults(out key);
}

I thought that making the last step of replacing the processing part with a Worker role would be a trivial step. But it wasn’t. My main problem was that when I published my worker role the role would not start. As far as I can see there are no logs to help figure out what is going wrong. So instead I found an example of a worker role that was deployable and slowly refactored this to be in a state where it would process my earthquake models. This is quite a slow process because each time I want to test if the worker role can be deployed it took around 9 minutes.

EF4 and bulk delete

2011-03-29T07:12:00.000-07:00

When you want to delete records in EF4 typically you load the objects and delete them in the database context. This is a little slow. Microsoft recommends in such cases to use a stored procedure which I find not so cool. Here is an alternative way of doing this:


        public void DeleteAll(IQueryable clause) where TEntity : class
        {
            string sqlClause = GetClause(clause);
            this.context.ExecuteStoreCommand(string.Format(CultureInfo.InvariantCulture, "DELETE {0}", sqlClause));
        }


        private string GetClause(IQueryable clause) where TEntity : class
        {
            string snippet = "FROM [dbo].[";

            string sql = ((ObjectQuery)clause).ToTraceString();
            string sqlFirstPart = sql.Substring(sql.IndexOf(snippet));

            sqlFirstPart = sqlFirstPart.Replace("AS [Extent1]", "");
            sqlFirstPart = sqlFirstPart.Replace("[Extent1].", "");

            // The above code works well for clauses like
            //     IQueryable clause = this.databaseContext.Query().Where(v => v.ModelRunId == 1);
            // BUT as soon as you replace "v => v.ModelRunId == 1" with "v => v.ModelRunId == myId" you will get sql looking like:
            // "SELECT \r\n[Extent1].[LOSS_FILE_ID] AS [LOSS_FILE_ID], \r\n ...WHERE [Extent1].[MODEL_RUN_ID] = @p__linq__0"
            // To evaluate @p__linq__0 you need to use the code below:
            foreach (ObjectParameter Param in (((ObjectQuery)clause)).Parameters) {
                switch (Param.ParameterType.FullName)
                {
                    case "System.Int32":
                        sqlFirstPart = sqlFirstPart.Replace("@"+Param.Name, Param.Value.ToString());
                        break;
                    case "System.String":
                        sqlFirstPart = sqlFirstPart.Replace("@"+Param.Name, "'"+Param.Value.ToString().Replace("'","''")+"'");
                        break;

                    default:
                        throw new NotImplementedException("Parameter conversion for this type is not yet implemented");
                }
            }
            return sqlFirstPart;
        }

Now the delete command can be called using a fluent linq clause in the following way:


            IQueryable clause = this.databaseContext.Query().Where(v => v.ModelRunId.Value == ModelRunId);
            this.databaseContext.DeleteAll(clause);

Paired programming: How to use Data Annotations in WPF without throwing Validation exceptions

2011-03-10T10:42:00.000-08:00

We want to add simple data validation rules directly on the property within the view model. Data annotation provides attributes designed to do exactly this. There are some great websites that describe how to do this

From http://msdn.microsoft.com/en-us/library/dd901590(v=vs.95).aspx we have a great example of how this is done in Silverlight using a sdk which includes a modified datagrid and a DescriptionViewer ( which I will return to later). Within this website we can find a wide range of simple validation attributes as illustrated in the sample shown below:


    public class Product
    {
        [Display(Name="Product Number")]
        [Range(0, 5000)]
        public int ProductID { get; set; }

        [Display(Name="Name")]
        [Required]
        public string ProductName { get; set; }

        [Display(Name="Price")]
        [DataType(DataType.Currency)]
        public double ListPrice { get; set; }

        [EnumDataType(typeof(ProductColor))]
        public ProductColor Color { get; set; }

        [Display(Name="Available")]
        public bool InStock { get; set; }

        public Product() { }

        public Product(int _productID, string _productName, 
            double _listPrice, ProductColor _color, bool _inStock)
        {
            this.ProductID = _productID;
            this.ProductName = _productName;
            this.ListPrice = _listPrice;
            this.Color = _color;
            this.InStock = _inStock;
        }
    }

It is also possible to validate regular expressions eg

[RegularExpression(@"^[a-zA-Z''-'\s]{1,40}$", ErrorMessage ="….")]

We want to incorporate this into our view model because these attributes help to document the property and provide a mechanism to validate against. Here's another great web site that goes through the steps of adding this to a WPF application:

http://outcoldman.ru/en/blog/show/259

If we run through the steps described in these websites and combine with our modified INotifyPropertyChanged base class that I described in my last blog entry we end up with a model-view that looks like:


        using System.ComponentModel.DataAnnotations;  
        using System.ComponentModel; 
        namespace Ccp.Wpf.Infrastructure.Entities
        {
            public class FranchiseStepPayouts : ModelSettingsBase
            {
                private double franchiseDeductiblePercentage;
                [Display(Name = "Franchise deductible percentage", Description = "Blanket Franchise deductible percentage used to bla bla")]
                [Range(0, 100, ErrorMessage = "Value for {0} must be between {1} and {2}.")]
                public double FranchiseDeductiblePercentage
                {
                    get
                    {
                        return franchiseDeductiblePercentage;
                    }
                    set
                    {
                        Validator.ValidateProperty(value,
                            new ValidationContext(this, null, null) { MemberName = "FranchiseDeductiblePercentage" }); 
                        franchiseDeductiblePercentage = value;
                        RaisePropertyChanged(() => FranchiseDeductiblePercentage);
                    }
                }

The validation happens on line

Validator.ValidateProperty(value, new ValidationContext(this, null, null) { MemberName = "FranchiseDeductiblePercentage" });

The corresponding view looks like:


  <UserControl.Resources>
  <local:EnumMatchToBooleanConverter x:Key="enumConverter" />
  <ControlTemplate x:Key="validationTemplate">
  <DockPanel>
  <TextBlock Foreground="Red" FontSize="20">*</TextBlock>
  <AdornedElementPlaceholder/>
  </DockPanel>
  </ControlTemplate>
  <Style x:Key="textBoxInError" TargetType="{x:Type TextBox}">
  <Style.Triggers>
  <Trigger Property="Validation.HasError" Value="true">
  <Setter Property="ToolTip"
  Value="{Binding RelativeSource={x:Static RelativeSource.Self},
  Path=(Validation.Errors)[0].ErrorContent}"/>
  </Trigger>
  </Style.Triggers>
  </Style>

...


  <!-- Franchise -->
  <StackPanel Orientation="Horizontal" Grid.Column="1" Grid.Row="0">
  <TextBox Text="{Binding FranchiseDeductiblePercentage, Mode= TwoWay, ValidatesOnExceptions=true, NotifyOnValidationError=true}"
  Width="100"
  Validation.ErrorTemplate="{StaticResource validationTemplate}"
  Style="{StaticResource textBoxInError}" />
  <Label Content="%" Height="23" VerticalAlignment="Top" />
  </StackPanel>

Breaking this down into parts: The text box contains a binding with an a term "ValidatesOnExceptions=true" This tells the view that it should listen for Validation exceptions that are thrown in the view model. This happens in the setter with the Validator.ValidateProperty. So in our case we have decorated our FranchiseDeductiblePercentage with

[Display(Name = "Franchise deductible percentage", Description = "Blanket Franchise deductible percentage used to bla bla")]
[Range(0, 100, ErrorMessage = "Value for {0} must be between {1} and {2}.")]

This means that if we enter a value outside of the range 0 to 100 inclusive then an execption will be raised when we validate in the code below:

Validator.ValidateProperty(value, new ValidationContext(this, null, null) { MemberName = "FranchiseDeductiblePercentage" });

Then in the textbox we specified a validation template as follows:

Validation.ErrorTemplate="{StaticResource validationTemplate}"

The definition of this template looks like:


  <ControlTemplate x:Key="validationTemplate">
  <DockPanel>
  <TextBlock Foreground="Red" FontSize="20">*</TextBlock>
  <AdornedElementPlaceholder/>
  </DockPanel>
  </ControlTemplate>

Actually we don't need to overwrite the default validation template but it's interesting to see how this is done so that we have more control over what we are doing. But we want to get some kind of message that describes why the entered value is wrong. This is done by creating a trigger that sets a Tooltip when an error occurs. This is done as shown below:


  <Style x:Key="textBoxInError" TargetType="{x:Type TextBox}">
  <Style.Triggers>
  <Trigger Property="Validation.HasError" Value="true">
  <Setter Property="ToolTip"
  Value="{Binding RelativeSource={x:Static RelativeSource.Self},
  Path=(Validation.Errors)[0].ErrorContent}"/>
  </Trigger>
  </Style.Triggers>
  </Style>

Notice the very interesting binding path, in particular

Path=(Validation.Errors)[0].ErrorContent}

If I understand this correctly the view is listening for a validation error and then is able to resolve Validation.Errors. The display and range attributes set metadata that the Validation object is able to reflect upon and incorporate in a formatted error message. In our case we have 2 attributes

[Display(Name = "Franchise deductible percentage", Description = "Blanket Franchise deductible percentage used to bla bla")]
[Range(0, 100, ErrorMessage = "Value for {0} must be between {1} and {2}.")]

This will build a tooltip that looks like "Value for Franchise deductible percentage must be between 0 and 100."

Our next step was to think about adding a glyph or information icon that contains the description of the field so that the user has a more precise description of what the field is about. This turned out to be a little tricky. You could create some property that have the decription eg:


      public double FranchiseDeductiblePercentageDescription
        {
            get
            {
                ValidationContext x = new ValidationContext(this, null, null) { MemberName = "FranchiseDeductiblePercentage" };
                return x.DisplayName;
            }

But this means a lot of extra typing. We found more information about the definition of the DisplayAttribute at:

http://msdn.microsoft.com/en-us/library/system.componentmodel.dataannotations.displayformatattribute.aspx

We decided to use reflector to disassemble the Silverlight SDK to figure out how they implemented the DisplayViewer. The assembly was located in

C:\Program Files (x86)\Microsoft SDKs\Silverlight\v4.0\Libraries\Client\System.Windows.Controls.Data.Input.dll

There was a lot of code and we managed to get quite far but decided to refine our google search criteria to include the term converter. And we found an example but I can't remember from which web site. After some adjustments the viewer looks like



namespace Ccp.Wpf.Infrastructure.Converter
{
    [ValueConversion(typeof(String), typeof(String))]
    public class MetaDataDisplayNameConverter : IValueConverter
    {
        public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
        {
            string result = "";

            if (value != null)
            {
                Type sourceType = value.GetType();
                var propertyName = parameter as string;
                
                if (propertyName != null)
                {
                    var propertyInfo = sourceType.GetProperty(propertyName);

                    foreach (DisplayAttribute attr in propertyInfo.GetCustomAttributes(typeof(DisplayAttribute), true))
                    {
                        result = attr.Description;
                    }
                }
            }
    
            return result;
        }


        public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
        {
            throw new NotImplementedException();
        }
    }
}

This converted is used in the following way:

<Image Source="/Ccp.Wpf.Infrastructure;Component/Images/Cute-Ball-Info-icon.png" Height="23" Stretch="Uniform" VerticalAlignment="Top"
ToolTip="{Binding Converter={StaticResource metaDataDisplayNameConverter}, ConverterParameter=FranchiseDeductiblePercentage}"/>

By not specifying a parameter in the Binding clause we pass the data context to the converter. The ConverterParameter specifies a string that we should search for when we make reflection. Reflection is a slow operation so we if we use this converter in large forms we should add a cache in the way of a static private property in the converter. This could be a Hashtable or a dictionary. Coming back to the validation mechanism. There are disadvantages to throwing exceptions in setters. These are described quite well by http://outcoldman.ru/en/blog/show/259

The main problem is that exceptions should only be thrown when an unforeseen condition is met. So we want to use data annotations without throwing exceptions in the setter. It turns out that these is another binding property called

ValidatesOnDataErrors=True

When this is set the view looks for an implementation of IDataErrorInfo. So this means instead of looking for an exception we are listening to an interface. This means that our binding looks like:


  <TextBox Text="{Binding FranchiseDeductiblePercentage, Mode= TwoWay, ValidatesOnDataErrors=True, NotifyOnValidationError=true}"
  Width="100"
  Validation.ErrorTemplate="{StaticResource validationTemplate}"
  Style="{StaticResource textBoxInError}" />

We found an implementation of IDataErrorInfo at

http://stackoverflow.com/questions/3739059/prism-idataerrorinfo-validation-with-dataannotation-on-viewmodel-entities

After some small modifications our implementation looked like:


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ComponentModel;
using System.ComponentModel.DataAnnotations;
using System.Reflection;
using System.Linq.Expressions;

namespace Ccp.Contracts.UI
{
    public class DataErrorInfo : NotificationObject, IDataErrorInfo 
    {
        public string Error
        {
            get
            {
                return null;
            }
        }
        string IDataErrorInfo.this[string columnName]
        {
            get
            {
                return ValidateProperty(columnName);
            }
        }
        protected virtual string ValidateProperty(string columnName)
        {
            // get cached property accessors       
            var propertyGetters = GetPropertyGetterLookups(GetType());
            if (propertyGetters.ContainsKey(columnName))
            {
                // read value of given property         
                var value = propertyGetters[columnName](this);
                // run validation               
                var results = new List();
                var vc = new ValidationContext(this, null, null) { MemberName = columnName };
                Validator.TryValidateProperty(value, vc, results);
                // transpose results           
                var errors = Array.ConvertAll(results.ToArray(), o => o.ErrorMessage);
                return string.Join(Environment.NewLine, errors);
            }
            return string.Empty;
        }
        private static readonly Dictionary<string, object>
            PropertyLookupCache = new Dictionary<string, object>();
        private static Dictionary<string, Func<object, object>> GetPropertyGetterLookups(Type objType)
        {
            var key = objType.FullName ?? ""; 
            if (!PropertyLookupCache.ContainsKey(key))
            {
                var o = objType.GetProperties().Where(p => GetValidations(p).Length != 0).ToDictionary(p => p.Name, CreatePropertyGetter);
                PropertyLookupCache[key] = o; 
                return o;
            } 
            return (Dictionary<string, Func>object, object>>)PropertyLookupCache[key];
        }
        private static Func<object, object> CreatePropertyGetter(PropertyInfo propertyInfo)
        {
            var instanceParameter = Expression.Parameter(typeof(object), "instance");
            var expression = Expression.Lambda<Func<object, object>>(Expression.ConvertChecked(Expression.MakeMemberAccess(Expression.ConvertChecked(instanceParameter, propertyInfo.DeclaringType), propertyInfo), typeof(object)), instanceParameter);
            var compiledExpression = expression.Compile(); return compiledExpression;
        }
        private static ValidationAttribute[] GetValidations(PropertyInfo property)
        {
            return (ValidationAttribute[])property.GetCustomAttributes(typeof(ValidationAttribute), true);
        }
    }
}

Notice that we inherit from our Notification object, this is so that we can use a func instead of a string to identify which property has changed. Then all we need to do to inherit from this base class in our view model. In our case we have an intermediate base class because we are implementing a MVVMC


namespace Ccp.Contracts.Model
{
    /// 
    /// This classes could be generated from a T4 template (pocos)
    /// 

    public abstract class ModelSettingsBase : DataErrorInfo
    {
        /// 
        /// Just an example of a field
        /// 

        public string Name
        {
            get { return this.GetType().Name; }
        }

        public abstract void Save(int ModelRunId);
        public abstract bool Validate();
    }
}

Then


namespace Ccp.Wpf.Infrastructure.Entities
{
    public class FranchiseStepPayouts : ModelSettingsBase
    {
        private double franchiseDeductiblePercentage;
        [Display(Name = "Franchise deductible percentage", Description = "Blanket Franchize deductible percentage")]
        [Range(0, 100, ErrorMessage = "Value for {0} must be between {1} and {2}.")]
        public double FranchiseDeductiblePercentage
        {
            get
            {
                return franchiseDeductiblePercentage;
            }
            set
            {
                //Validator.ValidateProperty(value,  new ValidationContext(this, null, null) { MemberName = "FranchiseDeductiblePercentage" });
 
                franchiseDeductiblePercentage = value;
                RaisePropertyChanged(() => FranchiseDeductiblePercentage);
            }
        }

And now it works without exceptions

A day of WPF paired programming

2011-03-04T12:05:00.001-08:00

Here’s a journal of code that was written in a paired programming session last week…

We started by looking at the FranchiseStepPayoutFunction Control. This control has some interesting behavior. Part of the control consists of a table of LowerLimit,UpperLimit and Payout. We would like that the Upper Limit of Row n populates the lower limit of Row n+1. We would also like to style the grid with characters like "-" between the upper and lower limit and a % for the payout.

Starting with the xaml, we decided to use a ItemControl to solve this:

This is bound to a List

    public class FranchiseStepPayouts : ModelSettingsBase
    {
        public List<StepPayout> StepPayouts { get; private set; }

        public FranchiseStepPayouts()
        {
            StepPayouts = new List<StepPayout>();

            // Populate 4 columns
            for (int i = 0; i < 4; i++)
            {
                StepPayout s = new StepPayout(0, 0, 0);
                s.UpperStepLimitChangedEvent += new StepPayout.UpperStepLimitChangedHandler(s_UpperStepLimitChangedEvent);
                StepPayouts.Add(s);
            }
        }

...
}

The StepPayout looks like:

namespace Ccp.Wpf.Infrastructure.Entities
{
public class StepPayout : NotificationObject
{

        public delegate void PayoutChangedHandler(object sender);
        public event PayoutChangedHandler PayoutChangedEvent;
        virtual protected void PayoutChangedRaiseEvent()
        {
            if (PayoutChangedEvent != null)
            {
                PayoutChangedEvent(this);
            }
        }

        private double payout;
        public double Payout
        {
            get
            {
                return this.payout;
            }
            set
            {
                payout = value;
                RaisePropertyChanged(()=>Payout);
                PayoutChangedRaiseEvent();
            }
        }
        .. dito for Lower and Upper Step Payout limits

        public StepPayout(double UpperStepLimit, double LowerStepLimit, double Payout)
        {
            this.lowerStepLimit = LowerStepLimit;
            this.upperStepLimit = UpperStepLimit;
            this.payout = Payout;
        }
    }
}

Notice that we used a home made NotificationObject, this is so that we don't need to write a string in the RaisePropertyChanged field. Which means, instead of writing
RaisePropertyChanged("Payout");
we can write
RaisePropertyChanged(()=>Payout);

We copied the code that makes this possible from Prism. Our home made class for this looks like:

namespace Ccp.Wpf.Infrastructure.Helpers
{
    public class NotificationObject : INotifyPropertyChanged
    {
        public event PropertyChangedEventHandler PropertyChanged;

        protected virtual void RaisePropertyChanged(string propertyName)
        {
            if (PropertyChanged != null)
            {
                PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
            }
        }

        protected void RaisePropertyChanged<T>(Expression<Func<T>> propertyExpression)
        {
            var propertyName = ExtractPropertyName(propertyExpression);
            this.RaisePropertyChanged(propertyName);
        }

        public static string ExtractPropertyName<T>(Expression<Func<T>> propertyExpression)
        {
            if (propertyExpression == null)
            {
                throw new ArgumentNullException("propertyExpression");
            }
            var memberExpression = propertyExpression.Body as MemberExpression;
            if (memberExpression == null)
            {
                throw new ArgumentException("NotMemberAccessExpression_Exception", "propertyExpression");
            }
            var property = memberExpression.Member as PropertyInfo;
            if (property == null)
            {
                throw new ArgumentException("ExpressionNotProperty_Exception", "propertyExpression");
            }
            var getMethod = property.GetGetMethod(true);
            if (getMethod.IsStatic)
            {
                throw new ArgumentException("StaticExpression_Exception", "propertyExpression");
            }
            return memberExpression.Member.Name;
        }
    }
}

Our application does not need to be localizable so we removed the references to resources.

The next thing that we looked at was the list that we used in the view model

    public class FranchiseStepPayouts : ModelSettingsBase
    {
        public List<StepPayout> StepPayouts { get; private set; }

This works well provided the number of items in the list does not change. If the number of items in the list changes you need to replace List with ObservableCollection and you need to RaisePropertyChanged so that the UI know it needs to update. If the list was very large this would lead to an avalanche of events that would make the UI unresponsive, in that case it is better to create the observable list and set the entire property in one go. In our case we only have a few items. This makes the code look like the following.

public class FranchiseStepPayouts : ModelSettingsBase
{

        private ObservableCollection<StepPayout> stepPayouts;
        public ObservableCollection<StepPayout> StepPayouts
        {
            get
            {
                return stepPayouts;
            }
            private set
            {
                stepPayouts = value;
                RaisePropertyChanged(() => StepPayouts);
            }
        }

private PayOutFunction payoutFunction;

        public PayOutFunction PayoutFunction
        {
            get
            {
                return payoutFunction;
            }
            set
            {
                payoutFunction = value;
                RaisePropertyChanged(() => PayoutFunction);
            }
        }

We register the events in the constructor

        public FranchiseStepPayouts()
        {
            StepPayouts = new ObservableCollection<StepPayout>();

We respond to the events by replacing the lower limit of the n+1 column
        void s_UpperStepLimitChangedEvent(object sender)
        {
            for (int i = 0; i < this.StepPayouts.Count - 1; i++)
            {
                this.StepPayouts[i + 1].LowerStepLimit = this.StepPayouts[i].UpperStepLimit;
            }
        }

Next adding an auto hide function with checkbox

public class FranchiseStepPayouts : ModelSettingsBase
{

private bool isPayoutFunctionVisible;

        public bool IsPayoutFunctionVisible
        {
            get
            {
                return isPayoutFunctionVisible;
            }
            set
            {
                isPayoutFunctionVisible = value;
                RaisePropertyChanged(() => IsPayoutFunctionVisible);
            }
        }

            </Grid>
        </GroupBox>
    </StackPanel>
</UserControl>

namespace Ccp.Wpf.Infrastructure.Converter
{
    public class BoolToVisibilityConverter : MarkupExtension, IValueConverter
    {
        public BoolToVisibilityConverter()
        {
            TrueValue = Visibility.Visible;
            FalseValue = Visibility.Collapsed;
        }

public Visibility TrueValue { get; set; }
public Visibility FalseValue { get; set; }

        public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
        {
            bool val = System.Convert.ToBoolean(value);
            return val ? TrueValue : FalseValue;
        }

        public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
        {
            return TrueValue.Equals(value) ? true : false;
        }

        public override object ProvideValue(IServiceProvider serviceProvider)
        {
            return this;
        }
    }
}

Next here is the implementation of a radio button

                <StackPanel Orientation="Vertical" HorizontalAlignment="Left" Grid.Row="0" Grid.RowSpan="2" Grid.Column="0" Width="100">
                    <StackPanel Orientation="Horizontal" HorizontalAlignment="Left" >
                        <RadioButton Content="Franchise" GroupName="Franchise" Height="23"
                             IsChecked="{Binding Path=PayoutFunction, Mode=TwoWay,
                                                 Converter={StaticResource enumConverter},
                                                 ConverterParameter=Franchise}" />
                        <Label Content=" " Height="23" VerticalAlignment="Top" />
                    </StackPanel>
                    <StackPanel Orientation="Horizontal" HorizontalAlignment="Left" >
                        <RadioButton Content="Step Payout" GroupName="Franchise" Height="23"
                             IsChecked="{Binding Path=PayoutFunction, Mode=TwoWay,
                                                 Converter={StaticResource enumConverter},
                                                 ConverterParameter=StepPayout}" />
                        <Label Content=" " Height="23" VerticalAlignment="Top" />
                    </StackPanel>
                </StackPanel>
...

            </Grid>
        </GroupBox>
    </StackPanel>
</UserControl>

namespace Ccp.Wpf.Infrastructure.Controls
{
    public class EnumMatchToBooleanConverter : IValueConverter
    {
        public object Convert(object value, Type targetType,
                              object parameter, CultureInfo culture)
        {
            if (value == null || parameter == null)
                return false;

            string checkValue = value.ToString();
            string targetValue = parameter.ToString();
            return checkValue.Equals(targetValue,
                     StringComparison.InvariantCultureIgnoreCase);
        }

        public object ConvertBack(object value, Type targetType,
                                  object parameter, CultureInfo culture)
        {
            if (value == null || parameter == null)
                return null;

            bool useValue = (bool)value;
            string targetValue = parameter.ToString();
            if (useValue)
                return Enum.Parse(targetType, targetValue);

            return null;
        }
    }
}

namespace Ccp.Wpf.Infrastructure.Entities
{
    public enum PayOutFunction
    {
        None,
        Franchise,
        StepPayout
    }
}

namespace Ccp.Wpf.Infrastructure.Entities
{
    public class FranchiseStepPayouts : ModelSettingsBase
    {
        private PayOutFunction payoutFunction;

Next here is how to call the wpf windows with a foriegn key (ExposureDataId)

Use the container to access the modelview of the WPF user control

Public Class frmCatFocusMain
    Implements IDisposable
..
   Private _container As CompositionContainer
..
    Public Sub New()

'Dim directoryCatalog As New DirectoryCatalog(".")
Dim iAggregateCatalog As New AggregateCatalog()

        iAggregateCatalog.Catalogs.Add(New AssemblyCatalog(Assembly.GetExecutingAssembly()))
        iAggregateCatalog.Catalogs.Add(New AssemblyCatalog(GetType(ViewFactory).Assembly))
        iAggregateCatalog.Catalogs.Add(New AssemblyCatalog(GetType(ModelSettingsService).Assembly))
        iAggregateCatalog.Catalogs.Add(New AssemblyCatalog(GetType(ModelSettingsForEarthquakeAfricaIndia).Assembly))
        'iAggregateCatalog.Catalogs.Add(New AssemblyCatalog(GetType(CommonModelSettingsView).Assembly))

        Dim batch As New CompositionBatch()
        batch.AddPart(Me)
        _container = New CompositionContainer(iAggregateCatalog)

        ViewFactory = New ViewFactory(_container)
        _container.ComposeExportedValue(Of ViewFactory)(ViewFactory)
        _container.ComposeParts(ViewFactory)
        _container.Compose(batch)
...
        InitializeComponent()
     End Sub
...
    Private Sub BtnNewRunModelWizard_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles BtnNewRunModelWizard.Click
        Dim ExposureDataId As Integer = 12123
        Dim iWindow As Windows.Window = ViewFactory.CreateWindow(Of ModelSettingsView)()
        Dim viewModel As ModelSettingsViewModel = _container.GetExportedValueOrDefault(Of ModelSettingsViewModel)()
        viewModel.ExposureDataId = ExposureDataId
        iWindow.ShowDialog()
    End Sub

namespace Ccp.UI.ModelResults.Views.ModelSettings
{
    [Export(typeof(ModelSettingsViewModel))]
    //[PartCreationPolicy(CreationPolicy.NonShared)]
    public class ModelSettingsViewModel : ViewModelBase
    {
        private int exposureDataId;
        public int ExposureDataId
        {
            get
            {
                return exposureDataId;
            }
            set
            {
                exposureDataId = value;
                RaisePropertyChanged("ExposureDataId");
            }
        }

Next we populate the comboboxs by joining some tables using our POCO EF DAL with Linq then creating an observablecollection from the result

                var result = from av in this.databaseContext.GetAll<CatModel>()
                                 .Where(z => z.PerilId.Value.Equals(selectedModelPeril.PERIL_ID))
                                 .Where(z => ListOfImportedCatModels.Contains(z.Id))
                             join vf in this.databaseContext.GetAll<ModelRegion>()
                             on av.RegionId equals vf.RegionId
                             select vf;

ModelRegions = new ObservableCollection<ModelRegion>(result);

Here's how joining the Lazy loaded collection of controls (via MEF) with meta data containing an enumeration that has a foriegn key to the corresponding model in our database. This is done within a setter….

                selectedModelRegion = value;
                CatModel Selected = databaseContext
                                    .Query<CatModel>()
                                    .Where(p => p.RegionId == selectedModelRegion.RegionId)
                                    .Where(p => p.PerilId == selectedModelPeril.PERIL_ID)
                                    .FirstOrDefault<CatModel>();

                var result = (from av in this.ModelSettings
                                 .Where(z => (int)z.Metadata.ModelType == Selected.Id)
                             select av).FirstOrDefault();

SelectedModelSetting = result;

RaisePropertyChanged("SelectedModelRegion");

Finally here is how we persist data

namespace Ccp.Contracts.Model
{
    public abstract class ModelSettingsBase : NotificationObject
    {
        public string Name
        {
            get { return this.GetType().Name; }
        }

        public abstract void Save(int ModelRunId);
        public abstract bool Validate();
    }
}

namespace Ccp.UI.ModelResults.Views.ModelSettings
{
    [Export(typeof(ModelSettingsViewModel))]
    //[PartCreationPolicy(CreationPolicy.NonShared)]
    public class ModelSettingsViewModel : ViewModelBase
    {
        public RelayCommand<ModelSettingsBase> SaveModelSettingsCommand
        {
            get
            {
                if (saveModelSettingsCommand == null)
                {
                    saveModelSettingsCommand = new RelayCommand<ModelSettingsBase>((ms) =>
                    {
                        // Save
                        int ModelRunId = 123;

                        ms.Save(ModelRunId);
                    }
                    , (ms) =>
                    {
                        // CanSave
                        return ms != null && ms.Validate();
                    });
                }

                return saveModelSettingsCommand;
            }
        }

namespace Ccp.Wpf.Infrastructure.Entities
{
    [CatModelExport(ModelType.EarthquakeAfricaIndia, typeof(ModelSettingsBase))]
    public class ModelSettingsForEarthquakeAfricaIndia : ModelSettingsBase
    {
        public CreditDeductible CreditDeductible { get; set; }
        public ZonalDeductibleByLOB ZonalDeductible3 { get; set; }

        public ModelSettingsForEarthquakeAfricaIndia()
        {
            CreditDeductible = new CreditDeductible();
            ZonalDeductible3 = new ZonalDeductibleByLOB();
        }
        /// <summary>
        /// Dummy implementation
        /// </summary>
        public override void Save(int ModelRunId)
        {
            CreditDeductible.Save(ModelRunId);
            ZonalDeductible3.Save(ModelRunId);
        }

        public override bool Validate()
        {
            return CreditDeductible.Validate() && ZonalDeductible3.Validate();
        }
    }
}

namespace Ccp.Wpf.Infrastructure.Entities
{
    public class CreditDeductible : ModelSettingsBase
    {
        public double Limit { get; set; }
        public double Deductible { get; set; }

        public override void Save(int ModelRunId)
        {
            // Update database here
        }

Swapping WPF controls using MVVMC aka MVVM+Controller

2011-02-14T22:08:00.001-08:00

We have an application that uses WPF User Controls that we would like to dynamically change using a combobox within a pluggable architecture. The solution is based on MVVMC sometimes called MVVM+Controller.

We did this by using a SpecificViewContentProvider which as the name suggests provides a view model for a ContentPresenter that has been bound to a view model with a user control as an observable property. In our case the application is very specific therefore we did not mind coupling the View coupled to the controller. If we wanted to break this dependency we would need to use MVVMLite or Prism. Again in this case this would be an overkill, therefore we decided to leave this dependency.

We where thinking of using MVVMLite in this application and there are some traces left, for this example it is not necessary. But frameworks like Prism and MVVMLite are very useful in simplifying applications that have complex controls. The idea is that the controls produce relay events that broadcast to anyone who is interested in there status. For example a complex tree view to complex controls to view different aspects of the application

We looked at how Research would like to build the individual controls. For reasons described in day 3 we cannot inherit xaml code, therefore if we wanted to have a base control we would have to only use code behind. In windows forms this was possible with visual inheritance. Since this is not possible within xaml the visual befit of inheritance is lost. Therefore we decided to take a different approach:

Most models different sets of controls. We decided a better approach would be to setup some custom controls like CreditDecductible, ZonalDeductibles etc and compose these within the control that is to be exported into the RunModelWizard.

Here is how we achieved this:

1. We create a user control within ArchitectureExample.PresentationLayer.WPF.Common
We make some binding eg {Binding Deductible, Mode= TwoWay}

We don't make separate ModelViews because this would be an overkill

2. Next we make our first injectable control ModelSettingsForEarthquakeAlaskaView.xaml within the Views/ModelSettings directory of ArchitectureExample.PresentationLayer.WPF.Client
The interesting line is
<my:CreditDeductibleView Name="creditDeductibleView1" DataContext="{Binding CreditDeductible}" />

We bind the ScreditDeductible view to CreditDeducble

3. In the code behind we this export this. In this way we make this available to the combo box via MEF as a dependency injection container

using System.ComponentModel.Composition;
using System.Windows.Controls;
using ArchitectureExample.BusinessLayer.Entities;

namespace ArchitectureExample.PresentationLayer.WPF.Client.Views.ModelSettings
{
    /// <summary>
    /// Interaction logic for ModelSettingsForModel2Control.xaml
    /// </summary>
    [CatModelExport(ModelType.EarthquakeAlaska, typeof(IModelSettingView))]
    public partial class ModelSettingsForEarthquakeAlaskaView : UserControl, IModelSettingView
    {
        public ModelSettingsForEarthquakeAlaskaView()
        {
            InitializeComponent();
        }
    }
}

4. The CatModelExport attribute is defined as:

namespace ArchitectureExample.BusinessLayer.Entities
{
    [MetadataAttribute]
    [AttributeUsage(AttributeTargets.Class, AllowMultiple = false)]
    public class CatModelExportAttribute : ExportAttribute
    {
        public CatModelExportAttribute(ModelType modelType, Type type)
            : base(type)
        {
            this.ModelType = modelType;
        }
        public ModelType ModelType { get; private set; }
    }
}

(Here we had some problems in getting MEF to work because originally we used (Type type, ModelType modelType) resulting in nothing being exported into the container.

5. The ModelType is also defined in this assembly as
namespace ArchitectureExample.BusinessLayer.Entities
{
    public enum ModelType
    {
        None = 0,
        ...
        EarthquakeAlaska = 6,
        ...
    }
}

6. This is imported in ModelSettingsView

The interesting lines are the binding that is made to the SelectedModelSetting
<ContentPresenter x:Name="SpecificModelSetting" DataContext="{Binding SelectedModelSetting}" Grid.Row="1" Grid.Column="0" Grid.ColumnSpan="2"/>

This is set with the combobox control
<ComboBox x:Name="cmbModel" Grid.Column="1" Grid.Row="0" ItemsSource="{Binding ModelSettings}" SelectedItem="{Binding SelectedModelSetting, Mode=TwoWay}" DisplayMemberPath="Value.Name"/>

The OK button is bound to the Command SaveModelSettingsCommand. By the way, if we where programming in Silverlight we would need MVVMLite or Prism to do this because Silverlight does not support ICommand. Actually the code needed is quite small so you could role your own, but the easiest way is to take a widely used third party library like MVVMLight
<Button x:Name="btnOk" Content="OK" Margin="0 0 5 0" Command="{Binding SaveModelSettingsCommand}" CommandParameter="{Binding SelectedModelSetting.Value}"/>

7. The code behind this imports the ViewFactory

namespace ArchitectureExample.PresentationLayer.WPF.Client.Views.ModelSettings
{
    [Export(typeof(ModelSettingsView))]
    [PartCreationPolicy(CreationPolicy.NonShared)]
    public partial class ModelSettingsView : UserControl
    {
        private ModelSettingsController controller;

[Import(typeof(ViewFactory))]
public ViewFactory ViewFactory { get; set; }

        [Import]
        [SuppressMessage("Microsoft.Design", "CA1044:PropertiesShouldNotBeWriteOnly", Justification = "Needs to be a property to be composed by MEF")]
        public ModelSettingsViewModel ViewModel
        {
            set
            {
                if (this.DataContext != value)
                {
                    this.DataContext = value;
                    controller = new ModelSettingsController(ViewFactory, value, this);
                }
            }
        }
        public ModelSettingsView()
        {
            InitializeComponent();
        }
    }
}

8. The View Factory contains and enumerable of Lazy<IModelSettingView, IModelSettingMetadata>

namespace ArchitectureExample.PresentationLayer.WPF.Client.Infrastructure
{
    public class ViewFactory
    {
        [ImportMany(typeof(IModelSettingView))]
        public IEnumerable<Lazy<IModelSettingView, IModelSettingMetadata>> ModuleSettingsViews { get; set; }

private readonly CompositionContainer container;

        public ViewFactory(CompositionContainer container)
        {
            this.container = container;
        }

        public T GetView<T>(string viewName)
        {
            var view = this.container.GetExportedValueOrDefault<T>(viewName);
            if (view == null)
            {
                throw new InvalidOperationException(string.Format("Unable to locate view with name {0}.", viewName));
            }

return view;
}

        public IModelSettingView GetView(ModelType modelType)
        {
            //var view = this.container.GetExportedValues<Lazy<T, TMetaData>>().Where<Lazy<T, TMetaData>>(a => a.Metadata.ModelType == modelType).First();
            var view = ModuleSettingsViews.Where(a => a.Metadata.ModelType == modelType).First();
            if (view == null)
            {
                throw new InvalidOperationException(string.Format("Unable to locate view with type {0}.", modelType));
            }

return view.Value;
}

        public T GetView<T>()
        {
            var view = this.container.GetExportedValueOrDefault<T>();
            if (view == null)
            {
                throw new InvalidOperationException(string.Format("Unable to locate view with type {0}.", typeof(T).Name));
            }

return view;
}

        public Window CreateWindow<T>()
        {
            var view = this.GetView<T>();

return CreateWindow<T>(view); ;
}

        public Window CreateWindow<T>(T content)
        {
            var window = new Window();
            // window.DataContext = dataContext;

window.Content = content;

return window;
}

        public ModelSettingsViewWithRegions CreateModelSettingsViewWithRegions()
        {
            IRegionManager regionManager = this.container.GetExportedValueOrDefault<IRegionManager>();
            ModelSettingsViewWithRegions view = new ModelSettingsViewWithRegions();// { DataContext = viewModel };
            this.container.ComposeParts(view);

            RegionManager.SetRegionManager(view, regionManager);
            RegionManager.UpdateRegions();
            return view;
        }
    }
}

Lets take a closer look ate the
[ImportMany(typeof(IModelSettingView))]
public IEnumerable<Lazy<IModelSettingView, IModelSettingMetadata>> ModuleSettingsViews { get; set; }

ImportMany asks MEF to populate the IEnumerable with exported objects that implement IModelSettingView
If we take a look into the defintion of Lazy we see...

    [Serializable]
    public class Lazy<T, TMetadata> : Lazy<T>
...
        //
        // Summary:
        //     Initializes a new instance of the System.Lazy<T,TMetadata> class with the
        //     specified metadata that uses the specified function to get the referenced
        //     object.
        //
        // Parameters:
        //   valueFactory:
        //     A function that returns the referenced object.
        //
        //   metadata:
        //     The metadata associated with the referenced object.
        [TargetedPatchingOptOut("Performance critical to inline this type of method across NGen image boundaries")]
        public Lazy(Func<T> valueFactory, TMetadata metadata);
...

So what this is saying is that this function provides an interface to the meta data that is added in our Attribute.

In our case the interface to the meta data is IModelSettingMetadata

namespace ArchitectureExample.BusinessLayer.Entities
{
    public interface IModelSettingMetadata
    {
        ModelType ModelType { get; }
    }
}

This meta data is needed to identify the control we want to import. Within the function GetView we use a linq expression to find the modelview that we are interested in. Since this is an enumeration of type lazy we need to use .value to instantiate the view model.

        public IModelSettingView GetView(ModelType modelType)
        {
            var view = ModuleSettingsViews.Where(a => a.Metadata.ModelType == modelType).First();
            if (view == null)
            {
                throw new InvalidOperationException(string.Format("Unable to locate view with type {0}.", modelType));
            }
            return view.Value;
        }

This is used in the ModelSettings Controller...

9. Here is the ModelSettingsController used in the MVVMC pattern

namespace ArchitectureExample.PresentationLayer.WPF.Client.Views.ModelSettings
{
    public class ModelSettingsController
    {
        private ViewFactory viewFactory;
        private ModelSettingsViewModel viewModel;
        private ModelSettingsView view;
        public ModelSettingsController(ViewFactory viewFactory,ModelSettingsViewModel viewModel, ModelSettingsView view)
        {
            this.viewFactory = viewFactory;
            this.viewModel = viewModel;
            this.view = view;
            //this.view.DataContext = viewModel;
//This listens to events created when the combobox changes
            this.viewModel.PropertyChanged += new System.ComponentModel.PropertyChangedEventHandler(viewModel_PropertyChanged);
            UpdateSpecificModelSettingsView();
        }

        void viewModel_PropertyChanged(object sender, System.ComponentModel.PropertyChangedEventArgs e)
        {
            if (e.PropertyName == "SelectedModelSetting")
            {
                UpdateSpecificModelSettingsView();
            }
        }

        private void UpdateSpecificModelSettingsView()
        {
            if (this.viewModel.SelectedModelSetting != null)
            {
//==Here is where the Imported Enumeration of Lazy IModelSettingView are used
                var specificView = viewFactory.GetView(this.viewModel.SelectedModelSetting.Metadata.ModelType) as UserControl;
//Here we have to set the data context because we are changing the control
                specificView.DataContext = this.viewModel.SelectedModelSetting.Value;
//Here we populate the place holder with our view (tightly coupled)
                this.view.SpecificModelSetting.Content = specificView;

}
}

10. The View model belonging to ModelSettingsView.xaml is ModelSettingsViewModel.cs
Inside this we expose Lazy Observable collections and selected values. Here we are using MVVMLite and the ViewModelBase. Actually we don't need to do this because WPF has an implementation of ICommand.

using GalaSoft.MvvmLight;
using GalaSoft.MvvmLight.Command;

namespace ArchitectureExample.PresentationLayer.WPF.Client.Views.ModelSettings
{
    [Export(typeof(ModelSettingsViewModel))]
    //[PartCreationPolicy(CreationPolicy.NonShared)]
    public class ModelSettingsViewModel : ViewModelBase
    {
        private IModelSettingsService moduleSettingsService;
        private ObservableCollection<Lazy<ModelSettingsBase, IModelSettingMetadata>> modelSettings;
        private Lazy<ModelSettingsBase, IModelSettingMetadata> selectedModelSetting;

private RelayCommand<ModelSettingsBase> saveModelSettingsCommand;

        public ObservableCollection<Lazy<ModelSettingsBase, IModelSettingMetadata>> ModelSettings
        {
            get
            {
                return modelSettings;
            }
            private set
            {
                if (modelSettings != value)
                {
                    modelSettings = value;

                    RaisePropertyChanged("ModelSettings");
                }
            }
        }

        public Lazy<ModelSettingsBase, IModelSettingMetadata> SelectedModelSetting
        {
            get
            {
                return selectedModelSetting;
            }
            set
            {
                if (selectedModelSetting != value)
                {
                    selectedModelSetting = value;

                    RaisePropertyChanged("SelectedModelSetting");
                    SaveModelSettingsCommand.RaiseCanExecuteChanged();
                }
            }
        }

        public RelayCommand<ModelSettingsBase> SaveModelSettingsCommand
        {
            get
            {
                if (saveModelSettingsCommand == null)
                {
                    saveModelSettingsCommand = new RelayCommand<ModelSettingsBase>((ms) =>
                    {
                        // Save
                        ms.Save();
                    }
                    , (ms) =>
                    {
                        // CanSave
                        return ms !=null && ms.Validate();
                    });
                }

                return saveModelSettingsCommand;
            }
        }

        [ImportingConstructor]
        public ModelSettingsViewModel(IModelSettingsService moduleSettingsService)
        {
            this.moduleSettingsService = moduleSettingsService;

            ModelSettings = new ObservableCollection<Lazy<ModelSettingsBase, IModelSettingMetadata>>(this.moduleSettingsService.GetModuleSettings());
            foreach (Lazy<ModelSettingsBase, IModelSettingMetadata> modelsettings in ModelSettings)
            {
                var test= modelsettings.Value;
            }
        }
   }
}

The interesting parts of the above are
         public RelayCommand<ModelSettingsBase> SaveModelSettingsCommand
Notice that in the can Execute Func we first chack if Ms !=null. Without this check first the function will throw an exception
                    , (ms) =>
                    {
                        // CanSave
                        return ms !=null && ms.Validate();
                    });

The ms.Validate part is invoking an implementation of an abstract class:

namespace ArchitectureExample.BusinessLayer.Entities
{
    /// <summary>
    /// This classes could be generated from a T4 template (pocos)
    /// </summary>
    public abstract class ModelSettingsBase
    {
        /// <summary>
        /// Just an example of a field
        /// </summary>
        public string Name
        {
            get { return this.GetType().Name; }
        }
        public abstract void Save();
        public abstract bool Validate();
    }
}

11. Here is where we have implemented this within
namespace ArchitectureExample.BusinessLayer.Entities
{
    public class CreditDeductible : ModelSettingsBase
    {
        public double Limit { get; set; }
        public double Deductible { get; set; }

        public CreditDeductible()
        {
            Limit = 100;
            Deductible = 0;
        }

        public override void Save()
        {
        }

        public override bool Validate()
        {
            return Limit <= 100 && Limit >= 0 && Deductible <= 100 && Deductible >= 0;
        }
    }
}

Notice that we decided to use a boolean to return validation. The alternative would have been to use an exception. We choose boolean because exceptions are slow because they need to serialize the stack trace.

There is a validation enterprise building block that uses attributes to set validation rules. We will look into using this because it provides a tidy way to associate validation rules to properties and will remove the untidy code in the Validate() function

We tested this with 2 imported controls with 1 and 2 of these user controls. The OK button of the main control was properly enabled and disabled according to the validation rules. Also as we changed the combo box the values from one custom control to another where correctly persisted...

Improvements in number crunching speeds from .Net 3.5 to .Net 4

2010-12-30T23:57:00.000-08:00

Here's something interesting: In a previous blog entry we found a 2 times speedup in the performance of a C# version of our Cat Models from .Net 3.5 to .Net 4.0. I followed up with Microsoft on this and got a very helpfull feedback from Surupa Biswas and Stephen Toub:

Hello again Nigel,

Someone on my team took a deeper look at this and was able to deterministically reproduce a ~2x speed up on his box. It looks like the primary wins are coming from the CRT, not the JIT. The 8.0/3.5 CRT was using x87 floating point instructions, but the 10.0/4.0 CRT is using SSE. The JIT uses the CRT for the Log/ Log10 and Exp calls. It seems like the SSE instructions used by the C runtime are the primary source of the wins although the JIT-ed code was better too.

.NET 3.5 also had the CRT installed in a global location (WinSxS) and hence could be affected by other applications that updated the CRT, which could explain the differences in our observations.

Thanks again for sharing your test case and reporting this to us.

Best,
Surupa

For the non technical:

JIT = Just In Time (it is the compiler that converts the IL Intermediate code (what we see as .exe file) into a native executable)
CRT = Common Runtime Library (It is all the libraries that the native code uses to interact with the hardware) This is a libray that is delivered with the framework and therefore has different versions:
.NET 2.0 -> CRT 8.0.50727.42
.NET 2.0 SP1 -> CRT 8.0.50727.762
.NET 3.5 -> CRT 9.0.21022.8
.NET 3.5 SP1 -> CRT 9.0.30729.1

I believe SSE means Streaming SIMD Extensions which are a set of processor extensions for Intel processors

From this web site http://neilkemp.us/src/sse_tutorial/sse_tutorial.html I found the following description of what SSE is
"First what the heck is SSE? Basically SSE is a collection of 128 bit CPU registers. These registers can be packed with 4 32 bit scalars after which an operation can be performed on each of the 4 elements simultaneously."

If I understand this correctly the .Net 4 takes advantage of certain processor extensions on the Intel processor via an improved version of the Common Runtime Library

Here's a great article over the CLR improvements on .Net 3.5 http://msdn.microsoft.com/en-us/magazine/dd569747.aspx

A WCF Hosting story

2010-12-21T11:18:00.001-08:00

I have a VS2010 project that has a WCF service that is ready to be deployed. I now want to setup IIS7 to use WAS to host the WCF components. Here are the steps needed to do this

1. Goto Control Panel/Administrative Tools/Internet Information Services (IIS) Manager
2. Browse to Sites/Default Website
3. Right mouse click and "Add Virzual Directory..."
4. In the Add Virtual Directory
- Alias : 05
- Physical path: C:\InetPub\wwwroot\wmi\2010\12\05

(The objective is: To have a simple URL. Have the ability to version both the service and the namespaces. Monitor when decommissioning a service so as to prevent breaking applications [In our case done via logging in a base class])

5. Browse to Application Pools
6. Right mouse click on the list box and Add Application Pool....
7. Add Application Pool
   - Name :Wmi
   - .Net Framework v4.0.30319
   - Managed pipeine mode : Integrated
8. Right mouse click on the newly created WMI service account and select Advanced Settings
9. Under Process Model click on the "..." button next to Identity
10. Select Custom account and enter the service account details

At this point you could try and publish...
i Right mouse click on the ASP.NET project containing the WCF service
ii Enter
   - Publish profile: BulsburgProfile
   - Publish method: File Sysetm
   - Target location \\chbbpresxxx\wwwroot$\wmi\2010\12\05
Then if you enter the url to the service http://chbbpresxxx/wmi/Disk.svc you will get the following error

Server Error in '/' Application.
________________________________________
Runtime Error
Description: An application error occurred on the server. The current custom error settings for this application prevent the details of the application error from being viewed remotely (for security reasons). It could, however, be viewed by browsers running on the local server machine.

Details: To enable the details of this specific error message to be viewable on remote machines, please create a <customErrors> tag within a "web.config" configuration file located in the root directory of the current web application. This <customErrors> tag should then have its "mode" attribute set to "Off".



<configuration>
    <system.web>
        <customErrors mode="Off"/>
    </system.web>
</configuration>

Notes: The current error page you are seeing can be replaced by a custom error page by modifying the "defaultRedirect" attribute of the application's <customErrors> configuration tag to point to a custom error page URL.



<configuration>
    <system.web>
        <customErrors mode="RemoteOnly" defaultRedirect="mycustompage.htm"/>
    </system.web>
</configuration>

So now we add the following to Web.Config and republish
    <system.web>
        <customErrors mode="Off"/>
    </system.web>

Then you will get the following error:

Server Error in '/' Application.
________________________________________
The type 'Ccp.Wcf.Wmi.Disk', provided as the Service attribute value in the ServiceHost directive, or provided in the configuration element system.serviceModel/serviceHostingEnvironment/serviceActivations could not be found.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

The solution is to
11. Go back to the Internet Information Services (IIS) Manager
12. Right mouse click on the newly created virtual directory and select "Convert to Application"
13. - Alias: wmi
- Application pool: wmi
- Physical path: C:\InetPub\wwwroot\wmi\2010\12\05
Then it works...

Next I want to add the finishing touches to the web.config
    <system.serviceModel>
      <services>
        <service behaviorConfiguration="Ccp.Wcf.Wmi.DiskBehavior"
          name="Ccp.Wcf.Wmi.Disk">
          <endpoint address="" binding="wsHttpBinding" contract="Ccp.Wcf.Wmi.IDisk">
            <identity>
              <dns value="localhost" />
            </identity>
          </endpoint>
          <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
          <host>
            <baseAddresses>
              
              
            </baseAddresses>
          </host>
        </service>
      </services>

If you happen to try the same thing on IIS6 there are some differences:

1. Inside the Internet Information Services (IIS) Manager
2. Browse to Web Service Extentions and enable ASP.NET v4.0.30319
3. Repeat the excersize of creating the application pool and virtual directory
4. Copy the bin and the web.config file from C:\InetPub\wwwroot\wmi\2010\12\05 to C:\InetPub\wwwroot\

Then when you try to do some special WMI calls you will end up with some errors like : Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)). In our case the IIS6 Server is a fringe server that will be updated to Server 2008

So now taking our system administrator hat off and putting our developer hat back on. We have an client app.config file that looks like:
            <endpoint address="http://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc"
                binding="wsHttpBinding" bindingConfiguration="DiskwsHttp"
                contract="WmiDeployedServiceReference.IDisk" name="DiskwsHttpBalsburg">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>

NB the dns entry is necessary otherwise the service will not be found

The corresponding code is as follows:

        [TestMethod]
        public void GetFreeSpaceInGBWCfDeployed_CheckDiskCapacityOnC_StringContainingDiskCapacity()
        {
            WmiDeployedServiceReference.DiskClient proxy = new WmiDeployedServiceReference.DiskClient("DiskwsHttpBalsburg");

            string diskCapacityString = "";
            string serverName = Environment.MachineName;
            string driveLetter = "C";

diskCapacityString = proxy.GetFreeSpaceInGB(serverName, driveLetter);

long AvailableSpaceOnC = 0;

            DriveInfo d = new DriveInfo(driveLetter);
            AvailableSpaceOnC = d.AvailableFreeSpace;
            AvailableSpaceOnC = AvailableSpaceOnC / 1024 / 1024 / 1024;

long diskCapacityLong = Convert.ToInt64(Convert.ToDouble(diskCapacityString));

Assert.IsFalse(diskCapacityString == "");
Assert.IsTrue(diskCapacityLong == AvailableSpaceOnC);

}

Now I want to use NetTcp because it is more efficient than wsHttp.
In some early iterations after adding the netTcp binding when I try http://chbbpresxxx/wmi/2010/12/05/Disk.svc
I get the error like an error like:

Server Error in '/Wmi/2010/12/05' Application.
--------------------------------------------------------------------------------

Could not find a base address that matches scheme net.tcp for the endpoint with binding NetTcpBinding. Registered base address schemes are [http].
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

To fix this error you need to add the net.tcp binding to the binding of the application in IIS.
1. Right click the virtual directory/application in IIS -> Manage application -> Advanced settings.
2. On the Enabled Protocols : http,net.tcp

I tried out the WCF configuration editor to setup a simple net-Tcp binding to the existing contract and I publish it. Then when I open the http://chbbpresxxx/wmi/2010/12/05/Disk.svc I got the error:

Cannot load the X.509 certificate identity specified in the configuration.

The solution in my case was to comment out the certificate in the web.config

Alternatively you can follow the instructions in the following url to setup the certificate

http://wcfsecurity.codeplex.com/wikipage?title=How%20To%20-%20Create%20and%20Install%20Temporary%20Certificates%20in%20WCF%20for%20Message%20Security%20During%20Development&ProjectName=wcfsecurity

After some other edits for the wsHttp binding works but the net-tcp binding gives an error like

Could not connect to net.tcp://chbbpresxxx.global.partnerre.net:8731/Wmi/2010/12/05/Disk.svc. The connection attempt lasted for a time span of 00:00:01.1986602. TCP error code 10061: No connection could be made because the target machine actively refused it 10.102.65.150:8731.

This error had me stuck for a while, I tried a lot of things, all failed. Here is a list:
- Look at the Advanced settings of wwwroot and add http,net.tcp to the enabled protocols (as in the Virtual directory)
- Right mouse click on wwwroot and add Binding.
- Modified existing binging from 808:* to 9* to 9001 and back
- Adding net-tcp with 9001
- Under Server Manager/Configuration/Microsoft Fire Wall with.../ look at the properties and set Firewall status to off at doamin, private and public levels

The problem was the setup of Features where different. To fix this we have to put on our system administrator hat on and do the following:

1. Open the Server Manager
2. Browse to Server Manager/features
3. Add Under .Net Framework 3.0 Features (Installed)
       Under WCF Activation
       + HTTP Activation
       + Non HTTP Activations
4. The installation took some time

Now if we try the test client by:
1. D:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\WcfTestClient.exe
2. File/ Add Service
3. http://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc
We get the error

Error: Cannot obtain Metadata from http://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc
If this is a Windows (R) Communication Foundation service to which you have access, please check that you have enabled metadata publishing at the specified address. For help enabling metadata publishing, please refer to the MSDN documentation at http://go.microsoft.com/fwlink/?LinkId=65455.WS-Metadata Exchange etc etc

I found a link that describes what is happening

http://blogs.msdn.com/b/webtopics/archive/2010/04/28/system-typeloadexception-for-system-servicemodel-activation-httpmodule-in-asp-net-4.aspx

Installation of .NET 3.5.1 WCF HTTP activation feature adds a global module for the 3.0 framework’s 'System.ServiceModel’ assembly for the type 'System.ServiceModel.Activation.HttpModule'. Since the application pool’s runtime version is v4.0, this assembly is tried to be loaded from the .NET 4 assemblies folder. Since, the definition of the 'System.ServiceModel.Activation.HttpModule’ is now moved to the “System.ServiceModel.Activation” assembly, it fails.

Following the instructions the to fix this open a command prompt:

C:\>cd C:\Windows\Microsoft.NET\Framework\v4.0.30319
C:\Windows\Microsoft.NET\Framework\v4.0.30319>aspnet_regiis -iru
Start installing ASP.NET (4.0.30319).
..................................
Finished installing ASP.NET (4.0.30319).

Then reboot the server...
By the way the following error happens when I tried adding a Binding in the IIS Manager for net-tcp port 9001

The protocol binding '9001' does not conform to the syntax for 'net.tcp'. The following is an example of valid 'net.tcp' protocol bindings: '808:*'.

The solution is to remove the added net-tcp binding from the IIS Server Manager.

The Final Solution with our developer hat on

We use baseAddress="net.tcp://localhost:8080/Wmi/2010/12/05/Disk.svc" in the Web.config that looks like:

<?xml version="1.0"?>
<configuration>
    <system.web>
        <compilation debug="true" targetFramework="4.0" />
      <customErrors mode="Off"/>
    </system.web>

    <system.serviceModel>
      <services>
        <service behaviorConfiguration="Ccp.Wcf.Wmi.DiskBehavior" name="Ccp.Wcf.Wmi.Disk">
          <clear />
          <endpoint binding="wsHttpBinding" name="DiskwsHttp" contract="Ccp.Wcf.Wmi.IDisk"
            listenUriMode="Explicit">
            <identity>
              <dns value="localhost" />
            </identity>
          </endpoint>
          <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"
            listenUriMode="Explicit">
            <identity>
            </identity>
          </endpoint>
          <endpoint address=""
            binding="netTcpBinding" name="DisknetTcp" contract="Ccp.Wcf.Wmi.IDisk" >
            <identity>
              <dns value="localhost" />
            </identity>
          </endpoint>
          <host>
            <baseAddresses>
              
              <add baseAddress="net.tcp://localhost:8080/Wmi/2010/12/05/Disk.svc" />
            </baseAddresses>
          </host>

        </service>
      </services>
      <behaviors>
        <serviceBehaviors>
          <behavior name="Ccp.Wcf.Wmi.DiskBehavior">
            
            <serviceMetadata httpGetEnabled="True"/>
            
            <serviceDebug includeExceptionDetailInFaults="true" />
          </behavior>
        </serviceBehaviors>
      </behaviors>
      <bindings>
        <netTcpBinding>
          <binding name="ReliableSessionBinding">
            <reliableSession ordered="false" inactivityTimeout="00:10:00" enabled="true" />
          </binding>
          <binding name ="DisknetTcp">
          </binding>
        </netTcpBinding>
      </bindings>
      <client>
        <endpoint address="net.tcp://services-tst.global.partnerre.net/Group/Common/Logging/2009/02/11/LogService.svc"
             binding="netTcpBinding" bindingConfiguration="ReliableSessionBinding"
             contract="LogServiceReference.ILogService" >
          <identity>
            <servicePrincipalName value="host/CHBBPRES563.global.partnerre.net" />
          </identity>
        </endpoint>
      </client>
    </system.serviceModel>
</configuration>

We use address="net.tcp://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc"in the App.config, that ends up looking like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <system.serviceModel>
        <client>
            <endpoint address="http://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc"
                binding="wsHttpBinding" bindingConfiguration="DiskwsHttp"
                contract="WmiDeployedServiceReference.IDisk" name="DiskwsHttpBalsburg">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
            <endpoint address="http://localhost:8732/Design_Time_Addresses/Ccp.Wcf.Wmi/Disk/"
                binding="wsHttpBinding" bindingConfiguration="WSHttpBinding_IDisk"
                contract="WmiServiceReference.IDisk" name="WSHttpBinding_IDisk">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
            <endpoint address="net.tcp://localhost:8731/Wmi/2010/12/05/Disk"
                binding="netTcpBinding" contract="WmiServiceReference.IDisk"
                name="NetTcpBinding_IDisk">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
            <endpoint address="http://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc"
                binding="wsHttpBinding" bindingConfiguration="DiskwsHttp1"
                contract="WmiDeployedServiceReference.IDisk" name="DiskwsHttp">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
            <endpoint address="net.tcp://chbbpresxxx.global.partnerre.net/Wmi/2010/12/05/Disk.svc"
                binding="netTcpBinding"
                contract="WmiDeployedServiceReference.IDisk" name="DisknetTcp">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
        </client>
    </system.serviceModel>
</configuration>

And the unit test looks like:
        [TestMethod]
        public void GetFreeSpaceInGBWCfDeployedTcp_CheckDiskCapacityOnC_StringContainingDiskCapacity()
        {
            WmiDeployedServiceReference.DiskClient proxy = new WmiDeployedServiceReference.DiskClient("DisknetTcp");

            string diskCapacityString = "";
            string serverName = Environment.MachineName;
            string driveLetter = "C";

diskCapacityString = proxy.GetFreeSpaceInGB(serverName, driveLetter);

long AvailableSpaceOnC = 0;

            DriveInfo d = new DriveInfo(driveLetter);
            AvailableSpaceOnC = d.AvailableFreeSpace;
            AvailableSpaceOnC = AvailableSpaceOnC / 1024 / 1024 / 1024;

long diskCapacityLong = Convert.ToInt64(Convert.ToDouble(diskCapacityString));

Assert.IsFalse(diskCapacityString == "");
Assert.IsTrue(diskCapacityLong == AvailableSpaceOnC);

}

Number crunching

2010-11-25T23:52:00.000-08:00

The benchmark tests that were carried out last year to include code generated with VS2010 and .Net 4.0. There was a significant improvement in speed (more than 10 times) which makes a model that has been completely written in C# about 0.7 times as fast as code written in unmanaged C++. We are looking into Intel C++ compiler options to determine if we are missing something. But the outcome is that it is no longer a clear cut decision between managed C# an unmanaged C++. The 30% drop in performance could be compensated with the business value of having all code in one domain.

For completeness: It is well know that Fortran compilers are more efficient than C and are used for the core algorithms in most super computer sites. Our test cases showed that this is 5 times faster than C#/C++ solution that we have now. If we compile everything in Intel C++ we get a 30% speed up on 32 bit and a 40% speed up on 64 bit.

Although we reserve the right to write code in the most efficient language available there is a lot of value in keeping to standard well know languages. We therefore considered 3 options:

1. Everything in C++
2. Everything in C#
3. The Event loop in C# and the location loop in C++

The idea of options 1 and 2 is to have everything in the same domain. This makes it easier to optimize both the event and the location loops. This is important for the case when we want to look into per risk policies. This is because instead of having an event loop as the outer loop it makes more sense to have the exposure location loop as the outer loop since instead of summing on an event basis we are summing on a per risk basis. The point is that the exposure table is very big and it is expensive to make copies of this data, hence it is better to change the inner and outer loop.

The decision was for option 3 for the following reasons:
- Preserves existing investment
- There will always be a need for a mathematical library, having this in unmanaged code is the optimal way from a compute point of view and allows portability (eg super computers)
- The Per Risk Policies would be handled by adding new functionality to the existing library
- The readability of the code will be improved by writing some more specific functions within the mathematical library

The discussions showed that by choosing this option we do give up the following things:
- A homogenous language where there is complete freedom over how to optimize and arrange code.
- There is a need to maintain know-how in 2 languages and there is an associated barrier for scientists to understand, debug and further develop core mathematical libraries
- IT will not be able to support C++. (This is not such a big issue because a deep domain knowledge is required which would make such collaboration difficult in the first place)

With the choice of language made we can look more seriously into the structure of base classes where code could be shared between the different peril models. The base class would consist of a template method pattern that would execute methods for:

- Loading exposure
- Disaggregation
- Running the Event Loop
- Calculating Ground up loss
- Calculating loss sigma
- Applying policy structures

The idea is that the models would override the model specific operations.

For the loading of exposure to work we need to use one exposure format.

Disaggregation: the conversion of locators into a distribution of lat long point values according to distributions like population, industry exposure, Commercial and residential distribution etc seems to be an operation that is quite independent. But the models make choices such as vulnerability, soil type etc based on geographic location. It is much easier to derive this information from a locator than to determine in which location a lat long is located. Although the building of such a database sounds a simple task there are implications on how we understand the model output. Basically it is important to have a thorough understanding of disaggregation in order to understand the model results and to make an underwriting decision that makes sense.

Looking at how we want to host the models on a server farm

We are looking into service providers that would allow us to rent virtual machines on a seasonal basis. It is not clear whether we will find a provider that is willing to do this at an appropriate price.

Our computational needs are not the same as some of the Reinsurance companies that rely completely on Vendor models. Since the license fee for the HPC with RMS is significant we don’t have an immediate advantage of using HPC for RMS. The existing Scheduler for the common platform is working well therefore there is no pressure from this side.

There is value in using a scheduler that has an industry following. There is also value in choosing a scheduler that will be commonly used in our industry. There could be value in the ability to scale out on VM Roles in the cloud. From a pure number crunching point of view the first priority should be to make the number crunching modules as efficient as possible by optimizing algorithms and choice of compiler etc.

We had a discussion over distributed computing. Algorithms such as finite element analysis need to pass an entire interface between iterations and will therefore cause a lot of messages to be passed with MPI. In this case it is really important that the compute nodes very well connected. In the case of cat modeling we are passing some totals between iterations and therefore MPI messages are very small. Therefore it might not be so essential that the compute nodes are very well connected. In the case of Windstorm models algoritm speed is disk io bound because the bottle neck is on reading storm footprint files, in which case it would make sense to partition jobs based on storm footprints.

My first steps in Azure Cloud Computing

2010-11-25T04:59:00.000-08:00

Here's a log of my first experiments in Azure. To start of with it's good to have a plan of what to learn. This is what I came up with :

1. Development Environment
-> Now works on PDC laptop
2. Storage Model
- Blobs
- Tables
- Cache
3. How to initialize data in the cloud
4. How to extract data from the cloud
5. Simple UI
6. Data Access layer
7. Worker Roles
8. Cache
9. Test Driven Development

Setting up the development environment has some hurdles. Firstly it does not seem to work on WindowsXP and it requires IIS7. Here's a list of steps:

1. Open VS2010
2. Select the Cload Project
3. Click on Download Windows Azure Tools
Windows Azure Tools for Microsoft Visual Studio 1.2 (June 2010)
4. Click Download
5. Run
6. Run

This time it worked well. When you run an Azure application in debug mode you need to start the AppFabric service on your PC. To do this there is a component installed ion the Azure SDK section of the start menu.

I had no problems in starting the Development AppFabric but could not start the development storage

To get around this I ran c:\Program Files\Windows Azure SDK\v1.2\bin\devstore\dsservice.exe which produced an error:

A network-related or instance-specific error occured while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that the SQL Server is configured to allow remote connections.( provider SQL Network Interfaces, error 26 - Error Locating Server/Instance Specified)

I started the SQL Server Service and tried again. and I was able to run my hallo world Azure application.

The next thing I wanted to do is see how easy it would be to deploy this to Azure To do this I did the following steps

1. Right mouse click cloud project an select publish
2. create a certificate and install on pc
3. export certificate to file
4. import certificate to azure
5. create storage and servce accounts
6. This takes a long time

Then I got the warning:

WARNING: This deployment has at least one role with only one instance. We recommend that you deploy at least two instances per role to ensure high availability in case one of the instances becomes unavailable. Doing so also enables coverage of the Windows Azure Compute SLA, which guarantees 99.95% uptime. For more information please visit this link.

To correct this:
1. Goto WebRoleTable under Roles in the TableCloadService subproject
2. Increase the instance count by 1 to 2

This got rid of the error message but the package could not be deployed to staging.

Some hallo world experiments later I got a message from Microsoft:

Sehr geehrte(r) Nigel Findlater!

Diese E-Mail-Benachrichtigung soll Sie über Ihre Nutzung von Windows Azure Platform informieren. Nach unseren Aufzeichnungen hat Ihr Abonnement 125 % der Angebots-Servernutzungszeit für den aktuellen Abrechnungszeitraum überschritten. Für alle Stunden, die die im Abonnement inbegriffenen Zeiten übersteigen, fallen Standardgebühren an.

Nachstehend sind die Nutzungswerte bis zum jetzigen Zeitpunkt im Rahmen des laufenden Abrechnungszeitraums aufgeführt.

: 42.000000

Soweit dieser Wert nicht unerwartet ist, sind keine Handlungen Ihrerseits erforderlich. Sollte der Nutzungswert unerwartet sein, melden Sie sich beim Windows Azure Dev-Portal an, um Ihre derzeit beanspruchten Dienste einzusehen, und nehmen Sie die erforderlichen Änderungen an diesen Diensten vor, um Ihre Nutzung wieder auf einen Nominalwert zu bringen.

Sie können sich jederzeit beim Microsoft Online Services-Kundenportal unter https://mocp.microsoftonline.com anmelden, um die Nutzungszeiten Ihres Abonnements einzusehen. Wenn Sie hier klicken, erhalten Sie detaillierte Anweisungen dazu, wie die Nutzungsinformationen auf Ihren Rechnungen zu lesen und auszulegen sind.

Antworten Sie nicht auf diese E-Mail. Das Postfach wird nicht überwacht. Wenn Sie Kundensupport benötigen, wenden Sie sich an einen Customer Partner Care-Mitarbeiter, indem Sie auf den folgenden
8<--

Name Ressource Verbraucht Inklusive Fakturierbar Satz Betrag
Windows Azure Compute Rechenstunden 43.000000 25.000000 18.000000 0.132000 CHF 2.38
Windows Azure-Speicher Speichertransaktionen (in 10.000) 0.027800 1.000000 0.000000 0.011000 CHF 0.00

This translates to 2.50 CHF which won't break the bank. I am surprised that a couple of Hallo world applications deployed to staging could run up more than 50 hours of compute time. To be sure that the time did not build up any more I delete the application and services. Next I am going to take a much closer look at http://msdn.microsoft.com/en-us/windowsazure/ff796218.aspx and order a book.

Notes from PDC2010

2010-11-11T05:49:00.000-08:00

High Performance Computing (HPC Server)

There is a re-insurance industrial move towards HPC Servers for modeling. RMS and AIR are planning to use HPC Servers as a platform for scheduling number crunching nat cat models.

The news is that the HPC Server Scheduler will support Azure computing nodes in the Cloud. This is very interesting for BUCat because for a large portion of the year the modeling machines are not under heavy load. The advantage of having cloud nodes is that we could scale out during the renewal seasons and scale back when we don’t need the computational power.

Since there is currently no fast connection between cloud nodes you must be careful when using MPI because the performance depends on the lowest connection. To get around this there are small medium and large hpc nodes that can ensure that processing takes place on the same box. I was talking to the ms consultant who will setup hpc for rms. He confirmed that Intel C with MPI would be a better choice of compiler than just MS C.

These Azure nodes are called “Virtual Server Role” and are virtual servers that can be RDPed into. Data can be transferred in a blob or you could host the database on site and use oData wcf services.

I had a discussion about how Sql Azure could be used to process EDM databases. SQL Azure cannot attach mdf database files therefore it is unlikely that we will use this to process EDMs. SQL Azure has some different concepts to SQL Server. For example the idea is to have elastic provisioning of databases. No VMs, no servers, Pay-as-you-go business model and Zero Physical Administration, Linear scaling through database independence. Fan-out expensive computation. For more information goto http://player.microsoftpdc.com/Session/1b08b109-c959-4470-961b-ebe8840eeb84

In the key note there was a presentation called “Pixar's RenderMan using Microsoft Azure by Chris Ford” where Pixar demonstrated their software that they use for rendering their films. Previously they needed a lot of hardware to render the films. Azure allows scalability such that additional hardware can be added or removed within minutes

HPC and Azure have 2 different perspectives on scalability. The Azure team looks at scalability being the ability to expand horizontally in an unlimited manor. The HPC team see that certain algorithms especially MPI really require strong connectivity between the nodes. To achieve this a compromise has been made where new nodes can be small (1 core), medium (4 core) or large (8 core). The Azure team don’t seem to see the need for fast network connectivity between nodes which unfortunately really limits the scalability for algorithms that use MPI. The scenario of scaling compute is quite common, therefore we can expect some movement on this front

Microsoft offers to test your application on their datacenter. They can host up to 8 customers at a time and have access to the product teams if things need tweaking or products need new features.

oData (Open Data Protocol)

oData is a layer that sits on top of EF. The idea is to produce a RESTfull web interface to the database thus making it open for all platforms. oData will include structures for relational data and a method for filtering data. According to Alex John it is not possible to expose Iqueryable functions through wcf. I need to double check this but Alex is the lead on the oData.

For more information on OData click on:

http://player.microsoftpdc.com/Session/a32fdeb8-c878-4bf8-8d5c-6a56309b525c

http://www.odata.org/blog

http://odata.org

The idea behind oData is to be able to embed a query within the URL in a RESTfull way. For example:

http://.../Items?$orderby=DateAdded&$top=10&$select=...

oAuth http://oauth.net

An open protocol to allow secure API authorization in a simple and standard method from desktop and web applications.”

Notes from other blogs http://deanhume.com/Home/BlogPost/custom-odata-api-using-wcf-data-services/44

Entity Framework

I had lunch with the presenter of Code First with EF. This was interesting because we discussed the various methods of building applications. There was an agreement that building the database model first is a valid approach because of the life time of a datamodel exceeds that of an application. There will be some improvements made on the validation. There was a discussion if this should be made within the edm or the poco template. The idea behind the t4 template was that this would somehow make the interface to odata easier. Currently I do this with partial classes and an additional layer over EF. They said that this will improve soon. We also discussed the problems of updating the EDM after updating the database. There are some discussions around how this will improve soon. Soon there will be some additional features that add heuristics to the poco class generator. The idea is that you can set up patterns how to deal with table and field naming like tbl_tableName.

For more information on Code First with EF goto

http://player.microsoftpdc.com/Session/68cfa011-c399-4151-ad9f-748d8723a19d

Within the above presentation a demonstration of the use of DataAnnotations with EF was demonstrated. Here is a small extract of how this makes a POCO look like:

public class User
{
public User()
{
Chirps = new List<Chirps>();
}
public int Id { get; set; }

[StringLength(5, MinimumLength =3) ]
public string Name { get; set; }
// public ICollection<Chirp> Users { get; set; }
public virtuel ICollection<Chirp> Users { get; set; } // For lazy loading
}

public class Chirp
{
public int Id { get; set; }
[StringLength(255, MinimumLength = 1)]
public string Message { get; set; }
public DatTime When { get; set; }
public virtual User User { get; set; }

[StoreIgnore]
public int ChirpActivity {
get
{
return (from c in Chirps
where c.When >DateTime.Now.addDay(1);
select c).count();
}
}
}

Feature Voting: http://ef.mswish.net

Design blog: http://blogs.msdn.com/efdesign/

Team blog: http://blogs.msdn.com/adonet/

Features (yet to come)

· Enums

· Spatial

· Alternate keys

· TVF support

· Migrations and deployment

· Performance & scalability

· Designer improvements

· LINQ improvements

Cloud Computing Workshop

I took part in a cloud computing workshop where participants shared their experiences in cloud computing. The main issues reported in this group where:

· Not knowing exactly on which server and hard disk the data is located. Some auditors need to know this because of the attach that a technician can take data by just removing the hard disk

· Making sure that data stays within the country.

· Configuration control: It is not acceptable to have the operating system update itself without proper testing (configuration management) This can be avoided by using virtual machines because the configuration of the machines is not maintained by Microsoft.

· Support

· Unclear pricing model

· Brand name

Many startup companies use this because it means practically no infrastructure is needed to be maintained in house

Later in the workshops there was some concern about the unlimited cost model. In other words there is currently no way to limit the monthly price. There was some discussion in the workshop about how this could be achieved. My suggestion was to have a burn down chart but it is open as which criteria could be used to slow down the burn down rate. There was also some discussion of what criteria could be used to create new worker instances.

Microsoft announced Visual Studio Team Foundation Server on Windows Azure

A trial can accessed from:

http://backstock.cloadapp.net/Signup

Security can be handled using

· Windows Live ID

· Google

· Yahoo

· CorporateID

Behind the scenes

· Job Agent as a Worker Role

· Team Build as a VM Role

· SQL Azure for tabular data, stored procedures etc

· Blob storage for files and attachments

Microsoft also announced Windows Azure Marketplace formally known as Microsoft Codename “Dallas”. This includes a Data Market currently with 35 Content partners with 60 data offerings including ESRI etc. The idea is to easily discover and explore datasets

https://datamarket.azure.com

For more information on Building, Deploying and Managing Windows Azure Applications

http://player.microsoftpdc.com/Session/19c4c834-3cae-408c-96f0-e87bfad92cff

An example of paired programing in making a deployment utility

2010-11-03T07:25:00.001-07:00

When I wrote this I was sitting in an airplane on my way to the PDC and I cannot sleep. So it is time to write on my blog. Last week in Paris I had a great experience doing paired programming. Here is what we did.

We started off from some Use cases that described things that we want to automate when we make our deployment of the modeling platform. For example:

Asynchronous update of storm foot print files
Synchronous build of server
Synchronous deleting of databases
Etc

Based on these Use Cases we set to work on the sequence diagrams of what we want the deployment tool to do. We came to the notion of a task that would have inputs, results and an execute method. Then we came to the notion of an asynchronous task that would inherit from the task and would have the additional property of a list of targets. We then started to think about the components that would make up the system. Keeping simplicity in mind, or in other words to treat simplicity as a feature we decided that the deployment mechanism should not be distributed unless we find no alternative. So we came up with a TaskDispatcher that would process a series of tasks and a scheduler that could do higher level operations like schedule tasks to run.

So with this ground work we set about building the interfaces that would be needed as input to the tasks and we came up with:

namespace PartnerRe.Deployment.Contracts
{
    public interface IParameters
    {
        string Description { get; set; }
        string SourcePath { get; set; }
        string DestinationPath { get; set; }
        bool FailOnFirstError { get; set; }
        TimeSpan TaskTimeOut { get; set; }
    }
}

We compared this interface with the list of Use cases and found that with some interpretation from the custom task implementations this would be sufficient for inputs required by the Tasks.

Next we looked at the outputs and came up with:

namespace PartnerRe.Deployment.Contracts
{
    public interface IExecutionResults
    {
        bool Success { get; set; }
        string Message { get; set; }
        string LogFilePath { get; set; }
    }
}

Again we considered each use case in turn and came to the conclusion that this would be enough.

Next we thought about how to setup the task structures. We decided to use abstract classes for the Tasks, in this way we have a class structure that is easier to extend. Starting with the Synchonous TaskBase we came up with:

namespace PartnerRe.Deployment.Contracts
{
    public abstract class TaskBase : PartnerRe.Deployment.Contracts.ITaskBase
    {
        public TaskEnumeration Task = TaskEnumeration.None;
        public TaskType TaskType = TaskType.Synchronous;
        public IParameters Parameters;
        public IExecutionResults ExecutionResults;
        public abstract void TaskOperation();
        public void Execute()
        {
            StartTime = DateTime.Now;
            TaskOperation();
            EndTime = DateTime.Now;
        }
        public DateTime StartTime { get; private set; }
        public DateTime EndTime { get; private set; }
    }
}

The implementation of the ITaskBase interface is a little overkill. We decided we need something to describe where the task is synchronous or asynchronous as well as an enumeration to identify what the task is. The Execute function implements a template method pattern where the programmer must implement the TaskOperation Method.

The Asynchonous task looks like:

namespace PartnerRe.Deployment.Contracts
{
    public abstract class AsynchronousTaskBase : TaskBase
    {
        public IList<string> Targets;
        public string CurrentTarget { get; set; }
        public AsynchronousTaskBase()
            : base()
        {
            this.TaskType = Contracts.TaskType.Asynchrounous;
        }
    }
}

Notice we have the additional list of targets and CurrentTarget properties. We thought for a while over whether we would rather implement a list of tasks. We decided that the above was simpler because all we are interested in is a list of results and not a list of tasks that also include the Input parameters and Execution methods. Our idea is to setup one task with place holders for the list of targets

Next we want to implement the Task dispatcher and we came up with the following

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using PartnerRe.Deployment.Contracts;
using System.Threading.Tasks;
using System.Threading;

namespace PartnerRe.Deployment
{
   public class TaskDispatcher
   {
      private List<object> taskDaisyChain;
      private List<object> targets;

      public TaskDispatcher()
      {
          taskDaisyChain = new List<object>;
      }
      public void AddTask<T>(T task)

{
            this.taskDaisyChain.Add(task);
        }
        public void ExecuteTasks()
        {
            for (int i = 0; i < taskDaisyChain.Count; i++)
            {
                TaskBase currentTask = this.taskDaisyChain[i] as TaskBase;

                //object task = this.TaskDaisyChain[i];
                if (currentTask.TaskType == TaskType.Synchronous)
                {
                    currentTask.Execute();
                }
                if (currentTask.TaskType == TaskType.Asynchrounous)
                {
                    if (targets == null)
                    {
                        throw new ArgumentException("Asynchronous tasks needs a list of targets to run on");
                    }
                    if (targets.Count == 0)
                    {
                        throw new ArgumentException("Asynchronous tasks needs a list of targets to run on");
                    }

AsynchronousTaskBase taskAsynchronous = this.taskDaisyChain[i] as AsynchronousTaskBase;
taskAsynchronous.Targets = targets;

IExecutionResults[] executionResults = new IExecutionResults[targets.Count];

                    Parallel.For(0, targets.Count, x =>
                    {
                        TaskFactory taskFactory = new TaskFactory();
                        AsynchronousTaskBase taskParallel = taskFactory.CreateTask(taskAsynchronous.Task) as AsynchronousTaskBase;
                        taskParallel.Parameters = taskAsynchronous.Parameters;
                        taskParallel.CurrentTarget = taskAsynchronous.Targets[x];
                        taskParallel.Targets = taskAsynchronous.Targets;
                        taskParallel.Execute();
                        lock (executionResults)
                        {
                            executionResults[x] = taskParallel.ExecutionResults;
                        }
                    }
                    );

                    taskAsynchronous.ExecutionResults.Message = "";
                    taskAsynchronous.ExecutionResults.Success = true;
                    for (int j = 0; j < targets.Count; j++)
                    {
                        taskAsynchronous.ExecutionResults.Message += executionResults[j].Message;
                        if (!executionResults[j].Success)
                        {
                            taskAsynchronous.ExecutionResults.Success = false;
                        }
                    }
                }
            }
        }
        public void AddListOfTargets(List<string> Targets)
        {
            this.targets = Targets;
        }
    }
}

We decided that we would need a queue of tasks implement above as a daisy chain. There is some scope for refactoring for example we have implemented another list of targets here. But the design principles are sound. Also probably the lock in the parallel for is over kill but it is a shared variable and in this case the time to execute the tasks are far greater than the time lost in locking the results. The syntax around the parallel for was a little different, we did not find many examples for to implement an itterator.

We implemented a one way file synchronization task that looks like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using PartnerRe.Deployment.Contracts;
using System.IO;
using System.Diagnostics;
using System.ComponentModel;
using System.ComponentModel.Composition;

namespace PartnerRe.Deployment
{
    [Export(typeof(TaskBase))]
    public class SynchronizeDirectoryTask : AsynchronousTaskBase
    {

private string WorkingDirectory = "C:\\Program Files\\Windows Resource Kits\\Tools";

        public SynchronizeDirectoryTask() : base()
        {
            this.Task = TaskEnumeration.SynchronizeDirectory;
            this.Parameters = new Parameters();
            this.ExecutionResults = new ExecutionResults();
        }

        public override void TaskOperation()
        {
            if (this.Task == TaskEnumeration.None)
            {
                throw new Exception("The programmer forgot to set the Task enumeration in the Task constructor");
            }
//… Lots more tests with ArgumentExceptions

//C:\Program Files\Windows Resource Kits\Tools\Robocopy.exe
// robocopy Source Destination *.* /XO

this.Parameters.DestinationPath = this.Parameters.DestinationPath.Replace("<SERVER>", this.CurrentTarget);

            ProcessStartInfo pCopy = new ProcessStartInfo();
            pCopy.WorkingDirectory = WorkingDirectory;
            pCopy.FileName = "Robocopy.exe";
            pCopy.UseShellExecute = false;
            pCopy.RedirectStandardOutput = true;
            pCopy.RedirectStandardError = true;
            pCopy.Arguments = this.Parameters.SourcePath + " "+this.Parameters.DestinationPath+" *.* /XO";
            Process proc = Process.Start(pCopy);

string output = proc.StandardOutput.ReadToEnd();
proc.WaitForExit();

string error = proc.StandardError.ReadToEnd();
proc.WaitForExit();

            this.ExecutionResults.Message = output;
            this.ExecutionResults.LogFilePath = "\\\\" + this.CurrentTarget + "\\c$\\MyLog";
            this.ExecutionResults.Success = true;

}
}
}

Here we wanted to program as little as possible, so we took Robocopy and redirected the standard output. We also implemented a MEF Export. The corresponding object factory looks like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using PartnerRe.Deployment.Contracts;
using System.ComponentModel;
using System.ComponentModel.Composition;
using System.ComponentModel.Composition.Hosting;

namespace PartnerRe.Deployment
{
    public class TaskFactory
    {
        public TaskFactory()
        {
            DirectoryCatalog directoryCatalog = new DirectoryCatalog(@".");
            CompositionBatch batch = new CompositionBatch();
            batch.AddPart(this);
            CompositionContainer container = new CompositionContainer(directoryCatalog);
            container.Compose(batch);
        }

[ImportMany(typeof(TaskBase))]
private IEnumerable<TaskBase> Tasks;

        public TaskBase CreateTask(TaskEnumeration TaskType)
        {
            foreach(TaskBase t in Tasks)
            {
                if (t.Task == TaskType)
                {
                    switch (t.TaskType)
                    {
                        case Contracts.TaskType.Asynchrounous:
                            {
                                return t as AsynchronousTaskBase;
                            }
                        case Contracts.TaskType.Synchronous:
                            {
                                return t as TaskBase;
                            }
                    }
                }
            }
            throw new Exception("This task has not yet been implemented in the TaskFactory");
        }
    }
}

The idea is that when we add new tasks we want them to be imported automatically into our application with the least amount of manual programming as possible. In this case we only need to add to the TaskEnumeration each time we add a new task.

We developed using Test Driven Development, this was used as a way to design the API. In the end our test looked like:

[TestMethod]
public void SynchronizeDirectory_UnsynchronizedFiles_TargetFilesSyncronized()
{
// Arrange

DestinationDir = "\\\\<SERVER>\\D$\\SOURCE\\PartnerRe.Deployment\\TestFiles\\Source";

    TaskDispatcher taskDispatcher = new TaskDispatcher();
    SynchronizeDirectoryTask synchronizeDirectoryTask = new SynchronizeDirectoryTask();
    synchronizeDirectoryTask.Parameters.DestinationPath = DestinationDir;
    synchronizeDirectoryTask.Parameters.SourcePath = SourceDir;

taskDispatcher.AddTask<TaskBase>(synchronizeDirectoryTask);
List<string> Targets = new List<string>();

    Targets.Add("CHZUPRELR886W9X");
    //Targets.Add("Server2");
    taskDispatcher.AddListOfTargets(Targets);

// Act
taskDispatcher.ExecuteTasks();

//// Assert
Assert.IsTrue(synchronizeDirectoryTask.ExecutionResults.Success);

}

As can be seen the test is not complete, but the method is sound.

In this case the experience I made with Paired programming showed it is an efficient method of programming. It has the advantage that it integrates at least 2 people directly into the decision making process of building and refactoring an architecture while sharing knowledge. The only time that I found paired programming not to be appropriate is when the programming method is unclear, for example when intensive googling is needed to resolve a small detail.

Making SQL Queries statement act on the same database across multiple servers

2010-09-22T12:03:00.001-07:00

Here’s a useful way to apply queries across a number of sql 2008 servers

1. Open Microsoft SQL Server Management Studio
2. Under Database Engine select Central Management Servers
(It is on the tab "Registered Servers" next to the tab "Object Explorer"
3. Right Mouse click "Central Management Servers" and select Register Central Management Server...
4. Choose the Server that you want to add in the "New Server Registration dialog"
This has the effect of adding a node directly under "Central Management Server". This is used to host the Server groups
5. Right Mouse click on the new node and select "New Server Group..."
6. Give a group name
7. Right mouse click on the new group and select "New Server Registration..."
8. Select the servers that you want to be part of the group

This completes the setup.

To create queries that act across the registered servers
1. Right mouse click on a server group
2. "New Query"

When you execute the query you will receive the results with an additional "Server Name" column giving the result of that query on that server

Setting up Transaction log shipping

2010-09-22T11:59:00.001-07:00

We have a high performance physical sql server that we want to backup via netApp shares. To get this to work we setup a second virtual sql server that uses netapp shares which are backup using the Snap manager tool.

Data is sent to the virtual sql server using Transaction log shipping. Here is how to setup transaction log shipping

1. Create a Backup of the database making shure that:
   - Recovery model = Full
   - Backup type = Full
2. Restore this backup onto the virtual sql server choosing the second option:
   - Leave the database non-operational, and do not roll back transactions. Additional transaction logs can be restored (RESTORE WITH NORECOVERY)
3. After the backup is complete there is a green upward pointing arrow on the database on the virtual server and the database has (restoring..) after it.
4. Goto the physical server, right mouse click the database and select properties
5. In the select page choose "Transaction Log Shipping"
6. tick the check box "Enable this as primary database in a log shipping configuration"
7. Click the button Backup Settings
   This will open "Transaction Log Backup Settings" dialog
8. In the Network path to backup folder give the path on the physical server where the tranaction logs are to be found eg
   \\Myserver\LOGSHIPPING$
   In the field "If the backup folder is located on the primary server, type the path to the folder"
   G:\LOGSHIPPING
   OK
9. In the Secondary server instances and databases press the Add button
10. Press the connect button and select the secondary database
11. In "Destination folder for copied files"
    G:\LOGSHIPPING
12. OK
13. Check that the backup and restore jobs created on both servers run successfully