Some ideas about Optimization and Restructuring

I am looking into restructuring and optimization of a catastrophe modeling platform. Here are some of the ideas that have come to mind.

The Aggregate Cat models consist of a disaggregation where the sums insured for locators are redistributed according to population fraction. Then some unmanaged mathematical routines are run that determine the loss ratio for each risk followed by the application of treaty structures. We will look into precalculating the loss ratios for each combination of locator, risk and vulnerability, in this way we can remove the disaggregation and ground up loss calculation from the productive code which will significantly improve the speed and simplify our aggregate models for all perils to run through the same code. The size of the lookup tables depend on the number of vulnerability functions. The models have various numbers of vulnerability functions meaning that the sizing of the hardware will depend on the model with the most vulnerability functions.

The optimization of the detailed models depends on the way that the loss is calculated for each peril. For example Wind storm calculations are based on storm footprint files that are a grid of information that is overlaid onto the exposure. In this case we could use the same approach as the aggregate models except we would need to determine from the Latitude and Longitude which storm footprint grid point is to be used. This means that we could use the geospatial queries that came with sql 2008. For Earthquake there are no grid points because the damage propagates in concentric circles from the earthquake. We could think of make a pseudo detailed model but there are about a thousand vulnerability functions meaning the database will be big and the time needed to precalculate the loss ratios will be long. So for Earthquake we need to optimize that mathematical component that calculates the loss ratio to run under a multi core 64 bit O/S. We will be considering:

• Using C# with immutable data
• Using F# or a combination of f# and C#
• Convert the existing C code to 64bit multi core
  

On the non science side of the platform we are thinking of replacing the pull based job scheduling with WCF Services that take parameters such as the DLL and instructions of where to read and write data. The idea is to asynchronously call this wcf service. This means the client implements an Event handler that is called on completion. A Scheduler is used to send jobs to the WCF Services distributed on the modeling machines. This approach improves the software distribution because only the code for a very generic WCF service is deployed on the modeling machines. The DLLs doing the work are distributed at run time from the Scheduler. But the scheduler is a single point of failure, so it would be best made on the central database server. The development machine would have the code for the scheduler as well as the WCF Service for running the DLLs that are doing the work.

At the moment the Server side consists of 140 sub projects. Instead of decomposing an application into sub projects we will create a few general projects but using folders instead of sub projects. This has the advantage that the program compiles faster and executes faster. Such a project could consist of:

• GraphicalUserInterface
• Utilities (eg Zip, log4Net)
• Interfaces
• DataAccessLayer
• BusinessObjects (Types)
• BusinessLogic

Sharing of objects between the client and the server:
1. The Server would reference the Client projects as done now
2. We could make WCF Services hosted on IIS7 on the scheduling server
3. We could load balance WCF Services if the clients have a heavy computational demand.

We will consider the Microsoft has a load Balancing implementation as a part of Windows 2008. If we need to load balance then we should use this because programming your own has a lot of things that can go wrong.

Implementation strategy:
My approach would be to progressively transition the live system into the target architecture. Since the live system must function every step of the way has exhaustive tests that guarantee that progress is made. I find the danger of a basic rewrite is that testing will never be a thorough as on a real productive system which means when the prototype is made live there will be a lot of bugs to fix.
Logging
As we restructure we will build in a configurable logging intensity so that when something is being debugged intensive debugging is possible and during normal operation minimal debugging is made.

Here is how we are thinking of splitting the Tiers of the application
1. GUI Layer
eg Has functions like GetLossFileView etc
2. Business Layer
eg LossFileBO as entity
3. Data access layer Client
eg WCF Services using generic functions with entities defined in the Business Layer
4. Data access layer server
eg WCF Services using generic functions directly accessing the database.
The DAL would use a Factory using the Type Off construct that will enable intellisence through the entity data types

Versioning
1. Using MEF: Probably not worthwhile for PRECCP, because this would require very careful source control branching and consequently the maintenance work increases
2. Using an interface and built in database functions: Depends on the level at which you want to track versions. Eg it does not make sense to track the version of events in loss files directly.
3. Just keep track of versions: This is the simplest method and what we are doing now. Currently if you ask underwriters the only versions that are interesting are the current version, the previous version and the next version.

A story of pifalls in hosting webservices or WCF Services on IIS6

It’s been a while since I made some WCF Services and even longer thatI had to do much with IIS. Here is a log of what I experienced as I was implementing some WCF functions in a LAN. Often I found web sites that had exactly the same problems that I had faced but no solution.

I have a number of servers on which I need a WCF Service. But I don’t want to enable IIS on all these servers, so I decided to host the WCF service in a Windows Service. Here are some links about this:

All informations about Windows Services:
http://msdn.microsoft.com/en-us/library/aa984074(VS.71).aspx

Walkthrough: Creating a Windows Service Application
http://msdn.microsoft.com/en-us/library/aa984464(VS.71).aspx

How to: Host a WCF Service in IIS
http://msdn.microsoft.com/en-us/library/ms733766.aspx

Hosting and Consuming WCF Services
http://msdn.microsoft.com/en-us/library/bb332338.aspx


Here are the steps I took to create a Windows Service:

1. Open Visual Studio and Create a Windows Service Project
2. Click in the design pain of Service1.cs
3. In the properties
Name = NigelsService
ServiceName = NigelsService
4. Drag and drop Eventlog from the toolbox into the design pain of Service1.cs
5. Edit the following code:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.Linq;
using System.ServiceProcess;
using System.Text;

namespace WindowsService1
{
public partial class NigelsService : ServiceBase
{
public NigelsService()
{
InitializeComponent();
// Create the source, if it does not already exist.
if (!EventLog.SourceExists("MySource"))
{
//An event log source should not be created and immediately used.
//There is a latency time to enable the source, it should be created
//prior to executing the application that uses the source.
//Execute this sample a second time to use the new source.
EventLog.CreateEventSource("MySource", "MyNewLog");
}

// Create an EventLog instance and assign its source.
this.eventLog1.Source = "MySource";

// This is exected on Power up
// Instanciate WCF service here


}

protected override void OnStart(string[] args)
{
// This is executed when the service starts
eventLog1.WriteEntry("Service Start");
// Start WCF service here
}

protected override void OnStop()
{
// This is executed when the service Stops
// (Power down is the garbage collector

eventLog1.WriteEntry("Service Stop");
}
}
}

6. Right mouse click in the design pain of Service1.cs and select Add Installer
7. Click the ServiceInstaller1 and change the following properties
Name = NigelsServiceInstaller1
ServiceName = NigelsService
StartType = Automatic
8. Click on ServiceProcessInstaller1 and change the following properties
Account = LocalSystem
9. Click File / add / New Project
10. Under Setup and Deployment select Setup Project
11. Right mouse click on the project root Setup1
12. Add / Project Output ...
13. Project = WindowsService1
Primary Output
Configuration Active
OK
14. Right mouse click on the project root Setup1
15. View / Custom Actions
16. Right Mouse click on the root of Custom items and select Add Custom Action..
17. Look in : Application Folder
Primary output from WindowsService1 (Active)
OK
18. Build the project

When installing the service you will notice that the Product name is used
When the service is started and stopped corresponding entries are made in the Application Event log

This worked very well because the service did not have much to do. I also had a service that had quite a lot of work to do. For this service I did not want to use a Windows Service because I wanted to make use of IIS’s ability to look after memory management and restarting the service if something goes wrong. I had an IIS6 server on to which I could deploy my application

Firstly I created a solution with a WCF Solution and for old times sake I created an additional WebService project.

Starting with WebService hosted on ASP.NET. Visual studio made a sample asmx file which I quickly modified to include all the functions that I needed. And it worked. So I renamed the Service1.asmx to something more appropriate and I got the following error

Parser Error
Description: An error occurred during the parsing of a resource required to service this request. Please review the following specific parse error details and modify your source file appropriately.
Parser Error Message: Could not create type 'PRECEDWebService.Service1'.
Source Error:
Line 1: WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.Service1"
Source File: /PRECEDWebService/PRECED.asmx Line: 1

So the fix for this was to change
WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.Service1"
to:
WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.PRECED"

The next step was I wanted to deploy this to my IIS6 Server. Its been about 7 years since the last time I really made something on IIS. So I was a little rusty. First on the server I setup a Virtuel directory in the IIS Manager and published my new webservice into it. But when I tried to invoke the Webservice I get a get an error.
By default all Web service extensions are disabled. I found some instructions in the following link:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/de6171d8-704c-431a-8117-915e02fac30b.mspx?mfr=true

To enable and disable a Web service extension
1. In IIS Manager, expand the local computer, and then click Web Service Extensions.
2. In the details pane, click the Web service extension that you want to enable or disable.
3.To enable a disabled Web service extension, click Allow.
4.To disable an enabled Web service extension, click Prohibit.
A message box with a list of applications that will be prevented from running on the IIS Web server displays.
5.Click OK to disable the Web service extension.
You will need to enable
- All Unkenown ISAPI Extentions
- Active Server Pages
- ASP.NET v2.0.50727

This time when trying to add a WebService I got the following error
The current identity (NT AUTHORITY\NETWORK SERVICE) does not have write access to 'C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files'.

To fix this
1. Goto IIS
2. Open the application Pools
3. Click on the properties of DefaultAppPool
4. Click on Identity
5. Change Predefined to Local System
6. Restart web server
Also I granted IUSR_ServerName access rights to ../Windows/Microsoft.NET/Framework/version
In addition I found that if the IIS had not been correctly configured the following command line statement would reinstall a number of components
C:\WINDOWS\Microsoft.NET\Framework\v3.0\Windows Communication Foundation>service
modelreg.exe –r

This worked quite nicely, but if the server is internet facing for security reasons I would not use IIS7 and do some research over security issues.
So next I looked again at the WCF solution to see if I could do the same. Starting off by trying to consume a WCF with the solution. The first thing I did was to change the binding type to basicHttpBinding. The first time I did this I used a capital B in the web.config file and got an error about an unrecognized binding type. I wish the compiler would also correct my config files! The web.config file ended up looking like:

client
endpoint address="http://localhost:1271/PRECED.svc" binding="basicHttpBinding"
bindingConfiguration="EndPointHTTP" contract="ServiceReference4.IPRECED"
name="EndPointHTTP"
/client

The corresponding code to call this with app.config looked like
Dim iPRECEDClient As New ServiceReference5.PRECEDClient
Console.WriteLine(vbLf & "Calling Service..." & vbLf)
Dim str As String
str = iPRECEDClient.HelloWorld
Console.WriteLine(str)
Console.ReadLine()
Alternatively without app.config

Dim proxy As IPRECED = ChannelFactory(Of IPRECED).CreateChannel(New BasicHttpBinding(), New EndpointAddress("http://localhost:1271/PRECED.svc"))
Console.WriteLine(vbLf & "Calling Service..." & vbLf)
Console.WriteLine(proxy.ValidateUser())

Console.ReadLine()

The final step was to publish this to the web server. I replaced http://localhost:1271/PRECED.svc with http://servername/VirtuelDir/PRECED.svc%20in%20both%20the%20Web.config and the app.config and everything worked well
For completeness I added a MEX endpoing as shown below

system.serviceModel
services
service behaviorConfiguration="PRECEDBehaviors" name="PRECEDWcfService.PRECED"
endpoint address="http://localhost:13243/" binding="basicHttpBinding"
name="EndPointHTTP" contract="PRECEDWcfService.IPRECED" /
/service
service behaviorConfiguration="PRECEDBehaviors" name="PRECEDWcfService.PRECEDMex"
endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" /
/service
/services
behaviors
serviceBehaviors
behavior name="PRECEDBehaviors"
serviceMetadata httpGetEnabled="true" /
/behavior
/serviceBehaviors
/behaviors
/system.serviceModel
This was quite a long and painful exercise but that was because I been working on thick clients apps and not web apps in the last 7 years. One nice trick I learned on the way was to use the browse function on the design time webserver because when you double click on the svc files you get a good error messages.

Here is how to create a synchronization with a database and cached data1

Here is a quick example of how to used the synchronization framework and cached data on the client:

1. Create a Windows Forms application
2. Under the tool bar select Data/Add New Data Server
3. Select database
4. Click the new connection button and select the database to connect to
5. Don't save the connection as
6. Select the tables to synchronize Select Enable local database caching (This creates a mini local database)
7. Select Synchronization Mode Incremental (Snap shot means every time the database is synchronized all data is downloaded ) Select Use SQL Server change tracking
8. Select Update server for incremntal changes
Save SQL scripts in the project for later use (This is useful if you move the database)
After this you will get a file like TEST_AynchCache.syncIf you double click on this you get a hyperlink to "Show Code Example".
This code can be used to perform the synchronization.
Here is an example of how to populate and synchronize a data grid view

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Me.TST_SLAVETableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_SLAVE)
Me.TST_MASTERTableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_MASTER)
End Sub

Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim syncAgent As TEST_SYNCHCacheSyncAgent = New TEST_SYNCHCacheSyncAgent()
Dim syncStats As Microsoft.Synchronization.Data.SyncStatistics = syncAgent.Synchronize()
Me.TST_SLAVETableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_SLAVE)
End Sub

syncStats.TotalChangesDownloaded contains the number of changes that where made

My first steps at Synchronizing SQL2008 databases

Here are some steps I have taken in testing the synchronization of databases using SQL

Install SQL-Server Replication
1. Run the MS SQL Setup.exe
2. Select Installation in the left hand panel
3. Click New SQL Server stand-alone installation or add features to an existing installation
4. Setup Support Files
Click install
5. Setpup Support Rules click next
6. Click Add features to an existing instance of SQL 2008
Select MSSQLSERVER
Next
7. Select Database Engine Service/SQL Server ReplicationNext
8. Disk space requirements
click next
9. Error and usage reporting
click next
10. Installation rules click next
11. Ready to install click Install
12. Installation progress click next
13. Complete click close

Setup the publisher
1. Open Microsoft SQL Server Manager Studio
2. Browse to Replication/Local Publications
3. right mouse click on Local Publications and select New Publication
4. New publication wizard click next
5. Select 'YourServerName' will act as its own Distributor; SQL Server will create a distribution database and log
Next
6. Select Yes, Configure the SQL Server Agent to start automatically
Next
7. Leave the snap shot folder in it's default location
C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\ReplDataNext8. Select the database that you want to synchronize
Next
9. Publication Type.
Transactional publication:The Publisher streams transactions to the Subscribers after they receive an initial snapshot of the published data.

Transactional publication with updatable subscriptions:The Publisher streams transactions to SQL Server Subscribers after they receive an initial snapshot of the published data. Transactions originating at the Subscriber are applied at the Publisher.

Merge publication:The Publisher and Subscribers can update the published data independently after the Subscribers receive an initial snapshot of the published data. Changes are merged periodically. Microsoft SQL Server Compact Edition can only subscribe to merge publications.

Select Transactional publication with updatable subscriptions
Next

10. Article
Select all the Tables including sysdiagrams(dbo)
Next
11. Articles issued click next
12. Filter Table Rows click
next
13. Snap Shot Agent select Create a snap shot immediately and keep the snapshot available to initialize ssubscriptions
Next
14. Agent Security
Click Security settings for SnapShot Agent and Queue Reader Agent
15. For both steps in 14 I selected
Run under the SQL Server Agent Service account (This is not recommended security best practice)
(I dont want to manage a lot of different service accounts so I choose just the account I use the one I configured for SQL Agent)
16. Wizard Actions Select
Cretae the publication
Next
17. Click Next and the Create Publication wizard will start
18. Close

Setup the Subscription
(Before starting create a linked server to the publisher that uses the current security context, it's needed in step 15)
1. Open Microsoft SQL Server Manager Studio and connect to the target server
2. Browse to Replication/Local Subcriptions
3. right mouse click on Local Subscriptions and select New Subscription
4. New subscription wizard click next
5. Select
6. Open the SQL Server that has the published database
7. Select the database and publications that was just created
8. Distribution Agent Location
Select Run each agent at it's Subscriber (pull subscriptions)(The alternative is to run all agents at the distributor (push subscriptions) which can be easier to administer)
Next
9. Subscribers Choose the Subscription database where the data will go(At this point you can also create a new database)
10. Distribution Agent SecurityClick the (...) button
11. Run under the SQL Server Agent Service account (This is not recommended security best practice)
(I dont want to manage a lot of different service accounts so I choose just the account I use the one I configured for SQL Agent)
OK
12. Distribution Agent Security Click Next
13. Synchronization Schedule Select Run continously (the alternative is to run on demand or to define a schedule)
Next
14. Updateable SubsscriptionsSelect Simultaneously commit changes (the alternative is to queue changes and commit when possible)
Next
15. Login For Updateable scriptionsSelect Use a linked server or remote server that you have already defined
16. Initialize SbscriptionsSelect Immediately
Next
17. Wizard actions select Create the subscriptions
Next
After this something like this will apear
• Create subscription(s).
Create a subscription to publication 'TEST' from Publisher 'CHZUPRES913'.
Create subscriptions at the following Subscriber(s):
CHZUPRES912
• Subscription database: TEST
• Agent location: Subscriber
• Agent schedule: Run continuously
• Agent process account: SQLServerAgent service account
• Connection to Distributor: Impersonate process account
• Connection to Subscriber: Impersonate process account
• Initialize: Immediately
• Replicate changes: Simultaneously commit changes
18 Creating Subscription(s)Close

Code to determine number of CPUs

To get the most out of multicore asynchronous the number of available cpus is need so as to determine the right amount of threads to use. Here are some examples of how to determine this

From Sql the following stored proceedure dilivers a lot of information including the number of cpus
EXEC xp_msver

From C# the following can be used

using System;
using System.Collections.Generic;
using System.Linq;using System.Text;
using System.Management;using System.Diagnostics;
namespace NumberOfProcessors
{
class Program
{
static void Main(string[] args)
{
ManagementObjectSearcher mgmtObjects = new ManagementObjectSearcher("Select * from Win32_ComputerSystem");

foreach (var item in mgmtObjects.Get())
{
Console.WriteLine("Number Of Processors - " + item["NumberOfProcessors"]);
try
{
Console.WriteLine("Number Of Logical Processors - " + item["NumberOfLogicalProcessors"]);
}
catch (ManagementException)
{ }
}
List processes = Process.GetProcesses().ToList();
Console.WriteLine("All running process");
processes.ForEach(x => Console.WriteLine(x.ProcessName));
Console.ReadLine();
}
}}

Some code about available disk space

My application runs across a server farm. I need to know when the disk capacity is running low. My first thought was to use SNMP traps that are built into HP servers and can monitor everything down to the temperature of the mother board. Unfortunately the servers do not have this configured and I am a little reluctant to start a project on Server monitoring especially since this is not my primary job. So I looked into what the SQL Server had to offer. I found the following quite usefull

EXEC xp_fixeddrives
Returns [drive] and [MB free]

EXEC xp_msver
Returns information about the current version of SQL server installed

exec master.dbo.xp_sqlagent_enum_jobs @is_sysadmin = 1, @job_owner = ''
Returns the jobs running under the SQL Agent

The first idea was to use the central SQL Server and use LinkedServers forwarding the user credentials. This this not work since the user has not sufficient access rights and that we did not want to hard code user name and password in the linked server connection

Since we don’t have IIS on these machines we decided for a Windows Service that hosts a WCP application. The Windows Service part of this looked like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.ServiceProcess;
using System.Text;

namespace ServerMonitorWindowsService
{
static class Program
{
///

/// The main entry point for the application.
///

static void Main()
{
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new ServerMonitorWindowsService()
};
ServiceBase.Run(ServicesToRun);
}
}
}


using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.Linq;
using System.ServiceProcess;
using System.Text;
using System.ServiceModel;

namespace ServerMonitorWindowsService
{
public partial class ServerMonitorWindowsService : ServiceBase
{
private Type serviceType;
private ServiceHost host;

public ServerMonitorWindowsService()
{
InitializeComponent();
serviceType = typeof(ServerMonitorService.ServerMonitorService);
}

protected override void OnStart(string[] args)
{
host = new ServiceHost(serviceType);
host.Open();
}

protected override void OnStop()
{
host.Close();
}
}
}

We used the tcp protocol and a WCP function that looked like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.Text;
using System.Management;

namespace ServerMonitorService
{
public class ServerMonitorService : IServerMonitorService
{

#region IServerMonitorService Members

public ulong GetFreeSpace()
{
return GetFreeDiskSpaceInBytes("d");
}

#endregion

private static ulong GetFreeDiskSpaceInBytes(string drive)
{
ManagementObject disk = new ManagementObject("win32_logicaldisk.deviceid=\"" + drive + ":\"");
disk.Get();
return (ulong)disk["FreeSpace"];
}
}
}

R script to look at data with a lot of columns

I had some performance data that I needed to analyse that had over 1000 rows. Excel cannot load such a file, so I resorted to some tools that mathematicians and scientists use. In this case I used the following R script. There are some interesting twists, for example when R comes across a character it does not recognise then you need to replace the character with ".". For example R does not like spaces, "(", ")" etc... Anyway this script is very usefull for analysing large csv files...

data <- read.csv(file="C:/Data/Work/EXposure+IT/ITBermuda/NetApp/PartnerRe_000005.csv",header=TRUE,sep=",")colnum <- c(116,137,97,118)

sstr <- c(
"BMHMPRES911.PhysicalDisk.0.C..D...Current.Disk.Queue.Length"
,"BMHMPRES911.PhysicalDisk.0.C..D...Avg..Disk.Queue.Length"
,"BMHMPRES911.PhysicalDisk.0.C..D...Disk.Read.Bytes.sec"
,"BMHMPRES911.PhysicalDisk.0.C..D...Disk.Write.Bytes.sec"
,"BMHMPRES911.PhysicalDisk.1.G...Current.Disk.Queue.Length"
,"BMHMPRES911.PhysicalDisk.1.G...Disk.Read.Bytes.sec"
,"BMHMPRES911.PhysicalDisk.1.G...Disk.Write.Bytes.sec")

results <- list()
for(i in 1:length(sstr)){
results[[i]] <- grep(sstr[i],names(data))
}

# make a time series#options("digits.secs") <- 3
#datetime <- as.character(data[,1])
#datetime <- strftime(datetime, format="%m/%d/%Y %H:%M:%S")time <- seq(1,1565)

for(i in 1:length(results)){
par(ask=F)

png(filename=paste("C:/Data/Work/EXposure+IT/ITBermuda/NetApp/",i,".png",sep=""))
plot(time,data[,results[[i]]],type="l")
title(main=names(data)[results[[i]]])
dev.off()
}