Up until a few weeks ago I believed that it does not matter whether code is compiled in Managed or unmanaged code. I have a friend who is doing a lot of iterative maths via a library written in MS C++. I was quite sure that this was sub optimal because pinvoke and the marshalling of data is expensive. So He went away and made a comparison with a reference problem on one of our modelling servers that involved 2 huge for loops. The outcome was that for this problem the in line VB.NET code was around 1.5 time faster that the VB.NET code that called the MS C++ library. So I was thinking great, next step lets use F-Sharp, after all it’s designed for scientists and parallelization is simple. Since my friend comes from the world of science he did one more test with Fortran 90 running on a Linux operating system and it was 100 times faster on a low spec machine he had at home. So this blew the idea of making this algorithm in managed code be it F-Sharp or C-Sharp completely out of the water. In fact with an MPICH Message Passing Interface with 4 nodes corresponding to the 4 cores of his CPU we got a 400 times increase.
We investigated further and it turns out the Fortran compiler had some HPO (High Performance Operations) that included the intelligent analysis of the code to do vectorisation and parallelization. Then we downloaded the Intel C++ and the Intel Fortran compiler because these had similar compiler optimizations. It turned out that we got a similar performance as the Linux version, which was a great relief because it illuminated the Operating System. The Intel Fortran90 compiler was just slightly faster than the Intel C++ Compiler. Apparently this is because it is difficult for the compiler to recognize patterns that can be safely optimized because C++ is based on pointers. Where as good old fortran is designed for simple array maths. In fact the name Fortran comes from FORMular TRANslator.
I visited the Intel stand at the PDC and they have a very good product that integrtates seamlessly into Visual Studio and they have a lot of tools designed to debug concurrent programs. For example there is a tool that highlights data races and deadlocks, also there is a tool in a similar style to Visual Studio 2010 that enables the stepping through of code on multiple tasks. Parallel Composer, Inspector and Parrallel Amplifier are very good tools for writing and debugging parallel C++ code. Just recently they launched a web site on a large spec machine that can test you application on varying numbers of cores.
Another nice thing about Intel C++ is that it is task based and just like in .Net 4.0 implements a task stealing architecture that lessons the need for equally sized amounts of work. Also it has a parrell.for that helps the compiler identify loops that can be parrelized. The MPI is used for algorithm’s that have boundary conditions, such as finite element analysis. This is required to pass the boundary conditions between the nodes via Messages. This is also needed to go across machine boundaries.
At the Key note of the PDC a comparison was made between CPUs and GPUs. The GPU is a lot bigger than the CPU and contains a lot more cores 32 or 64. With hyper threading you have around 1000 threads to play with. This can give 1 Tflop performance which is similar to a cray 5 years ago. It’s funny to think all this power was developed for gaming. With DirectX it is possible to use this functionality. There is a compiler and a special language that creates an Intermediate language in a .o file. This is compiled into hardware specific machine code with a Just in time compiler. The coding is somewhat cryptic but could be worth it. With such super high performance it is possible to do other types of analysis such as Monte Carlo Simulations. Wall street has taken a big interest in this. Therefore it will not be long until there are multiple GPUs on one server. Intel is creating a new cpu that has a GPU built in. This would be ideal because the Intel compilers will probably take advantage of this without the programmer having to think too much about it. The clock speed of a GPU is much slower than a CPU and also generates less heat. One down side is that you must be very careful when designing the data centric algorithms because for every gpu cycle that is missed more than a 1000 operations are lost in other words a massive slow down. One more thing to be careful about is that not all gpus support double precision. The reason is that games produces don’t need this level of precision and that to save costs they settle for single precision.
I also looked into the Microsoft HPC Server. This is a grid computing infrastrure that comprises of a Scheduler and Nodes. A broker can be used with WCF to give things a SOA flavour. This is a low cost grid that seems to be very flexible. The example given was an excel spread sheet that calculated derivatives. With HPC the expensive calculations can be spread across machines and cores.
Cross computer parrallization has different considerations than just multi core. For example in multi core programming you need to pay attention to the cache invalidation problems as you iterate through integer arrays. Cross computer computing has latency, therefore you need to be careful not to chop up the work in the tasks to finely. On the other hand there is isolation. There is a performance concideration concerning passing data greater than 64k on the HPC Server. The problem is that serializing this data is slow which means you need to send a reference where the data is being stored.
Saturday, 21 November 2009
Wednesday, 21 October 2009
Migrating a .net application to a 64 bit Server
Viusual Studio compiles VB and C# to an intermediate language that by default can run on any cpu. The problem comes when the application accesses some unmanaged DLLs. I got the error
An attempt was made to load a program with an incorrect format. {Exception from HRESULT: 0x8007000B)}
On Google I found something quite similar http://social.msdn.microsoft.com/forums/en-US/netfx64bit/thread/35b09f74-1d8e-4676-90e3-c73a439bf632/
In this article the error was
errMsg = "System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B) at Wcr_.NET_2005.Module1.dllBIN2_iv(Double price_u, Double ex, Double d_exp, Double d_v, Double price, Double rate_ann,...
The correct answer was
"BadImageFormatException" is thrown on you when the application loads a DLL having incompatible bitness, that is a, 64 bit exe tried to load a 32bit DLL or, a 32 bit exe tried to load a 64 bit DLL.In your case, it's probably the first, so please make sure you compile your vb.net exe as 32 bit (X86) and NOT MSIL.Note also that you should stay away from SysWow64 and System folders, they are reserved by the system and should not contain user applications/DLL's, user stuff must reside under "program files" or "program files (X86)" in an another private folder.So I would suggest you delete all this from the system folders, and move the DLL and the exe to a private folder under "program files (X86)". The lib file has nothing to do with this as it should have beed statically linked when the dll was build.
I solved the problem by following these steps
1. Goto the solution properties
2. Open the Configuration Properties tress and press the configuration Manager button
3. Change Active solution platform from Any CPu to
4. Change Type from Itanium to x86
I ran some tests and my first impression is that the DLL seems to be running much slower than on a 32 bit OS, but I am not sure if this is because of virtuel instead of physical hardware or if it is a problem with the 32 bit mode of a Windows 2003 64 bit OS.
An attempt was made to load a program with an incorrect format. {Exception from HRESULT: 0x8007000B)}
On Google I found something quite similar http://social.msdn.microsoft.com/forums/en-US/netfx64bit/thread/35b09f74-1d8e-4676-90e3-c73a439bf632/
In this article the error was
errMsg = "System.BadImageFormatException: An attempt was made to load a program with an incorrect format. (Exception from HRESULT: 0x8007000B) at Wcr_.NET_2005.Module1.dllBIN2_iv(Double price_u, Double ex, Double d_exp, Double d_v, Double price, Double rate_ann,...
The correct answer was
"BadImageFormatException" is thrown on you when the application loads a DLL having incompatible bitness, that is a, 64 bit exe tried to load a 32bit DLL or, a 32 bit exe tried to load a 64 bit DLL.In your case, it's probably the first, so please make sure you compile your vb.net exe as 32 bit (X86) and NOT MSIL.Note also that you should stay away from SysWow64 and System folders, they are reserved by the system and should not contain user applications/DLL's, user stuff must reside under "program files" or "program files (X86)" in an another private folder.So I would suggest you delete all this from the system folders, and move the DLL and the exe to a private folder under "program files (X86)". The lib file has nothing to do with this as it should have beed statically linked when the dll was build.
I solved the problem by following these steps
1. Goto the solution properties
2. Open the Configuration Properties tress and press the configuration Manager button
3. Change Active solution platform from Any CPu to
4. Change Type from Itanium to x86
I ran some tests and my first impression is that the DLL seems to be running much slower than on a 32 bit OS, but I am not sure if this is because of virtuel instead of physical hardware or if it is a problem with the 32 bit mode of a Windows 2003 64 bit OS.
Wednesday, 14 October 2009
Invalid search path '%LIB%' specified in 'LIB environmental variable' 'The system cannot find the path specified
I had a little problem installing Visual Studio 2008 on a 64 bit version of Windows 2003 with NAG libraries. On the 32 bit Windows 2003 Server the solution compiled but on the 64 bit version I got the error:
Warning as Error: Invalid search path '%LIB%' specified in 'LIB environmental variable' 'The system cannot find the path specified.
One thing is that the path is slightly different between the different versions:
LIB = "C:\Program Files\Numerical Algorithms Group\FLDLL204Z"
LIB = "C:\Program Files (x86)\Numerical Algorithms Group\FLDLL204Z"
But the real problem was solved by
1. Goto MyComputer—> Properties—-> Advanced Settings—-> and click Environment Variables
2. Delete User and System Variables for LIB ONLY (you need to keep the system variable)
3. Reboot
NB if you don't reboot the problem remains. It really is necesary to reboot
Warning as Error: Invalid search path '%LIB%' specified in 'LIB environmental variable' 'The system cannot find the path specified.
One thing is that the path is slightly different between the different versions:
LIB = "C:\Program Files\Numerical Algorithms Group\FLDLL204Z"
LIB = "C:\Program Files (x86)\Numerical Algorithms Group\FLDLL204Z"
But the real problem was solved by
1. Goto MyComputer—> Properties—-> Advanced Settings—-> and click Environment Variables
2. Delete User and System Variables for LIB ONLY (you need to keep the system variable)
3. Reboot
NB if you don't reboot the problem remains. It really is necesary to reboot
Saturday, 5 September 2009
Some ideas about Optimization and Restructuring
I am looking into restructuring and optimization of a catastrophe modeling platform. Here are some of the ideas that have come to mind.
The Aggregate Cat models consist of a disaggregation where the sums insured for locators are redistributed according to population fraction. Then some unmanaged mathematical routines are run that determine the loss ratio for each risk followed by the application of treaty structures. We will look into precalculating the loss ratios for each combination of locator, risk and vulnerability, in this way we can remove the disaggregation and ground up loss calculation from the productive code which will significantly improve the speed and simplify our aggregate models for all perils to run through the same code. The size of the lookup tables depend on the number of vulnerability functions. The models have various numbers of vulnerability functions meaning that the sizing of the hardware will depend on the model with the most vulnerability functions.
The optimization of the detailed models depends on the way that the loss is calculated for each peril. For example Wind storm calculations are based on storm footprint files that are a grid of information that is overlaid onto the exposure. In this case we could use the same approach as the aggregate models except we would need to determine from the Latitude and Longitude which storm footprint grid point is to be used. This means that we could use the geospatial queries that came with sql 2008. For Earthquake there are no grid points because the damage propagates in concentric circles from the earthquake. We could think of make a pseudo detailed model but there are about a thousand vulnerability functions meaning the database will be big and the time needed to precalculate the loss ratios will be long. So for Earthquake we need to optimize that mathematical component that calculates the loss ratio to run under a multi core 64 bit O/S. We will be considering:
At the moment the Server side consists of 140 sub projects. Instead of decomposing an application into sub projects we will create a few general projects but using folders instead of sub projects. This has the advantage that the program compiles faster and executes faster. Such a project could consist of:
The Aggregate Cat models consist of a disaggregation where the sums insured for locators are redistributed according to population fraction. Then some unmanaged mathematical routines are run that determine the loss ratio for each risk followed by the application of treaty structures. We will look into precalculating the loss ratios for each combination of locator, risk and vulnerability, in this way we can remove the disaggregation and ground up loss calculation from the productive code which will significantly improve the speed and simplify our aggregate models for all perils to run through the same code. The size of the lookup tables depend on the number of vulnerability functions. The models have various numbers of vulnerability functions meaning that the sizing of the hardware will depend on the model with the most vulnerability functions.
The optimization of the detailed models depends on the way that the loss is calculated for each peril. For example Wind storm calculations are based on storm footprint files that are a grid of information that is overlaid onto the exposure. In this case we could use the same approach as the aggregate models except we would need to determine from the Latitude and Longitude which storm footprint grid point is to be used. This means that we could use the geospatial queries that came with sql 2008. For Earthquake there are no grid points because the damage propagates in concentric circles from the earthquake. We could think of make a pseudo detailed model but there are about a thousand vulnerability functions meaning the database will be big and the time needed to precalculate the loss ratios will be long. So for Earthquake we need to optimize that mathematical component that calculates the loss ratio to run under a multi core 64 bit O/S. We will be considering:
- Using C# with immutable data
- Using F# or a combination of f# and C#
- Convert the existing C code to 64bit multi core
At the moment the Server side consists of 140 sub projects. Instead of decomposing an application into sub projects we will create a few general projects but using folders instead of sub projects. This has the advantage that the program compiles faster and executes faster. Such a project could consist of:
- GraphicalUserInterface
- Utilities (eg Zip, log4Net)
- Interfaces
- DataAccessLayer
- BusinessObjects (Types)
- BusinessLogic
Sharing of objects between the client and the server:
1. The Server would reference the Client projects as done now
2. We could make WCF Services hosted on IIS7 on the scheduling server
3. We could load balance WCF Services if the clients have a heavy computational demand.
We will consider the Microsoft has a load Balancing implementation as a part of Windows 2008. If we need to load balance then we should use this because programming your own has a lot of things that can go wrong.
Implementation strategy:
My approach would be to progressively transition the live system into the target architecture. Since the live system must function every step of the way has exhaustive tests that guarantee that progress is made. I find the danger of a basic rewrite is that testing will never be a thorough as on a real productive system which means when the prototype is made live there will be a lot of bugs to fix.
Logging
As we restructure we will build in a configurable logging intensity so that when something is being debugged intensive debugging is possible and during normal operation minimal debugging is made.
Here is how we are thinking of splitting the Tiers of the application
1. GUI Layer
eg Has functions like GetLossFileView etc
2. Business Layer
eg LossFileBO as entity
3. Data access layer Client
eg WCF Services using generic functions with entities defined in the Business Layer
4. Data access layer server
eg WCF Services using generic functions directly accessing the database.
The DAL would use a Factory using the Type Off construct that will enable intellisence through the entity data types
Versioning
1. Using MEF: Probably not worthwhile for PRECCP, because this would require very careful source control branching and consequently the maintenance work increases
2. Using an interface and built in database functions: Depends on the level at which you want to track versions. Eg it does not make sense to track the version of events in loss files directly.
3. Just keep track of versions: This is the simplest method and what we are doing now. Currently if you ask underwriters the only versions that are interesting are the current version, the previous version and the next version.
Saturday, 29 August 2009
A story of pifalls in hosting webservices or WCF Services on IIS6
It’s been a while since I made some WCF Services and even longer thatI had to do much with IIS. Here is a log of what I experienced as I was implementing some WCF functions in a LAN. Often I found web sites that had exactly the same problems that I had faced but no solution.
I have a number of servers on which I need a WCF Service. But I don’t want to enable IIS on all these servers, so I decided to host the WCF service in a Windows Service. Here are some links about this:
All informations about Windows Services:
http://msdn.microsoft.com/en-us/library/aa984074(VS.71).aspx
Walkthrough: Creating a Windows Service Application
http://msdn.microsoft.com/en-us/library/aa984464(VS.71).aspx
How to: Host a WCF Service in IIS
http://msdn.microsoft.com/en-us/library/ms733766.aspx
Hosting and Consuming WCF Services
http://msdn.microsoft.com/en-us/library/bb332338.aspx
Here are the steps I took to create a Windows Service:
1. Open Visual Studio and Create a Windows Service Project
2. Click in the design pain of Service1.cs
3. In the properties
Name = NigelsService
ServiceName = NigelsService
4. Drag and drop Eventlog from the toolbox into the design pain of Service1.cs
5. Edit the following code:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.Linq;
using System.ServiceProcess;
using System.Text;
namespace WindowsService1
{
public partial class NigelsService : ServiceBase
{
public NigelsService()
{
InitializeComponent();
// Create the source, if it does not already exist.
if (!EventLog.SourceExists("MySource"))
{
//An event log source should not be created and immediately used.
//There is a latency time to enable the source, it should be created
//prior to executing the application that uses the source.
//Execute this sample a second time to use the new source.
EventLog.CreateEventSource("MySource", "MyNewLog");
}
// Create an EventLog instance and assign its source.
this.eventLog1.Source = "MySource";
// This is exected on Power up
// Instanciate WCF service here
}
protected override void OnStart(string[] args)
{
// This is executed when the service starts
eventLog1.WriteEntry("Service Start");
// Start WCF service here
}
protected override void OnStop()
{
// This is executed when the service Stops
// (Power down is the garbage collector
eventLog1.WriteEntry("Service Stop");
}
}
}
6. Right mouse click in the design pain of Service1.cs and select Add Installer
7. Click the ServiceInstaller1 and change the following properties
Name = NigelsServiceInstaller1
ServiceName = NigelsService
StartType = Automatic
8. Click on ServiceProcessInstaller1 and change the following properties
Account = LocalSystem
9. Click File / add / New Project
10. Under Setup and Deployment select Setup Project
11. Right mouse click on the project root Setup1
12. Add / Project Output ...
13. Project = WindowsService1
Primary Output
Configuration Active
OK
14. Right mouse click on the project root Setup1
15. View / Custom Actions
16. Right Mouse click on the root of Custom items and select Add Custom Action..
17. Look in : Application Folder
Primary output from WindowsService1 (Active)
OK
18. Build the project
When installing the service you will notice that the Product name is used
When the service is started and stopped corresponding entries are made in the Application Event log
This worked very well because the service did not have much to do. I also had a service that had quite a lot of work to do. For this service I did not want to use a Windows Service because I wanted to make use of IIS’s ability to look after memory management and restarting the service if something goes wrong. I had an IIS6 server on to which I could deploy my application
Firstly I created a solution with a WCF Solution and for old times sake I created an additional WebService project.
Starting with WebService hosted on ASP.NET. Visual studio made a sample asmx file which I quickly modified to include all the functions that I needed. And it worked. So I renamed the Service1.asmx to something more appropriate and I got the following error
Parser Error
Description: An error occurred during the parsing of a resource required to service this request. Please review the following specific parse error details and modify your source file appropriately.
Parser Error Message: Could not create type 'PRECEDWebService.Service1'.
Source Error:
Line 1: WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.Service1"
Source File: /PRECEDWebService/PRECED.asmx Line: 1
So the fix for this was to change
WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.Service1"
to:
WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.PRECED"
The next step was I wanted to deploy this to my IIS6 Server. Its been about 7 years since the last time I really made something on IIS. So I was a little rusty. First on the server I setup a Virtuel directory in the IIS Manager and published my new webservice into it. But when I tried to invoke the Webservice I get a get an error.
By default all Web service extensions are disabled. I found some instructions in the following link:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/de6171d8-704c-431a-8117-915e02fac30b.mspx?mfr=true
To enable and disable a Web service extension
1. In IIS Manager, expand the local computer, and then click Web Service Extensions.
2. In the details pane, click the Web service extension that you want to enable or disable.
3.To enable a disabled Web service extension, click Allow.
4.To disable an enabled Web service extension, click Prohibit.
A message box with a list of applications that will be prevented from running on the IIS Web server displays.
5.Click OK to disable the Web service extension.
You will need to enable
- All Unkenown ISAPI Extentions
- Active Server Pages
- ASP.NET v2.0.50727
This time when trying to add a WebService I got the following error
The current identity (NT AUTHORITY\NETWORK SERVICE) does not have write access to 'C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files'.
To fix this
1. Goto IIS
2. Open the application Pools
3. Click on the properties of DefaultAppPool
4. Click on Identity
5. Change Predefined to Local System
6. Restart web server
Also I granted IUSR_ServerName access rights to ../Windows/Microsoft.NET/Framework/version
In addition I found that if the IIS had not been correctly configured the following command line statement would reinstall a number of components
C:\WINDOWS\Microsoft.NET\Framework\v3.0\Windows Communication Foundation>service
modelreg.exe –r
This worked quite nicely, but if the server is internet facing for security reasons I would not use IIS7 and do some research over security issues.
So next I looked again at the WCF solution to see if I could do the same. Starting off by trying to consume a WCF with the solution. The first thing I did was to change the binding type to basicHttpBinding. The first time I did this I used a capital B in the web.config file and got an error about an unrecognized binding type. I wish the compiler would also correct my config files! The web.config file ended up looking like:
client
endpoint address="http://localhost:1271/PRECED.svc" binding="basicHttpBinding"
bindingConfiguration="EndPointHTTP" contract="ServiceReference4.IPRECED"
name="EndPointHTTP"
/client
The corresponding code to call this with app.config looked like
Dim iPRECEDClient As New ServiceReference5.PRECEDClient
Console.WriteLine(vbLf & "Calling Service..." & vbLf)
Dim str As String
str = iPRECEDClient.HelloWorld
Console.WriteLine(str)
Console.ReadLine()
Alternatively without app.config
Dim proxy As IPRECED = ChannelFactory(Of IPRECED).CreateChannel(New BasicHttpBinding(), New EndpointAddress("http://localhost:1271/PRECED.svc"))
Console.WriteLine(vbLf & "Calling Service..." & vbLf)
Console.WriteLine(proxy.ValidateUser())
Console.ReadLine()
The final step was to publish this to the web server. I replaced http://localhost:1271/PRECED.svc with http://servername/VirtuelDir/PRECED.svc%20in%20both%20the%20Web.config and the app.config and everything worked well
For completeness I added a MEX endpoing as shown below
system.serviceModel
services
service behaviorConfiguration="PRECEDBehaviors" name="PRECEDWcfService.PRECED"
endpoint address="http://localhost:13243/" binding="basicHttpBinding"
name="EndPointHTTP" contract="PRECEDWcfService.IPRECED" /
/service
service behaviorConfiguration="PRECEDBehaviors" name="PRECEDWcfService.PRECEDMex"
endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" /
/service
/services
behaviors
serviceBehaviors
behavior name="PRECEDBehaviors"
serviceMetadata httpGetEnabled="true" /
/behavior
/serviceBehaviors
/behaviors
/system.serviceModel
This was quite a long and painful exercise but that was because I been working on thick clients apps and not web apps in the last 7 years. One nice trick I learned on the way was to use the browse function on the design time webserver because when you double click on the svc files you get a good error messages.
I have a number of servers on which I need a WCF Service. But I don’t want to enable IIS on all these servers, so I decided to host the WCF service in a Windows Service. Here are some links about this:
All informations about Windows Services:
http://msdn.microsoft.com/en-us/library/aa984074(VS.71).aspx
Walkthrough: Creating a Windows Service Application
http://msdn.microsoft.com/en-us/library/aa984464(VS.71).aspx
How to: Host a WCF Service in IIS
http://msdn.microsoft.com/en-us/library/ms733766.aspx
Hosting and Consuming WCF Services
http://msdn.microsoft.com/en-us/library/bb332338.aspx
Here are the steps I took to create a Windows Service:
1. Open Visual Studio and Create a Windows Service Project
2. Click in the design pain of Service1.cs
3. In the properties
Name = NigelsService
ServiceName = NigelsService
4. Drag and drop Eventlog from the toolbox into the design pain of Service1.cs
5. Edit the following code:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Diagnostics;
using System.Linq;
using System.ServiceProcess;
using System.Text;
namespace WindowsService1
{
public partial class NigelsService : ServiceBase
{
public NigelsService()
{
InitializeComponent();
// Create the source, if it does not already exist.
if (!EventLog.SourceExists("MySource"))
{
//An event log source should not be created and immediately used.
//There is a latency time to enable the source, it should be created
//prior to executing the application that uses the source.
//Execute this sample a second time to use the new source.
EventLog.CreateEventSource("MySource", "MyNewLog");
}
// Create an EventLog instance and assign its source.
this.eventLog1.Source = "MySource";
// This is exected on Power up
// Instanciate WCF service here
}
protected override void OnStart(string[] args)
{
// This is executed when the service starts
eventLog1.WriteEntry("Service Start");
// Start WCF service here
}
protected override void OnStop()
{
// This is executed when the service Stops
// (Power down is the garbage collector
eventLog1.WriteEntry("Service Stop");
}
}
}
6. Right mouse click in the design pain of Service1.cs and select Add Installer
7. Click the ServiceInstaller1 and change the following properties
Name = NigelsServiceInstaller1
ServiceName = NigelsService
StartType = Automatic
8. Click on ServiceProcessInstaller1 and change the following properties
Account = LocalSystem
9. Click File / add / New Project
10. Under Setup and Deployment select Setup Project
11. Right mouse click on the project root Setup1
12. Add / Project Output ...
13. Project = WindowsService1
Primary Output
Configuration Active
OK
14. Right mouse click on the project root Setup1
15. View / Custom Actions
16. Right Mouse click on the root of Custom items and select Add Custom Action..
17. Look in : Application Folder
Primary output from WindowsService1 (Active)
OK
18. Build the project
When installing the service you will notice that the Product name is used
When the service is started and stopped corresponding entries are made in the Application Event log
This worked very well because the service did not have much to do. I also had a service that had quite a lot of work to do. For this service I did not want to use a Windows Service because I wanted to make use of IIS’s ability to look after memory management and restarting the service if something goes wrong. I had an IIS6 server on to which I could deploy my application
Firstly I created a solution with a WCF Solution and for old times sake I created an additional WebService project.
Starting with WebService hosted on ASP.NET. Visual studio made a sample asmx file which I quickly modified to include all the functions that I needed. And it worked. So I renamed the Service1.asmx to something more appropriate and I got the following error
Parser Error
Description: An error occurred during the parsing of a resource required to service this request. Please review the following specific parse error details and modify your source file appropriately.
Parser Error Message: Could not create type 'PRECEDWebService.Service1'.
Source Error:
Line 1: WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.Service1"
Source File: /PRECEDWebService/PRECED.asmx Line: 1
So the fix for this was to change
WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.Service1"
to:
WebService Language="VB" CodeBehind="PRECED.asmx.vb" Class="PRECEDWebService.PRECED"
The next step was I wanted to deploy this to my IIS6 Server. Its been about 7 years since the last time I really made something on IIS. So I was a little rusty. First on the server I setup a Virtuel directory in the IIS Manager and published my new webservice into it. But when I tried to invoke the Webservice I get a get an error.
By default all Web service extensions are disabled. I found some instructions in the following link:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/de6171d8-704c-431a-8117-915e02fac30b.mspx?mfr=true
To enable and disable a Web service extension
1. In IIS Manager, expand the local computer, and then click Web Service Extensions.
2. In the details pane, click the Web service extension that you want to enable or disable.
3.To enable a disabled Web service extension, click Allow.
4.To disable an enabled Web service extension, click Prohibit.
A message box with a list of applications that will be prevented from running on the IIS Web server displays.
5.Click OK to disable the Web service extension.
You will need to enable
- All Unkenown ISAPI Extentions
- Active Server Pages
- ASP.NET v2.0.50727
This time when trying to add a WebService I got the following error
The current identity (NT AUTHORITY\NETWORK SERVICE) does not have write access to 'C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Temporary ASP.NET Files'.
To fix this
1. Goto IIS
2. Open the application Pools
3. Click on the properties of DefaultAppPool
4. Click on Identity
5. Change Predefined to Local System
6. Restart web server
Also I granted IUSR_ServerName access rights to ../Windows/Microsoft.NET/Framework/version
In addition I found that if the IIS had not been correctly configured the following command line statement would reinstall a number of components
C:\WINDOWS\Microsoft.NET\Framework\v3.0\Windows Communication Foundation>service
modelreg.exe –r
This worked quite nicely, but if the server is internet facing for security reasons I would not use IIS7 and do some research over security issues.
So next I looked again at the WCF solution to see if I could do the same. Starting off by trying to consume a WCF with the solution. The first thing I did was to change the binding type to basicHttpBinding. The first time I did this I used a capital B in the web.config file and got an error about an unrecognized binding type. I wish the compiler would also correct my config files! The web.config file ended up looking like:
endpoint address="http://localhost:1271/PRECED.svc" binding="basicHttpBinding"
bindingConfiguration="EndPointHTTP" contract="ServiceReference4.IPRECED"
name="EndPointHTTP"
/client
The corresponding code to call this with app.config looked like
Dim iPRECEDClient As New ServiceReference5.PRECEDClient
Console.WriteLine(vbLf & "Calling Service..." & vbLf)
Dim str As String
str = iPRECEDClient.HelloWorld
Console.WriteLine(str)
Console.ReadLine()
Alternatively without app.config
Dim proxy As IPRECED = ChannelFactory(Of IPRECED).CreateChannel(New BasicHttpBinding(), New EndpointAddress("http://localhost:1271/PRECED.svc"))
Console.WriteLine(vbLf & "Calling Service..." & vbLf)
Console.WriteLine(proxy.ValidateUser())
Console.ReadLine()
The final step was to publish this to the web server. I replaced http://localhost:1271/PRECED.svc with http://servername/VirtuelDir/PRECED.svc%20in%20both%20the%20Web.config and the app.config and everything worked well
For completeness I added a MEX endpoing as shown below
services
service behaviorConfiguration="PRECEDBehaviors" name="PRECEDWcfService.PRECED"
endpoint address="http://localhost:13243/" binding="basicHttpBinding"
name="EndPointHTTP" contract="PRECEDWcfService.IPRECED" /
/service
service behaviorConfiguration="PRECEDBehaviors" name="PRECEDWcfService.PRECEDMex"
endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" /
/service
/services
behaviors
serviceBehaviors
behavior name="PRECEDBehaviors"
serviceMetadata httpGetEnabled="true" /
/behavior
/serviceBehaviors
/behaviors
/system.serviceModel
This was quite a long and painful exercise but that was because I been working on thick clients apps and not web apps in the last 7 years. One nice trick I learned on the way was to use the browse function on the design time webserver because when you double click on the svc files you get a good error messages.
Tuesday, 9 June 2009
Here is how to create a synchronization with a database and cached data1
Here is a quick example of how to used the synchronization framework and cached data on the client:
1. Create a Windows Forms application
2. Under the tool bar select Data/Add New Data Server
3. Select database
4. Click the new connection button and select the database to connect to
5. Don't save the connection as
6. Select the tables to synchronize Select Enable local database caching (This creates a mini local database)
7. Select Synchronization Mode Incremental (Snap shot means every time the database is synchronized all data is downloaded ) Select Use SQL Server change tracking
8. Select Update server for incremntal changes
Save SQL scripts in the project for later use (This is useful if you move the database)
After this you will get a file like TEST_AynchCache.syncIf you double click on this you get a hyperlink to "Show Code Example".
This code can be used to perform the synchronization.
Here is an example of how to populate and synchronize a data grid view
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Me.TST_SLAVETableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_SLAVE)
Me.TST_MASTERTableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_MASTER)
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim syncAgent As TEST_SYNCHCacheSyncAgent = New TEST_SYNCHCacheSyncAgent()
Dim syncStats As Microsoft.Synchronization.Data.SyncStatistics = syncAgent.Synchronize()
Me.TST_SLAVETableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_SLAVE)
End Sub
syncStats.TotalChangesDownloaded contains the number of changes that where made
1. Create a Windows Forms application
2. Under the tool bar select Data/Add New Data Server
3. Select database
4. Click the new connection button and select the database to connect to
5. Don't save the connection as
6. Select the tables to synchronize Select Enable local database caching (This creates a mini local database)
7. Select Synchronization Mode Incremental (Snap shot means every time the database is synchronized all data is downloaded ) Select Use SQL Server change tracking
8. Select Update server for incremntal changes
Save SQL scripts in the project for later use (This is useful if you move the database)
After this you will get a file like TEST_AynchCache.syncIf you double click on this you get a hyperlink to "Show Code Example".
This code can be used to perform the synchronization.
Here is an example of how to populate and synchronize a data grid view
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Me.TST_SLAVETableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_SLAVE)
Me.TST_MASTERTableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_MASTER)
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
Dim syncAgent As TEST_SYNCHCacheSyncAgent = New TEST_SYNCHCacheSyncAgent()
Dim syncStats As Microsoft.Synchronization.Data.SyncStatistics = syncAgent.Synchronize()
Me.TST_SLAVETableAdapter.Fill(Me.TEST_SYNCHDataSet.TST_SLAVE)
End Sub
syncStats.TotalChangesDownloaded contains the number of changes that where made
My first steps at Synchronizing SQL2008 databases
Here are some steps I have taken in testing the synchronization of databases using SQL
Install SQL-Server Replication
1. Run the MS SQL Setup.exe
2. Select Installation in the left hand panel
3. Click New SQL Server stand-alone installation or add features to an existing installation
4. Setup Support Files
Click install
5. Setpup Support Rules click next
6. Click Add features to an existing instance of SQL 2008
Select MSSQLSERVER
Next
7. Select Database Engine Service/SQL Server ReplicationNext
8. Disk space requirements
click next
9. Error and usage reporting
click next
10. Installation rules click next
11. Ready to install click Install
12. Installation progress click next
13. Complete click close
Setup the publisher
1. Open Microsoft SQL Server Manager Studio
2. Browse to Replication/Local Publications
3. right mouse click on Local Publications and select New Publication
4. New publication wizard click next
5. Select 'YourServerName' will act as its own Distributor; SQL Server will create a distribution database and log
Next
6. Select Yes, Configure the SQL Server Agent to start automatically
Next
7. Leave the snap shot folder in it's default location
C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\ReplDataNext8. Select the database that you want to synchronize
Next
9. Publication Type.
Transactional publication:The Publisher streams transactions to the Subscribers after they receive an initial snapshot of the published data.
Transactional publication with updatable subscriptions:The Publisher streams transactions to SQL Server Subscribers after they receive an initial snapshot of the published data. Transactions originating at the Subscriber are applied at the Publisher.
Merge publication:The Publisher and Subscribers can update the published data independently after the Subscribers receive an initial snapshot of the published data. Changes are merged periodically. Microsoft SQL Server Compact Edition can only subscribe to merge publications.
Select Transactional publication with updatable subscriptions
Next
10. Article
Select all the Tables including sysdiagrams(dbo)
Next
11. Articles issued click next
12. Filter Table Rows click
next
13. Snap Shot Agent select Create a snap shot immediately and keep the snapshot available to initialize ssubscriptions
Next
14. Agent Security
Click Security settings for SnapShot Agent and Queue Reader Agent
15. For both steps in 14 I selected
Run under the SQL Server Agent Service account (This is not recommended security best practice)
(I dont want to manage a lot of different service accounts so I choose just the account I use the one I configured for SQL Agent)
16. Wizard Actions Select
Cretae the publication
Next
17. Click Next and the Create Publication wizard will start
18. Close
Setup the Subscription
(Before starting create a linked server to the publisher that uses the current security context, it's needed in step 15)
1. Open Microsoft SQL Server Manager Studio and connect to the target server
2. Browse to Replication/Local Subcriptions
3. right mouse click on Local Subscriptions and select New Subscription
4. New subscription wizard click next
5. Select
6. Open the SQL Server that has the published database
7. Select the database and publications that was just created
8. Distribution Agent Location
Select Run each agent at it's Subscriber (pull subscriptions)(The alternative is to run all agents at the distributor (push subscriptions) which can be easier to administer)
Next
9. Subscribers Choose the Subscription database where the data will go(At this point you can also create a new database)
10. Distribution Agent SecurityClick the (...) button
11. Run under the SQL Server Agent Service account (This is not recommended security best practice)
(I dont want to manage a lot of different service accounts so I choose just the account I use the one I configured for SQL Agent)
OK
12. Distribution Agent Security Click Next
13. Synchronization Schedule Select Run continously (the alternative is to run on demand or to define a schedule)
Next
14. Updateable SubsscriptionsSelect Simultaneously commit changes (the alternative is to queue changes and commit when possible)
Next
15. Login For Updateable scriptionsSelect Use a linked server or remote server that you have already defined
16. Initialize SbscriptionsSelect Immediately
Next
17. Wizard actions select Create the subscriptions
Next
After this something like this will apear
• Create subscription(s).
Create a subscription to publication 'TEST' from Publisher 'CHZUPRES913'.
Create subscriptions at the following Subscriber(s):
CHZUPRES912
• Subscription database: TEST
• Agent location: Subscriber
• Agent schedule: Run continuously
• Agent process account: SQLServerAgent service account
• Connection to Distributor: Impersonate process account
• Connection to Subscriber: Impersonate process account
• Initialize: Immediately
• Replicate changes: Simultaneously commit changes
18 Creating Subscription(s)Close
Install SQL-Server Replication
1. Run the MS SQL Setup.exe
2. Select Installation in the left hand panel
3. Click New SQL Server stand-alone installation or add features to an existing installation
4. Setup Support Files
Click install
5. Setpup Support Rules click next
6. Click Add features to an existing instance of SQL 2008
Select MSSQLSERVER
Next
7. Select Database Engine Service/SQL Server ReplicationNext
8. Disk space requirements
click next
9. Error and usage reporting
click next
10. Installation rules click next
11. Ready to install click Install
12. Installation progress click next
13. Complete click close
Setup the publisher
1. Open Microsoft SQL Server Manager Studio
2. Browse to Replication/Local Publications
3. right mouse click on Local Publications and select New Publication
4. New publication wizard click next
5. Select 'YourServerName' will act as its own Distributor; SQL Server will create a distribution database and log
Next
6. Select Yes, Configure the SQL Server Agent to start automatically
Next
7. Leave the snap shot folder in it's default location
C:\Program Files\Microsoft SQL Server\MSSQL10.MSSQLSERVER\MSSQL\ReplDataNext8. Select the database that you want to synchronize
Next
9. Publication Type.
Transactional publication:The Publisher streams transactions to the Subscribers after they receive an initial snapshot of the published data.
Transactional publication with updatable subscriptions:The Publisher streams transactions to SQL Server Subscribers after they receive an initial snapshot of the published data. Transactions originating at the Subscriber are applied at the Publisher.
Merge publication:The Publisher and Subscribers can update the published data independently after the Subscribers receive an initial snapshot of the published data. Changes are merged periodically. Microsoft SQL Server Compact Edition can only subscribe to merge publications.
Select Transactional publication with updatable subscriptions
Next
10. Article
Select all the Tables including sysdiagrams(dbo)
Next
11. Articles issued click next
12. Filter Table Rows click
next
13. Snap Shot Agent select Create a snap shot immediately and keep the snapshot available to initialize ssubscriptions
Next
14. Agent Security
Click Security settings for SnapShot Agent and Queue Reader Agent
15. For both steps in 14 I selected
Run under the SQL Server Agent Service account (This is not recommended security best practice)
(I dont want to manage a lot of different service accounts so I choose just the account I use the one I configured for SQL Agent)
16. Wizard Actions Select
Cretae the publication
Next
17. Click Next and the Create Publication wizard will start
18. Close
Setup the Subscription
(Before starting create a linked server to the publisher that uses the current security context, it's needed in step 15)
1. Open Microsoft SQL Server Manager Studio and connect to the target server
2. Browse to Replication/Local Subcriptions
3. right mouse click on Local Subscriptions and select New Subscription
4. New subscription wizard click next
5. Select
6. Open the SQL Server that has the published database
7. Select the database and publications that was just created
8. Distribution Agent Location
Select Run each agent at it's Subscriber (pull subscriptions)(The alternative is to run all agents at the distributor (push subscriptions) which can be easier to administer)
Next
9. Subscribers Choose the Subscription database where the data will go(At this point you can also create a new database)
10. Distribution Agent SecurityClick the (...) button
11. Run under the SQL Server Agent Service account (This is not recommended security best practice)
(I dont want to manage a lot of different service accounts so I choose just the account I use the one I configured for SQL Agent)
OK
12. Distribution Agent Security Click Next
13. Synchronization Schedule Select Run continously (the alternative is to run on demand or to define a schedule)
Next
14. Updateable SubsscriptionsSelect Simultaneously commit changes (the alternative is to queue changes and commit when possible)
Next
15. Login For Updateable scriptionsSelect Use a linked server or remote server that you have already defined
16. Initialize SbscriptionsSelect Immediately
Next
17. Wizard actions select Create the subscriptions
Next
After this something like this will apear
• Create subscription(s).
Create a subscription to publication 'TEST' from Publisher 'CHZUPRES913'.
Create subscriptions at the following Subscriber(s):
CHZUPRES912
• Subscription database: TEST
• Agent location: Subscriber
• Agent schedule: Run continuously
• Agent process account: SQLServerAgent service account
• Connection to Distributor: Impersonate process account
• Connection to Subscriber: Impersonate process account
• Initialize: Immediately
• Replicate changes: Simultaneously commit changes
18 Creating Subscription(s)Close
Subscribe to:
Posts (Atom)