Transfer Extremely Large Files Using Windows Service and BizTalk Server

Introduction

In a BizTalk project, I had to receive extremely large files (up to 2GB), zip them and then transfer them via FTP to a known destination.

My first idea was to write a custom pipeline component to receive the file and write it to disk for temporary storage. The pipeline would create a small XML file containing the path to the large file. A component would be called from an orchestration (or send pipeline) and do the transfer process so that the large file would never be sent through the BizTalk Message Box. My experience was that this worked very fine for files up to 1GB, but when they were bigger, the receive pipeline became too slow when more files were received at the same time.

The solution for this project was to create a Windows Service with a FileSystemWatcher object to watch for new large files and create a small XML file containing the path to the large file when a new file is created. The Windows Service reads information about the senders and the receiver at startup from a BizTalk Policy. The small XML file is received by BizTalk and passed to an orchestration. The orchestration reads the path to the large file and waits until the large file is ready to be transferred. It then calls an external component to zip the large file and send it to the FTP destination.

The main reason for using BizTalk for this project is to centralize all the messages and the logging and monitoring functionality. Moreover, the BizTalk Policy makes it easy to add or delete senders and receivers. (In this project, there are many senders and only one receiver.) It is also possible to set up correlation between the messages and any kind of receipts in the orchestration. If the receiver system does not give any receipt back, it is possible to make a kind of transport receipt in the orchestration based on the return value from the component that does the transfer process.

Create a BizTalk Policy

The first thing to do is to create and deploy a BizTalk Policy so that the Windows Service can read the necessary information. I will not go into detail about how to create a BizTalk Policy in this article, but I will explain shortly how I did it.

First, I created an assembly which contains only one simple class with all the information I wanted about the senders and the receiver. The class looked like this:

using System;
using System.Collections.Generic;
using System.Text;

namespace LargeFileTransfer.LFTRulesHelper
{
    public class LFTRulesHelper
    {
        private string _partnerName;
        private string _ftpPath;
        private string _catalog;
        private string _userName;
        private string _userPassword;
        private string _largeFilePath;
        private string _allPartners;
        private string _xmlInfoOutputPath;

        public string PartnerName
        {
            get { return _ partnerName; }
            set { _ partnerName = value; }
        }
        // Make one property for each variable
}

PartnerName is the name of the sender of the large file.

FtpPath, Catalog, UserName and UserPassword are properties used by the component doing the transfer process to connect to the FTP server. The user details can be encrypted to make it more secure.

LargeFilePath is the path where the large file is found. It is a kind of receive location. This project requires that the last catalog in this path (the catalog in which the large file is found) has the same name as the sender. For example, if the name of the sender is “Norway” then the large file should be picked up in a path looking like this “..\Norway\LargeFile.xml”.

AllPartners is a comma separated list of all senders.

XmlInfoOutputPath is the path where the small XML file is created.

This class is used to build the BizTalk Policy. It must be compiled and the DLL must be added to the BizTalk Servers GAC before making the BizTalk Policy.

The next step is to open the BizTalk Rule Composer and create a new rule set based on this class. Create one rule for each partner and set the rule name to be the same as the sender name. Give values to all the properties you need. (There are plenty of articles describing in detail how to do this, so I will not do it here.) Also create one rule named “All”, set Partnername like “All” and list all partners in the AllPartners property. Save and deploy the BizTalk Policy and then it is ready to be used.

Create and Set Up the Windows Service

I will explain the main functionality in the Windows Service and how to install it. There are a lot of articles describing the basics about Windows Services, so I will not do it here.

First, I created a new Windows Service project in Visual Studio, added references and included the namespaces below in the main class.

using Microsoft.RuleEngine;

This namespace provides functionality to access the BizTalk Policy.

using LargeFileTransfer.LFTRulesHelper;

In this namespace, I find the class I created above. I use this when I get information from the BizTalk Policy.

using System.IO;

In this namespace, I find the FileSystemWatcher class.

In this Windows Service, I implemented three methods: OnStart which runs when the Windows Service starts, OnChanged which is used by the FileSystemWatcher and GetPartnername which simply returns the partner name from a string. I will describe in detail how these methods work.

This Windows Service reads at startup all sender names from the BizTalk Policy. It then loops through all senders and creates a FileSystemWatcher for each sender to watch for new files in all locations.

protected override void OnStart(string[] args)
{
    	Policy policy = new Policy("LargeFileTransfer.LFTRules");
         LargeFileTransfer.LFTRulesHelper.LFTRulesHelper myHelper = _
new LFTRulesHelper.LFTRulesHelper();

The name of the BizTalk Policy on the BizTalk Server is LargeFileTransfer.LFTRules. Above, I create a policy of this type and also an object from the class LFTRulesHelper named myHelper.

// Read all partner names from the BizTalk Policy
myHelper.PartnerName = "ALL";
object objMyHelper = new object();
objMyHelper = myHelper;
policy.Execute(objMyHelper);
_xmlInfoPath = myHelper.XmlInfoOutputPath;
string partners = myHelper.AllPartners;
string[] partnersArray = partners.Split(',');

In the code above, I first set the PartnerName to “ALL”. The PartnerName property is the same as the sender and must be the same as the BizTalk Rule name as mentioned above. I then create the objMyHelper and use this as parameter to execute the policy. The myHelper now has the same information as the rule named “ALL”. This works fine since I used this class to create the BizTalk Policy. In my case, _xmlInfoPath (the path where I create the small XML file) is the same for all senders. This is a private variable in the Windows Service class.

An array named partnersArray contains all sender names. In the code below, I loop through all these senders. For each sender, I execute the policy and get all the information about the sender. I also create a FileSystemWatcher object for each sender, set its path to the path where the large files from that sender are going to be created and add an event to be raised every time a new XML file is created in that path.

// Get a path for each partner and start waiting for new files in that path.
for (int i = 0; i < partnersArray.Length; i++)
{
    myHelper.PartnerName = partnersArray[i];
    policy.Execute(objMyHelper);
    System.IO.FileSystemWatcher watcher = new System.IO.FileSystemWatcher();
    watcher.Path = MyHelper.LargeFilePath;
    watcher.Filter = "*.xml";
    watcher.Created += new FileSystemEventHandler(OnChanged);
    watcher.EnableRaisingEvents = true;
}
}// End of OnStart

The OnChanged method is fired every time a new file is created in the path (LargeFilePath). The code is below. This method uses a FileStream object to create a new XML file to the path I got from the policy (_xmlInfoPath). The object e of type FileSystemEventArgs contains information from the FileSystemWatcher object. From here, I get the e.FullPath which is the path to where the FileSystemWatcher is watching for new files. As mentioned above, this project requires that the name of the catalog in which the large file is stored must be the same as the sender's name. I get this name from the GetPartnerName method.

Next, I build the xmlString. This will be the XML file. The XML file needs to match a schema on the BizTalk Server in order to work in the orchestration. The xmlString contains information about the schema namespace and the path to the large file. It also contains some additional information.

When the xmlString is done, I convert it to an array of bytes and write it to an XML file using the FileStream object. The XML file is now ready to be picked up by BizTalk.

private void OnChanged(object sender, FileSystemEventArgs e)
{
     	FileStream fs = new FileStream(_xmlInfoPath + @"XmlInfo" + 
		Guid.NewGuid() + ".xml", FileMode.CreateNew);
     	string partnerName = GetPartnerName(e.FullPath);
      	string receiverName = "LargeFileReceiver";
      	string xmlString = "<xmlinfo xmlns="\"LargeFileTransfer.Schemas\"">" 
	+ "<msgid />" + Guid.NewGuid() + "</msgid />" + "<sender />" 
	+ partnerName + "</sender />" + "<receiver />" + receiverName 
	+ "</receiver />" + "<filepath />" + e.FullPath + "</filepath />" 
	+ "<msgdatetime />" + DateTime.Now.ToString() + "</msgdatetime />" 
	+ "</xmlinfo />";

	byte[] byteArray = ASCIIEncoding.UTF8.GetBytes(xmlString);
      	fs.Write(byteArray, 0, byteArray.Length);
      	fs.Close();
}

The GetPartnerName method looks like this:

// Get partner name from a path. The partner name is the same 
// as the name of the inner catalog.
private string GetPartnerName(string fullPath)
{
	int i = fullPath.LastIndexOf("\\");
      	if (i == -1)
      		i = fullPath.LastIndexOf("//");

      	string tmpString = fullPath.Remove(i);

      	int j = tmpString.LastIndexOf("\\");
      	if (j == -1)
      		j = tmpString.LastIndexOf("//");
	string partnerName = tmpString.Substring(j + 1);
      	return partnerName;
}

I built this project and got a .exe file. I used the InstallUtil.exe to install it as a Windows Service and then I opened the Service Manager and started it. It worked fine.

To install it, run this command:

C:\Windows\Microsoft.NET\Framework\v2.0.50727\InstallUtil.exe 
Ex: InstallUtil.exe "C:\MyService.exe"

Remember to restart the Windows Service every time changes are done in the BizTalk Policy.

The BizTalk Orchestration

A schema which matches the XML file created by the Windows Service needs to be deployed to BizTalk so that the orchestration can receive the XML message. My XML file contains fields like MsgId, Sender, Receiver and FilePath. I promoted all the fields so the orchestration can get all the values from the file. That means that a property schema is also deployed. The first thing the orchestration does is to read the path to the large file from the small XML file. In the orchestration, I named this message XmlInfoIn. Since the field FilePath is promoted, I can easily write the value to a string variable in an expression shape like this:

strFilePath = XmlInfoIn(LargeFileTransfer.Schemas.PropertySchema.FilePath)

The next thing the orchestration does is to check if the large file is ready to be transferred. The Windows Service creates the small XML file as soon as the large file is created in the filesystem, but it may happen that the large file is not finished when the orchestration receives the small file. For example, it is not finished being copied from another destination. In a new expression shape, I do this check like this:

blnFileReady = LargeFileTransfer.LFTHelper.IsFileReady(strFilePath)

blnFileReady is a boolean value which is true if the file is ready. I built some additional code into an assembly where I have a class named LFTHelper. This class contains some methods which are used by the orchestration to process the transfer of the large file. This assembly is installed into the GAC. One of the methods is the IsFileReady which simply checks if the file is ready to be transferred.

public static bool IsFileReady(string filePath)
	{
            string readyString = "";
            try
            {
                // The file is ready if it can be renamed.
                File.Move(filePath, filePath + "Ready");
                readyString = "Ready";
                return true;
            }
            catch (Exception ex)
            {
                return false;
            }
            finally
            {
                if (readyString == "Ready")
                {
                    // Rename back to the original name if the file was renamed.
                    File.Move(filePath + "Ready", filePath);
                }
            }
        }

This method simply tries to rename the large file. If the file can be renamed, then it is ready and the method renames it back to the original name and returns true. Next in the orchestration, there is a loop shape which loops until the file is ready, that means until blnFileReady is true. This loop contains two shapes. The first one waits one minute and the second one does the same check again to see if the file is ready. Doing this makes the orchestration wait until the large file is ready. When the file is ready, the orchestration zips the file and then transfers it via FTP. This functionality is also included in the class LFTHelper. In a new expression shape one method is first called to zip the file.

strZipFileName = LargeFileTransfer.LFTHelper.LFTHelper.ZipFile(strFilePath)

strZipFileName now contains the path to the zipped file. This file is now the one to be transferred. The transfer is done by calling another method:

strStatus = LargeFileTransfer.LFTHelper.LFTHelper.TransferFile(strZipFileName)

strStatus is a string which contains the status after trying to transfer the file via FTP. The zipFileMethod uses an object in the Ionic.Zip library, so a reference to this library must be made. This library is free and can be downloaded from http://dotnetzip.codeplex.com.

public static string ZipFile(string filePath)
{
   using (FileStream fs = new FileStream(filePath, FileMode.Open))
   {
      string zippedFileName = filePath + ".zip";
      using (ZipOutputStream zipOutputStream = new ZipOutputStream(zippedFileName))
      {
         byte[] buffer = new Byte[1024];

         string fileName = GetFileName(filePath);
         zipOutputStream.PutNextEntry(fileName);

         int bytesRead = 1024;
         while (bytesRead != 0)
         {
            bytesRead = fs.Read(buffer, 0, buffer.Length);
            zipOutputStream.Write(buffer, 0, bytesRead);
         }
      }
      return zippedFileName;
   }
}

// Get the file name from a full path.
private static string GetFileName(string fullPath)
{
   int i = fullPath.LastIndexOf("\\");
   if (i == -1)
      i = fullPath.LastIndexOf("//");

   string fileName = fullPath.Substring(i + 1);
   return fileName;
}

The file name to the zipped file is the name to the large file plus ”.zip”. It is stored at the same location as the large file. Here is a simple method to transfer the zipped file via FTP:

public static string TransferFile(string filePath)
{
	string partnerName = GetPartnerName(filePath);
         LFTRulesHelper.LFTRulesHelper lftRulesHelper = new _
		LargeFileTransfewr.LFTRulesHelper.LFTRulesHelper();
         Policy policy = new policy("LargeFileTransfer.LFTRules");

         // Get all the details from the policy to transfer the file.
         lftRulesHelper.PartnerName = partnerName;
         object objLftHelper = new object();
         objLftHelper = lftRulesHelper;
         policy.Execute(objLftHelper);

         try
         {
         		// The FTP request
                	FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create _
		(lftRulesHelper.FtpPath + "/" + Path.GetFileName(filePath));
                	request.Method = WebRequestMethods.Ftp.UploadFile;
                	request.Credentials = new NetworkCredential _
		(lftRulesHelper.UserName, lftRulesHelper.UserPassword);
                	request.UsePassive = true;
                	request.UseBinary = true;
                	request.KeepAlive = false;

               	// Loading the file
                	FileStream stream = File.OpenRead(filePath);
                	byte[] buffer = new byte[stream.Length];
                	stream.Read(buffer, 0, buffer.Length);
                	stream.Close();

                	// Uploading the file
                	Stream reqStream = request.GetRequestStream();
                	reqStream.Write(buffer, 0, buffer.Length);
                	reqStream.Close();
                	return "OK";
      	}
         catch(Exception ex)
         {
          	return ex.Message;
         }
}

First the method looks up in the policy in the same way as the Windows Service to get all the stored details for the FTP connection. In the try block, an FTP request is made using these details. Then the zipped file is loaded into a buffer and then the file is transferred by writing the data from the buffer to a FTP request stream. When the transfer is completed, the method returns ”OK”. In the orchestration, the strStatus string now contains either ”OK” or an error message returned from the TransferFile method. The last thing this orchestration does is to send the small XML file to disk for logging via a send shape.

Complete the Project

To complete this project, the zipped file and the large file must be deleted from the disk when the methods using them are finished. This can easily be done by, for example, adding a deleteFile method to the LFTHelper class. This method can be called from the expressions in the orchestration.

Methods for encryption and signing can also easily be added to make the solution more secure. Certificate information can be added to the BizTalk policy and read by the methods. These methods can be called from the same shape which calls the zip method.

Another thing that can also be done is to write the status from the transfer method into a field in the XML file which is sent from the orchestration. This can be done by building up a new XML message in the orchestration in the same way as done in the Windows Service, add the status to one field and send this new XML file out of the orchestration instead of the original one.