Click here to Skip to main content
14,491,358 members

Transfer Large Files using BizTalk - Send Side

Rate this:
4.29 (6 votes)
Please Sign up or sign in to vote.
4.29 (6 votes)
20 Apr 2011CPOL
How to transfer large files using BizTalk - Send side


Dealing with large files (200MB+) on BizTalk can be a major performance bottleneck when a file is parsed by a receive pipeline and stored into the MessageBox. The CPU usage of both the BizTalk Server host instance that receives the file and the SQL Server can go very high and slow down the system, especially when more large files are received at the same time.

One way to solve this problem is to create a custom pipeline component that receives the large file, stores it to disk and creates a small XML message that contains the information about where the large file is stored. The small XML message is stored into the MessageBox instead of the large file. It contains the same context information as the large file and it can be picked up by an orchestration or send pipeline. The sending component then gets the location to the large file and does the sending process. Deleting the large files from disk can be done in a scheduled task.

Transfer large files using BizTalk - Receive side describes the receive side on BizTalk. In this article, I describe how to get the information about where the large file is stored and send it in a custom send pipeline component. I also describe briefly how to delete the large files from disk.

The Custom Pipeline Component

Much of this code is basic when creating a custom pipeline component and can also be generated by wizards. However, I list all code below and give a short explanation. The essential code is found in the Execute method which is implemented from the IComponent interface.

The custom pipeline component is created as new Class Library in Visual Studio and signed with a strong name key file.

A reference to Microsoft.BizTalk.Pipeline.dll is added.

The code starts by adding the following namespaces to create a custom pipeline component and to read the files from disk.

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component.Interop;
using System.IO;

The code at the beginning of the class is listed below. One attribute tells that this is a pipeline component and a second restricts the component to be used only in the encode stage in a pipeline. The encoder pipeline component needs to implement the interfaces IBaseComponent, IComponentUI and IComponent.

namespace Stm.LargeFileEncoder
   public class LargeFileEncoder: IBaseComponent, 
		IComponentUI, IComponent, IPersistPropertyBag

The IBaseComponent interface provides properties that provide basic information about the components.

#region IBaseComponent
private const string _description = 
	"Pipeline component used to read large files from disk";
private const string _name = "LargeFileEncoder";
private const string _version = "";

public string Description
   get { return _description; }
public string Name
   get { return _name; }
public string Version
   get { return _version; }

The interface IComponentUI defines a method and property that are used within the Pipeline Designer environment. To keep it simple, I have not provided any code here.

#region IComponentUI
private IntPtr _icon = new IntPtr();
public IntPtr Icon
   get { return _icon; }
public System.Collections.IEnumerator Validate(object projectSystem)
   return null;

The core interface is IComponent. In this case, it contains one method that executes the pipeline component and reads the large file from disk and returns it as a message.

#region IComponent
public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
   // Read filepath from the context properties            
   string largeFilePath = pInMsg.Context.Read("LargeFilePath", 

   // Read file from disk
   int bufferSize = 1024; // 4096
   FileStream fs = new FileStream(largeFilePath, FileMode.Open, 
		FileAccess.Read, FileShare.Read, bufferSize);

   if (fs.CanSeek)           
      fs.Position = 0;
   pInMsg.BodyPart.Data = fs;
   return pInMsg;

As mentioned in the article Transfer large files using BizTalk - Receive side, a schema for the small XML message is deployed and in this case the field that contains the location to the large file is promoted. The first thing to do in this method is to read that location from the message. Next, a FileStream reads from the file using an internal buffer. The Data property of the BodyPart of the message is set to the FileStream and the message is returned. There is no code to delete the large file here. If an error occurs in a send adapter and BizTalk is set to resend the message after a period, the large file cannot be deleted.

The Send Pipeline

When the component above is built, it can be stored in the same folder as the other pipeline components on the BizTalk Server. In a BTS send pipeline project, this component DLL must be added to the toolbar and used in the encode stage and the pipeline must be deployed.

Testing on BizTalk

To test this project, the components on the receive side(a custom decoder component, a schema and a receive pipeline), the custom encoder component and the send pipeline need to be built and deployed. The large file will be stored as a .msg file on disk and the output from the send port will be a unique name plus the original name when using %MessageID%%SourceFileName% as macros in the send port. This works fine for all types of files.

Deleting the Large Files

Deleting the large files can be done in a scheduled task. In cases where there are more partners sending and receiving large files, there can also be more locations where the large files are stored and need to be deleted.

One way to do this is to create an XML file that contains one node for each of these locations. These nodes contain the path to the directory where the large files are stored. A .NET assembly can be set up as a scheduled task and read the directories from the XML file and delete each file in each directory. Check first if the file is not used by another process if the scheduled task is run often.

There are many ways to delete the large files and to store the path to the directories. This is just one simple suggestion.

Related Articles

The article Transfer large files using BizTalk - Receive side describes the receive side on BizTalk.

The article Transfer extremely large files using Windows Service and BizTalk Server describes how to transfer extremely large files (up to 2GB) using Windows Service and BizTalk.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

No Biography provided

Comments and Discussions

QuestionError : Object reference not set to an instance of an object. Pin
Member 1107234513-Mar-15 3:42
MemberMember 1107234513-Mar-15 3:42 
GeneralGetting Object reference not set to an instance of an object. error Pin
sarvanvs1232-Aug-12 0:05
Membersarvanvs1232-Aug-12 0:05 
GeneralRe: Getting Object reference not set to an instance of an object. error Pin
Member 1107234513-Mar-15 3:41
MemberMember 1107234513-Mar-15 3:41 
GeneralSend Host Adapter not closing filestream Pin
WernerDT12-May-11 23:48
professionalWernerDT12-May-11 23:48 
GeneralRe: Send Host Adapter not closing filestream Pin
Steinar Moen from Norway28-May-11 17:34
MemberSteinar Moen from Norway28-May-11 17:34 
Hi Thank you for the message!
When using FileStream we have no idea of how long it is or when it will end. The only thing we can control is the size of the data that we read at any one time (the buffer size). That means that one part of the file has already been processed in by the pipeline and sent through the send adapter while some part of the file is still in the pipeline component. Another part of the file has not even been read. That is why we cannot close the stream or delete or change the file name in the pipeline component.

What if the connection to the ftp server is down or for some reason the transfer stops? If you want to resend the file it still needs to be on the disk, so a good idea is to wait a little while before you delete the file.

In my case I have made a .Net assembly to delete the files and I have set up this as a Scheduled Task. It works fine.
The name of the file is the InterchangeID GUID for the message flow. It is possible to read this from the MessageBox and find out if the message is suspended if you want to avoid deleting files that have stopped in the send process.
GeneralRe: Send Host Adapter not closing filestream Pin
WernerDT31-May-11 21:57
professionalWernerDT31-May-11 21:57 
GeneralRe: Send Host Adapter not closing filestream Pin
Steinar Moen from Norway6-Jun-11 21:01
MemberSteinar Moen from Norway6-Jun-11 21:01 
GeneralRe: Send Host Adapter not closing filestream Pin
WernerDT8-Jun-11 1:52
professionalWernerDT8-Jun-11 1:52 
GeneralMy vote of 5 Pin
Member 432084426-Apr-11 2:45
MemberMember 432084426-Apr-11 2:45 
GeneralRe: My vote of 5 Pin
Steinar Moen from Norway26-Apr-11 22:48
MemberSteinar Moen from Norway26-Apr-11 22:48 
GeneralMy vote of 5 Pin
rue00120-Apr-11 23:58
Memberrue00120-Apr-11 23:58 
GeneralRe: My vote of 5 Pin
Steinar Moen from Norway22-Apr-11 18:18
MemberSteinar Moen from Norway22-Apr-11 18:18 
GeneralMy vote of 1 Pin
srikanthpp20-Apr-11 1:40
Membersrikanthpp20-Apr-11 1:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Posted 19 Apr 2011

Tagged as


6 bookmarked