Dime Buffered Upload

• Download demo app (WinForms, Web Service) - 493 Kb
• Download source (WinForms, Web Service) - 545 Kb

Notice: New article available

This article uses version 2 of the Web Service Enhancements (WSE) and .NET version 1.1. I have put together a new article with WSE 3 and .NET 2.0, using MTOM instead of DIME, which is faster and simpler to use. This article is being left for people who need a .NET 1.1 / WSE 2 solution.

Summary

This article documents a solution to upload large files over an HTTP web service, overcoming IIS request timeouts and IIS/ASP.NET request size limits. As well as providing a reliable transport system for large files, it enables a Windows Forms application to provide regular feedback on the status of the transfer, using a progress bar.

Introduction

If you have ever developed a Windows Forms application that sends files (possibly large) to a web server with a HTTP web service, you will probably already know how poorly the .NET framework handles this task. Web services were designed to send single messages, but us developers being the creatures we are, started using them to send arrays of bytes, i.e. the binary contents of a file. This approach is not scalable for large files because IIS has a request time out, and a maximum request size, which puts constraints on the size of any web service request we might send to the server. You are also leaving your user to wait possibly a long time until the success or failure of the operation is returned. You may also encounter OutOfMemoryExceptions in your client application, or on the server (since the array of bytes must be stored all at once in memory).

Acknowledgement to Tulio Rodriguez

This article originally outlined a solution to split a file into X number of file fragments, and then send them all to the server, where they would be combined to reproduce the original file on the server. This was a fast way of achieving 'chunking' over HTTP and providing regular feedback to the user (provided the chunks were small enough). The problem with this is obviously the risk of orphaned files scattered on the client and server computer, if any unexpected errors occurred.

This article has now been updated based on some code that Tulio Rodriguez kindly gave me permission to use and post here. His approach was not to split the file into fragments, but to send byte arrays one at a time to the server, which simply appended each chunk to a file on the server. This is a much more elegant solution, and he gets all the credit for the idea to use this approach. I made the following changes to his code:

• added performance improvements, fixed bugs, removed unnecessary code.
• added structured exception handling, rather than returning error codes.
• implemented Dime attachments instead of byte arrays. This increases performance and reduces bandwidth. The reason for this is because bytes are serialised into XML when sent as an array to a normal web method. Dime allows unserialised binary bytes to be sent as an attachment to the XML message, which avoids the costly 'padding' applied to the XML during the serialisation and deserialisation process.
• added support for very large files.

Alternative Approaches

Before you go implementing this into your application, check out other options that may save you time and effort:

• Use TCP or UDP with the Dime Chunking feature of Web Services Enhancements (WSE). WSE version 2 supports 'chunking' where attachments are broken up into small parts and sent individually, but this doesn't work over HTTP which many applications prefer to use for firewall and deployment issues. If you can use TCP or UDP, you'd probably be better off going that route.
• If you just want a progress bar for a normal web service, and you aren't worried about time-outs or request size limits, there is a well documented solution by Matt Powel, which intercepts the outgoing traffic and reports back to your progress bar. I found this solution a bit messy because you need to add a SOAP extension to the application, which complicated my deployment scenario. But his solution won't solve IIS timeouts or max request length problems, as far as I can tell.

Step by Step overview

In the demo application provided with this article, there is a Windows Forms application, which connects to a web service on a Web application. Files are sent from the client application and saved on to the web server, via the web service.

• The user selects a file in the Win-Forms application, and can specify the size used to upload each chunk. The user then clicks the Upload button.
• An 'instance' object is created on the server, which manages the upload on the server and keeps track of the file. This object is kept in the session state on the server, to be used by each AppendChunk() method invoked by the client application.
• The upload is started on a worker thread, which sends the chunks one by one, until the upload is complete. The server receives each chunk and appends it to the file. Some error checking is done to ensure that the chunks are appended at the right location.
• The worker thread also raises an event each time a chunk is sent to allow the user interface to update the progress bar and/or status bar message.
• After the upload is complete, the 'instance' object is removed from the session state to release memory on the server.

Performance

In my tests in a local network environment, I uploaded a 2.5 gigabyte DVD ISO image, split into 16 MB chunks, in just under 8 minutes. (I had to change some data types from int to long to accommodate the massive file size!) 16 MB chunks are obviously very large, but probably appropriate in an intranet environment. In my CMS app that has clients connecting over dial-up connections, I have the chunk size set to 32 Kb, which provides quick feedback to the user all the way through the upload.

During some performance analysis, I noted that for fast networks the chunk size did not have a noticeable effect on the time taken for each AppendChunk() operation. It is therefore more efficient to go for the biggest chunk size your network can accommodate, while still providing a reasonable frequency of updates to the progress bar.

Installation Requirements

• A virtual directory called UploadWeb which should point to the UploadWeb folder.
• Anonymous permissions on the virtual directory.
• Write permissions for the 'Uploads' folder in the virtual directory.

Possible Compile Errors

If you want to edit the source and recompile the application, you must download WSE (Web Service Enhancements) for Visual Studio from Microsoft. Some users have had trouble updating the web service reference, because without WSE, Visual Studio won't create a WSE version of the proxy class. If you get any of the compile errors below, then not having WSE installed for Visual Studio is likely the problem.

• 'BufferedUploadWin.BufferedUpload' does not contain a definition for 'RequestSoapContext'
• The type or namespace name 'UploadWse' does not exist in the class or namespace 'UploadWin.BufferedUploadServer' (are you missing an assembly reference?)

If you have problems with WSE, please visit the MSDN site for WSE.

Code Summary

• Step 1: the user clicks the Upload button

  First of all we create a 'BufferedUpload' object. This is a helper class for the WinForms application, to send the file chunks to the web service. As you will see from the code below, the chunk size is set to the value in the "Chunk Size" Domain-Up-Down control. Two event handlers are added, one for updating the progress bar, and the other for responding to changes in the state of the upload, i.e. Completed, Failed, etc. The 'BeginUpload()' method is asynchronous so the application remains responsive during the upload.

  ws = new BufferedUpload();
  ws.ChunkSize = Int32.Parse(this.dudChunkSize.Text)*1000;    // kilobytes
  
  ws.ProgressChanged += new ProgressChangedEventHandler(Upload_ProgressChanged);
  ws.StateChanged += new EventHandler(Upload_StateChanged);
  ws.BeginUploadChunks(this.textBox1.Text);

• Step 2: initialise the 'Instance' object on the server and begin uploading the chunks

  The code below outlines a simplified version of the upload routine. It is essentially a while loop that keeps reading bytes into a buffer, attaching the bytes as a Dime attachment (via MemoryStream) and sending the attachment to the web service. Further explanation of the code can be found in the comments in the source code.

  FileStream fs = new FileStream(Filename, FileMode.Open, FileAccess.Read);
  int bytesRead = 0; // used to store the number of bytes written each time 
  
  byte[] buffer = new byte;
  if (this.sentBytes == 0) 
  {
      // initialise an upload Instance to manage the file I/O for this upload
  
      this.instanceId = this.Initialize(Path.GetFileName(Filename), 
                                                   this.overwrite);
  }
  bytesRead = fs.Read(buffer, 0, bufferSize);
  while (bytesRead > 0 && this.state != UploadState.Aborting) 
  { 
      // copy the byte buffer into a memory stream
  
      // to send in a dime attachment
  
      MemoryStream ms = new MemoryStream(buffer, 0, buffer.Length); 
      DimeAttachment dimeAttach = new DimeAttachment("image/gif", 
                                       TypeFormat.MediaType, ms);
      this.RequestSoapContext.Attachments.Add(dimeAttach); 
      this.AppendChunk(this.instanceId, sentBytes, bytesRead);
      this.sentBytes += bytesRead; 
      bytesRead = fs.Read(buffer, 0, bufferSize); // read the next chunk (if it 
  
  }

• Step 3: (on the server) Append each chunk to the file

  The AppendChunk() web method simply reads the Dime attachment back into a byte[] and calls the Instance.AppendChunk() method, as shown below:

  public void AppendChunk(byte[] buffer, long offset, int bufferSize)
  {
      // make sure that the file exists
  
      if (!System.IO.File.Exists(this.tempFilename))
          CustomSoapException("Cannot Find Temp File", this.tempFilename);
               // if the temp file size is not the same
  
               // as the offset then something went wrong....
  
      if (tempFilesize != offset) 
          CustomSoapException("Transfer Corrupted", 
             String.Format("The file size is {0}, expected {1} bytes", 
             tempFilesize, offset));
      else
      {
          // offset matches the filesize, so the chunk
  
          // is to be inserted at the end of the file.
  
          FileStream FileStream = new FileStream(this.tempFilename, 
                                                  FileMode.Append);
          FileStream.Write(buffer, 0, bufferSize);
          FileStream.Close();
      }
  }

Integrating this code in your own application

In your Win-Forms app, add the 'BufferUpload.cs' file, and follow the logic used in the demo app. You don't need WSE2 installed on your PC to make this work, because the DLL has been bundled with the demo app. Simply reference this DLL in your WinForms and WebForms projects.

In your web app, add the 'Instance.cs' and 'upload.asmx' files (with code-behind for the .asmx), and make sure the Win app is referencing the correct URL for your web service. Also make sure to include the WSE sections in your web.config (copy them from the one in the demo app).

Extras in the Windows Forms App

You'll notice in the demo app there is a wait cursor icon displayed to the left of the status bar. This conveys a further sense of 'busy' to the user as well as the progress bar. I borrowed this trick from the FotoVision .NET sample application. Also, in case you're wondering, the progress bar isn't actually part of the status bar, it is just placed on top of it and anchored to the corner. I put these two features together into a 'FormBar' control which is supposed to be some kind of general purpose progress tool for Win-Forms apps. It provides methods to update the status bar message and the progress bar, which can be safely called from the UI thread, or from worker threads.

I also threw in a simple 'Retry' feature. It is a form that pops up if the upload fails for any reason, and picks up from the last good position if a chunk failed to arrive. To test this out, pause your web server mid-transfer and you'll see the Retry window pop up. It will keep failing until you resume the web server.

Conclusions

I've spent considerable effort over the last two years on and off trying to find a suitable solution to uploading large files over HTTP, and I think this is the simplest and most efficient approach so far. It's very robust and the fact that it supports resume functionality is an added bonus. Also, being able to reliably upload a DVD ISO image is not bad in my opinion!

Comments

If you have any problems / suggestions, post them below.

Enjoy!