Socket-based HTTP Client with Bandwidth Limit

• Download HttpLibrary with sample solution - 839.53 KB

Introduction

Communication over HTTP is a very common task. Most of the .NET programmers are familiar with classes like WebRequest, WebResponse or WebClient which are very useful, but sometimes one needs features they don't offer. In these cases, we could not have another option but to write our own HTTP client implementation based on TCP sockets. This article is an example of how to do so. The reader should know the basics of HTTP protocol.

Background

My first reason for writing my own HTTP client library was the need to upload large files to the server and show the current upload speed. It didn't seem to be very difficult, but I found out standard classes load all output data into memory before send despite the face that the WebRequest class offers an output stream for POST data. HTTP protocol is not really complex, so why not just open a socket on port 80 and start communication?

I didn't want to rewrite all my application but just to do some experiments first to ensure my idea is sustainable. I needed to create an abstract interface, which enables me to return to standard solution at any time. These are my classes:

• HttpConnection - represents an abstract connection to the web server which can handle only one request at a time
• HttpSocketConnection - new client implementation based on TCP sockets
• HttpWebRequestConnection - implementation using standard WebRequest and WebResponse classes
• HttpMessage - container for HTTP request or response including headers and data stream

These classes are all you need to run simple HTTP requests with data uploading/downloading in both ways. I also prepared some code which enables control bandwidth used by network communication. This can be accomplished simply by slowing down read streams - first for the POST data and second for server response, so I created a stream proxy class slowing down read/write operations for a given stream held inside the proxy.

• WatchedStream - Serves as a proxy for given stream calling events before every read/write operation
• WatchedStreamEventArgs - Event arguments for stream transfer events
• HttpAdvancedStream - Uses the events from base class to insert some wait time
• BandWidthManager - Responsible for measuring time and computing transferred data to keep the desired speed

If you would like to use the library to send POST requests, you could use some framework to create POST body. Of course I have created one, and it can prepare request body in two common formats: simple URL encoded and multi-part format.

• HttpPostBodyBuilder - abstract class for POST body builders
• HttpPostBodyBuilder.Multipart - Prepares a request body in standard multipart format which allows you upload files
• HttpPostBodyBuilder.UrlEncoded - Prepares a request body in standard URL format
• MergingStream - This stream reads data from multiple given streams, which allows to prepare an entire request body without the need to load all necessary data into memory
• HttpUtility - URL encoding logic extracted from System.Web.dll. This allows you to avoid referencing this library and keep application under .NET Framework Client Profile.

Using the Code

I have prepared and attached a client/server sample solution for you to test and better understand the library. There is an ASP.NET web site which allows you to upload image, then it does some graphics operations and sends the result back to the browser. To invoke the service, you can either use your standard browser or the WinForms client emulating browser behavior.

This block of the code is crucial:

this.NotifyState("Converting file...", 0);

// choose connection
HttpConnection http;
switch (httpMethod)
{
	case 0:
		http = new HttpSocketConnection();
		break;
	case 1:
		http = new HttpWebRequestConnection();
		break;
	default:
		throw new NotSupportedException();
}

// prepare request
var url = "http://localhost:12345/Page.aspx";
var postBody = new HttpPostBodyBuilder.Multipart();
var fileStream = new FileStream
		(this.openFileDialog.FileName, FileMode.Open, FileAccess.Read);
var advStream = new BandwidthControlledStream(fileStream);
lock (this.bandWidthSync)
{
	this.uploadSpeed = advStream.ReadSpeed;
	this.uploadSpeed.BytesPerSecond = 1024 * (int)this.upSpeedBox.Value;
}
postBody.AddData(
	"imageFile",
	advStream,
	Path.GetFileName(this.openFileDialog.FileName),
	GetMimeType(this.openFileDialog.FileName)
	);
var bodyStream = postBody.PrepareData();
bodyStream.Position = 0;
var req = new HttpMessage.Request(bodyStream, "POST");
req.ContentLength = bodyStream.Length;
req.ContentType = postBody.GetContentType();
req.Headers["Referer"] = url;

// send request
advStream.BeforeRead +=
	(s, e) => this.NotifyState("Uploading...", e.Position, bodyStream.Length);
var response = http.Send(url, req);

// get response
var readStream = new BandwidthControlledStream(response.Stream);
lock (this.bandWidthSync)
{
	this.downloadSpeed = readStream.ReadSpeed;
	this.downloadSpeed.BytesPerSecond = 1024*(int) this.downSpeedBox.Value;
}
readStream.BeforeRead +=
	(s, e) => this.NotifyState("Downloading...", e.Position, response.ContentLength);
this.convertedFile = ReadAll(readStream, (int) response.ContentLength);
this.NotifyState("Done", 100, true);

As you can see, the code snippet is not quite short, but it does a lot of work:

• Chooses method to upload data (WebRequests/sockets)
• Prepares request body with file to upload (file is loaded continuously during sending)
• Prepares speed-limited streams (you can also choose not to use them, or change speed during transfer)
• Uses BeforeRead events to inform user about progress
• Shows how to set request headers

Limitations

Please don't consider this library as full replacement of HttpWebRequest and other classes. This is only a simple solution and cannot be used in any scenario because of its limits:

• No support for HTTPS
• No support for proxy
• No caching
• Does not keep opened socket
• Only few HTTP statuses are treated correctly (100, 206, 302 and of course 200)
• Only seek-capable streams can be used as source for upload

History

• 1.0 - Initial version