Click here to Skip to main content
11,432,427 members (68,830 online)
Click here to Skip to main content

Adding a zip filter to web services

, 16 May 2006
Rate this:
Please Sign up or sign in to vote.
Passing large amounts of data through web services can become a huge bottle-neck in a WAN application architecture (i.e., server is on the web), and in any case, a real load on the network traffic. This is one solution for downsizing the network costs.

Introduction

Hold on, I know there is no picture involved, give it a chance all the same, especially if you are a Web services developer Wink | ;-)

While writing WebServices, in cases where the client and the server have to communicate through the web, the cost of transferring data through the network is becoming an issue, costly both in the aspect of general responsiveness and the network load.

This lib Zips the SOAP envelope body at the server side, and unzips it back in the client side, behind the scenes, through the use of SOAP filters.

Background

There are some common recommendations that one should follow when writing such an app, amongst which you can find the following list:

  1. Strive to call your server once per page - usually, the actual cost is accessing back and forth to the server, and not passing the actual data.
  2. Cache data on the client side to reduce the need to access the server whenever possible.
  3. Design your app to pass only the exact necessary info (i.e., don't make a "all suppliers" screen, instead make a "search a specific supplier" screen).

Having that said, sometimes you do need to pass a big chunk of data through the web, usually a chubby DataSet, with thousands of rows or alike. This is where my code comes into the picture.

I have written a Zip and a corresponding Unzip filter that you very simply add to your Web service (adding a reference and a filter declaration in the web.config file) and to your WinForms client (again, a reference and a line of code), and you're set.

There is absolutely no code changes to make nor any other overhead.

The code description

The code is a DLL that should be referenced by both projects - client and server, and consists of 3 simple classes:

The ZipWrapper

ZipWrapper is a wrapper class around the ICSharpZipLib library (a freeware code library for Zip utilities taken from here), through which you can select your desired zipping method (BZ2, GZip, tar, etc.) while working with the same interface. (The code for this class was taken from this link, and was mildly amended to fit my needs.)

The Zip type I'm using is GZip; although not the best compression, it is extremely faster than its brother the BZip2, and proved more desired in our case.

The ZipFilter

ZipFilter is a SoapOutputFilter class that is in charge of zipping the body of the SOAP envelope. It also exposes two properties:

  • MinFilterSizeKB: the minimum size of body to start zipping, not all messages are worth zipping.
  • Enabled: is the filter enabled or not.

This class can also determine the zipping method.

The UnZipFilter

UnZipFilter is a SoapInputFilter class that reverses the zipped body back into its original form. The unzip filter will only operate if the Zip filter made its mark on the envelope, and hence doesn't require any configuration params.

Code snippet

We must override the ProcessMessage to change the body of the envelope through the pipeline of the SOAP mechanism. First, we verify that the filter is enabled and the message is big enough to Zip. Then, we create a custom header to announce the client that the message should be unzipped. Next, we create a zipped stream of the envelope's body element. And lastly, we replace the body with the new zipped object.

public override void ProcessMessage(SoapEnvelope envelope)
{
   if ( !m_bEnabled )
      return;

   //adding an attribute to specify that the filter has been 
   //applied on this envelope.
   XmlElement soapHeader = envelope.CreateHeader();

   if ( envelope.Body.InnerText.Length < ( m_MinFilterSize ) )
      return;
   else
      soapHeader.AppendChild(CreateCustomHeader(soapHeader, "1" ));

   //compress the body element.
   MemoryStream result = new 
      MemoryStream( ZipWrapper.Compress( 
      Encoding.UTF8.GetBytes( envelope.Body.InnerXml ) ) );

   //Attach zipped result to the envelope.
   Microsoft.Web.Services2.Attachments.Attachment attch = 
      new Microsoft.Web.Services2.Attachments.Attachment( 
      "APPLICATION/OCTET-STREAM",result );

   envelope.Context.Attachments.Add( attch );

   //remove old body.
   XmlElement newBody = envelope.CreateBody();
   newBody.RemoveAll();

   envelope.SetBodyObject( newBody );
}

Using the lib

Prerequisites

  • ICSharpZipLib.dll - look for the link in the Resources section, WSE 2.0 - Microsoft's web services enhancements package.

Server side usage

  1. Create a web service project.
  2. Right click the project in the Solution Explorer, and choose the WSE 2.0 settings.
  3. In the General tab, enable both checkboxes.
  4. Add a reference to WebServiceZipFilter.dll.
  5. In the web.config file, add under the <microsoft.web.services2> tag:
    <filters>
       <output>
          <add type="WebServiceZipFilter.ZipFilter, WebServiceZipFilter" />
       </output>
    </filters>

In order to config the filter, you should set its attributes:

// setting minimum zip size requirement to 10KB.
WebServiceZipFilter.ZipFilter.MinFilterSizeKB = 10; 
// enabling the filter
WebServiceZipFilter.ZipFilter.Enabled = true;

Please take note: it is strongly recommended to set these values through values from a configurable part in the web.config, so it can be dynamically changed according to the tuning requirements.

Client side usage

Important note: Many users out there have repeatedly asked me if it is possible to work in duplex mode and if the client side can be configured as the server does. The answer is yes and yes. The list below is an alternative code to add the filter to the pipeline, but you can add the necessary sections to the app.config file and get it done without coding (see the "Server side usage" section above). Regarding the duplex mode, both sides can zip and unzip without a worry!

  1. Create a WinForms client project.
  2. Right click the project in the Solution Explorer, and choose the WSE 2.0 settings.
  3. In the General tab, enable the first checkbox.
  4. Add the web reference of your server.
  5. Create a server proxy (instantiate the ServerNameWSE proxy class).
  6. Add the following code right after the proxy server creation:
    // creating the server
    TestServer.Service1Wse myServer = 
       new TestClient.TestServer.Service1Wse();
    
    // Adding the unzip filter
    myServer.Pipeline.InputFilters.Add( 
       new WebServiceZipFilter.UnZipFilter() );

Some empirical results

I've conducted my tests on a server, dedicated to me only, that is located in a web farm in my hometown, providing a 5MB download and 1 MB upload, using an ADSL 750 KB connection on a relaxed weekend noon, so the conditions were optimal for the non-zipping tests, and still the differences are quite notable:

Sending 570 records of heterogeneous data took around 2.281 seconds without zipping, compared with 1.843 seconds with zipping (20% reduction).

Sending 10570 records (570 heterogeneous data + 10000 different although similar dummy records) took around 29.43 seconds without zipping, compared with 6.04 seconds with zipping (80% reduction !!!). A completely heterogeneous data would have made the difference negligible.

Future versions

Will there be any? Apparently not. Is this good news? Yes! Why? Because folks at Redmond realized the compression feature is missing, and a few good men released the WCF implementation, and those same guys also released a version for WSE 3.0 here, so my mission is quite done. Everyone is happy.

Resources

History

  • 18-7-2004 v.1.0 - first release.
  • 12-5-2006 v.1.1 - fixed bug reported by tl11, zips according to byte count, not text length.

You are strongly urged to review and comment.

Happy coding.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author


Comments and Discussions

 
QuestionThis way VS System.io.compression?? Pin
zorro91128-May-08 9:41
memberzorro91128-May-08 9:41 
GeneralGreat work - you got my 5! Pin
Eli Gazit3-Sep-07 1:09
memberEli Gazit3-Sep-07 1:09 
GeneralRe: Great work - you got my 5! Pin
DaberElay3-Sep-07 1:45
memberDaberElay3-Sep-07 1:45 
GeneralWSE 2.0 and WSE 3.0 Interoperability/compatibility Pin
tcbarton15-Nov-06 15:24
membertcbarton15-Nov-06 15:24 
GeneralRe: WSE 2.0 and WSE 3.0 Interoperability/compatibility Pin
DaberElay19-Nov-06 1:17
memberDaberElay19-Nov-06 1:17 
GeneralRe: WSE 2.0 and WSE 3.0 Interoperability/compatibility Pin
tcbarton20-Nov-06 9:52
membertcbarton20-Nov-06 9:52 
GeneralPipeline Error Pin
olkor8-Aug-06 4:27
memberolkor8-Aug-06 4:27 
GeneralRe: Pipeline Error [modified] Pin
DaberElay9-Aug-06 23:31
memberDaberElay9-Aug-06 23:31 
GeneralThanks! Pin
anichin19-Jun-06 12:41
memberanichin19-Jun-06 12:41 
Generaltime taken to download without compression is faster than compressed request Pin
kkiran2-Jun-06 10:29
memberkkiran2-Jun-06 10:29 
GeneralRe: time taken to download without compression is faster than compressed request Pin
DaberElay2-Jun-06 16:44
memberDaberElay2-Jun-06 16:44 
GeneralGetting Exception during use. Pin
88Keys15-May-06 16:12
member88Keys15-May-06 16:12 
GeneralRe: Getting Exception during use. Pin
DaberElay16-May-06 7:05
memberDaberElay16-May-06 7:05 
GeneralRe: Getting Exception during use. Pin
88Keys16-May-06 10:12
member88Keys16-May-06 10:12 
GeneralHTTP SOAP Remoting Pin
Tim Friesen14-May-06 16:23
memberTim Friesen14-May-06 16:23 
Questionwhy not "Content-Encoding: gzip" in http response? Pin
NewSilence12-May-06 16:35
memberNewSilence12-May-06 16:35 
AnswerRe: why not "Content-Encoding: gzip" in http response? Pin
Jon Rista12-May-06 22:47
memberJon Rista12-May-06 22:47 
GeneralRe: why not "Content-Encoding: gzip" in http response? Pin
pblse29-Apr-07 5:41
memberpblse29-Apr-07 5:41 
GeneralRe: why not "Content-Encoding: gzip" in http response? Pin
Simon McKenzie30-Nov-11 17:53
memberSimon McKenzie30-Nov-11 17:53 
QuestionHow to change the code to work with WSE 3.0? Pin
fewfewfewfgwew7-May-06 12:19
memberfewfewfewfgwew7-May-06 12:19 
AnswerRe: How to change the code to work with WSE 3.0? Pin
DaberElay14-May-06 2:29
memberDaberElay14-May-06 2:29 
QuestionDIME Attachments? Pin
Oskar Austegard5-May-06 5:55
memberOskar Austegard5-May-06 5:55 
AnswerRe: DIME Attachments? Pin
DaberElay7-May-06 0:54
memberDaberElay7-May-06 0:54 
QuestionRe: DIME Attachments? Pin
Oskar Austegard16-May-06 5:08
memberOskar Austegard16-May-06 5:08 
AnswerRe: DIME Attachments? CORRECTED Pin
DaberElay16-May-06 7:28
memberDaberElay16-May-06 7:28 
Generalproblem with FilterSize check Pin
tl113-May-06 11:28
membertl113-May-06 11:28 
GeneralRe: problem with FilterSize check Pin
DaberElay7-May-06 0:51
memberDaberElay7-May-06 0:51 
GeneralRe: problem with FilterSize check Pin
tl118-May-06 6:57
membertl118-May-06 6:57 
GeneralRe: problem with FilterSize check Pin
DaberElay11-May-06 13:14
memberDaberElay11-May-06 13:14 
GeneralRe: problem with FilterSize check Pin
tl1112-May-06 4:30
membertl1112-May-06 4:30 
GeneralRe: problem with FilterSize check Pin
DaberElay12-May-06 5:33
memberDaberElay12-May-06 5:33 
GeneralRe: problem with FilterSize check CORRECTED Pin
DaberElay14-May-06 2:26
memberDaberElay14-May-06 2:26 
GeneralNewbie Question. Pin
tobikage6-Apr-06 0:09
membertobikage6-Apr-06 0:09 
GeneralCompressing client request sent to server Pin
deepakisonline@yahoo.com30-Mar-06 21:33
memberdeepakisonline@yahoo.com30-Mar-06 21:33 
GeneralRe: Compressing client request sent to server Pin
DaberElay24-Apr-06 11:14
memberDaberElay24-Apr-06 11:14 
GeneralThanks Pin
Eniac016-Mar-06 11:51
memberEniac016-Mar-06 11:51 
GeneralRe: Thanks Pin
DaberElay18-Mar-06 6:31
memberDaberElay18-Mar-06 6:31 
GeneralThe other SOAP Extension - Compression Extension Pin
fewfewfewfgwew10-Mar-06 7:02
memberfewfewfewfgwew10-Mar-06 7:02 
GeneralRe: The other SOAP Extension - Compression Extension Pin
DaberElay10-Mar-06 7:49
memberDaberElay10-Mar-06 7:49 
GeneralRe: The other SOAP Extension - Compression Extension Pin
fewfewfewfgwew10-Mar-06 7:58
memberfewfewfewfgwew10-Mar-06 7:58 
QuestionRe: The other SOAP Extension - Compression Extension Pin
kkiran2-Jun-06 10:18
memberkkiran2-Jun-06 10:18 
AnswerRe: The other SOAP Extension - Compression Extension Pin
DaberElay3-Jun-06 1:02
memberDaberElay3-Jun-06 1:02 
GeneralWSE question Pin
fewfewfewfgwew4-Mar-06 2:05
memberfewfewfewfgwew4-Mar-06 2:05 
GeneralRe: WSE question Pin
DaberElay4-Mar-06 11:32
memberDaberElay4-Mar-06 11:32 
GeneralThanks and a question. Pin
e_Kevin21-Feb-06 22:45
membere_Kevin21-Feb-06 22:45 
GeneralRe: Thanks and a question. Pin
DaberElay24-Feb-06 0:04
memberDaberElay24-Feb-06 0:04 
GeneralThanks + 1 question Pin
SpArtA9-Feb-06 8:17
memberSpArtA9-Feb-06 8:17 
GeneralRe: Thanks + 1 question Pin
DaberElay9-Feb-06 21:50
memberDaberElay9-Feb-06 21:50 
GeneralRe: Thanks + 1 question Pin
SpArtA11-Feb-06 5:52
memberSpArtA11-Feb-06 5:52 
AnswerRe: Thanks + 1 question Pin
SpArtA11-Feb-06 14:26
memberSpArtA11-Feb-06 14:26 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.150428.2 | Last Updated 16 May 2006
Article Copyright 2004 by DaberElay
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid