Click here to Skip to main content
Click here to Skip to main content

zipstream, bzip2stream: iostream wrappers for the zlib and bzip2 libraries

, 2 Oct 2003
Rate this:
Please Sign up or sign in to vote.
STL compliant, stream-to-stream, zlib and bzip2 wrapper with wide char support.

Introduction

This article presents two zipped STL iostream implementation based on the library zlib (see download link above) and bzip2 (see download link above). This means that you can easily manipulate zipped streams like any other STL ostream/istream.

To give you an idea, consider following snippet that prints "Hello World":

ostringstream output_buffer;
// writing data
output_buffer<<"Hello world"<<endl 

Now, the same snippet but with zipped output using zlib:

// zip_ostream uses output_buffer as output buffer :)
zip_ostream zipper( output_buffer );

// writing data as usual
zipper<<"Hello world"<<endl 

or using bzip2:

// zip_ostream uses output_buffer as output buffer :)
bzip2_ostream bzipper( output_buffer );

// writing data as usual
bzipper<<"Hello world"<<endl 

As you can see adding zipped buffers into your existing applications is quite straightforward. To summarize, let's see some quick facts about zipstream and bzip2stream:

  • STL compliant,
  • any-stream-to-any-stream support,
  • char, wchar_t support,
  • fining tuning of compression properties,
  • support custom allocators (New!)

Why another wrapper? Why not use gzstream?

Writing wrappers around the zlib library is popular on CodeProject. If you search for 'zip' you will find at least 14 articles on the topic. Moreover, if you crawl on the web and especially on zlib home page, you can find dozens of other wrappers.

So why another wrapper? Well, none of the wrappers are fully STL compliant. Ok, this is not true since gzstream (see download link above) implements fstream-like STL streams. However, gzstream has three drawbacks:

  1. It does not allow buffer to buffer compression since it is based on gzip i/o methods: only file to buffer or buffer to file are supported,
  2. it is licensed under LGPL which makes it difficult to use in commercial apps,
  3. it does not support wchar_t

Last reason to write this wrapper: it is a good exercise to understand and implement iostreams.

Wrapper architecture

The three drawbacks of gzstream pushed me to re-implement an STL wrapper for zlib (and later on, do some cut and paste to get bzip2 working).

This wrapper takes a user defined i/ostream to write or read compressed data. This approach is quite flexible since the user can give any stream (istringstream, ifstream, or custom stream) to store or load the compressed data.

Internally zip_stream acts as a triple buffer: the streambuf object in itself, zlib library and the user-defined stream. For example, during the compression process, the buffers are used:

  • first buffer: the data to compress is buffered into a streambuf object,
  • second buffer: when overflow is called, the first buffer data is sent to zlib which also buffers the data internally. If zlib outputs data, it is sent to the user-defined stream,
  • third buffer: the user-defined stream is buffered.

Some care must be taken when flushing: you must use the method zflush that will first flush the streambuf, then flush the zlib buffer, then flush the user-defined stream. Note that you should avoid flushing as it degrades compression.

Implementing iostreams

Since I'm not an STL expert, I will very briefly discuss this part. There's room for a tutorial on this topic...

To implement custom iostream, you need to take the following steps:

  • implement a custom my_streambuf, inherited from streambuf. You need to override the virtual methods sync, underflow and overflow. sync and overflow are used in output streams and underflow is used in input streams.
  • implement a custom my_ostream, inherited from ostream. It will use my_streambuf
  • implement a custom my_istream, inherited from istream. It will use my_streambuf as stream buffer.

Class quick reference

All the zlib classes are in the zlib_stream namespace and all the bzip2 classes are in the bzip2_stream namespace.

The two main classes of the zlib wrapper are basic_zip_ostream and basic_zip_istream which implement respectively compression and decompression and behave like classic basic_ostream and basic_istream.

Classical typedef are also provided for these classes:

  • zip_ostream, zip_istream for char streams
  • wzip_ostream, wzip_istream for wchar_t streams.

The bzip2 classes have similar names, just replace zlib by bzip2: basic_zip_streambuf becomes basic_bzip2_streambuf.

basic_zip_ostream

This class inherits from basic_ostream:

template<
    typename Elem, 
    typename Tr = char_traits<Elem;>,
    typename ElemA = std::allocator<Elem>,
    typename ByteT = unsigned char,
    typename ByteAllocatorT = std::allocator<ByteT> 
    >
basic_zip_ostream : public basic_ostream<Elem, Tr>

where

  • Elem,Tr are the classical basic_ostream template parameters,
  • ElemA is the allocator for a Elem buffer used internally,
  • ByteT is the byte type used internally (you should not change that),
  • ByteAT is a custom allocator for a ByteT buffer used internally

Constructor

basic_zip_ostream( 
    ostream_reference ostream_, 
    bool is_gzip_ = false,
    size_t level_ = Z_DEFAULT_COMPRESSION,
    EStrategy strategy_ = DefaultStrategy,
    size_t window_size_ = 15,
    size_t memory_level_ = 8,
    size_t buffer_size_ = 4096
);
  • ostream_ is a user defined output stream,
  • is_gzip_, true if you want to add the gzip header and footer,
  • level_, compression level 0, bad and faster to 9 max and slower,
  • strategy_, compression strategy, see EStrategy enum,
  • window_size_, memory_level_ are advanced zlib settings, check zlib manual,
  • buffer_size_, read buffer size

Note that if you choose the gzip option, a header will be automatically added in the constructor and the gzip footer (CRC + data size) will be added in the destructor.

Other methods

  • Flush all buffers (zlib and ostream):
    basic_zip_ostream& zflush()

    This method must be called before using the compressed data! Since zlib does it's own buffering and ostream::flush is not virtual there is no way to avoid this problem.

  • Return the CRC of the uncompressed data:
    long get_crc();
  • Return the uncompressed data size:
    long get_in_size();
  • Return the compressed data size:
    long get_out_size();

Predefined typedefs

typedef basic_zip_ostream<char> zip_ostream;
typedef basic_zip_ostream<wchar_t> zip_wostream;

basic_zip_istream

This class inherits from basic_istream:

template<
    typename Elem, 
    typename Tr = char_traits<Elem;>,
    typename ElemA = std::allocator<Elem>,
    typename ByteT = unsigned char,
    typename ByteAT = std::allocator<ByteT> 
    >
basic_zip_istream : public basic_istream<Elem, Tr>

Constructor

basic_zip_istream( 
    istream_reference istream_, 
    size_t window_size_ = 15,
    size_t read_buffer_size_ = 1024 * 10,
    size_t input_buffer_size_ = 1024 * 5
)
  • istream_, input stream containing the compressed data,
  • window_size_, should be compatible with compression window size,
  • read_buffer_size_, size of the streambuf buffer size,
  • input_buffer_size_, size of the zlib input buffer size

Other methods

  • Tells if it is a gzip file:
    bool is_gzip() const
  • Checks CRC (must be a gzip file)
    bool check_crc() const
  • Return the CRC of the uncompressed data:
    long get_crc() const;
  • Return the uncompressed data size:
    long get_out_size() const;
  • Return the compressed data size:
    long get_in_size() const;

Predefined typedefs

typedef basic_zip_istream<char> zip_istream;
typedef basic_zip_istream<wchar_t> zip_wistream;

How to ...

All the following examples are valid for both zlib and bzip2 wrappers.

Compress to a buffer

ostringstream buffer;
zip_ostream zipper(buffer);

// writing stuff
zipper<<...

//flushing VERY IMPORTANT!
zipper.zflush();

// buffer.str() is ready

Compress to a file

ofstream file("test.zip",ios::out | ios::binary);
{
zip_ostream zipper(file, true /* gzip file*/);

// writing stuff
zipper<<...

} // the stream is flushed, the destructor is called and gzip header appended
// the file is ready

Decompress from a buffer

istringstream buffer;
zip_istream unzipper(buffer);

// reading stuff
unzipper>>...

Decompress from a file

ifstream file("test.zip", ios::in | ios::binary);
zip_istream unzipper(file);

// reading stuff
unzipper>>...

// if the file was gzip, we can check the crc
if (unzipper.is_gzip())
    std::cout<<"crc check: "<<( unzipper.check_crc() ? "ok" : "failed");

Using it in your project

Zlib wrapper

  • Read the license terms,
  • Copy zipstream.hpp and zipstream.ipp in your include directory,
  • Make sure zlib is available,
  • Add #include "zlibstream.hpp" to include the headers,

bzip2 wrapper

  • Read the license terms,
  • Copy bzip2stream.hpp and bzip2stream.ipp in your include directory,
  • Make sure zlib is available,
  • Add #include "bzip2stream.hpp" to include the headers,

History

  • 30-09-2003, 1.7, added custom allocators (suggestion of <forgot, send me your e-mail>), fixed warning in ostream constructors
  • 21-09-2003, fixed bugs with CRC and size, writing thanks to Jeroen Dirks and gigimenegolo
  • 08-08-2003, 1.5
    • Fixed gzip footer problem: CRC is read
    • data is put back in the buffer when zip file has finished
  • 07-18-2003, 1.4 Fixed gzip header problem.
  • 07-3-2003, 1.3, add bzip2 wrapper
  • 07-2-2003, 1.2, wchar_t working
  • 07-2-2003, 1.1, fixed bugs in gzip header and zip_to_stream
  • 07-01-2003, 1.0, initial release

Reference

License

These zlib and bzip2 wrappers are licensed under the zlib/libpng license.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Jonathan de Halleux
Engineer
United States United States
Jonathan de Halleux is Civil Engineer in Applied Mathematics. He finished his PhD in 2004 in the rainy country of Belgium. After 2 years in the Common Language Runtime (i.e. .net), he is now working at Microsoft Research on Pex (http://research.microsoft.com/pex).

Comments and Discussions

 
Questiongithub hosting!? PinmemberRRAALLVV30-Jun-14 12:29 
Generalx64 warnings! Pinmemberare_all_nicks_taken_or_what5-Mar-10 11:43 
Questionzipstream.hpp and zipstream.ipp? Pinmemberdbbtvlzfpz27-Oct-08 12:30 
AnswerRe: zipstream.hpp and zipstream.ipp? PinmemberTux`1-Apr-09 10:45 
GeneralFixes for modern STL (gcc 4) PinmemberNoGoodNicks6-Oct-07 9:58 
Generalzipstream is not necessary anymore, you can use the boost::iostreams lib instead Pinmemberherculon27-Jan-06 3:30 
Generalzipstream with boost::archive Pinmemberjhsy11-Aug-05 20:12 
GeneralRe: zipstream with boost::archive Pinmemberherculon12-Aug-05 4:47 
Generalsmall comment on namespace Pinmemberjhsy11-Aug-05 19:40 
Generalbzip2stream file &lt;-&gt; file works fine, zipstream file2file fails. Pinmemberherculon25-Jul-05 10:07 
Questionworking on linux? Pinmemberdekim19-May-05 13:06 
AnswerRe: working on linux? PinsussAnonymous8-Jun-05 16:13 
GeneralRe: working on linux? Pinmemberirotas28-Oct-05 18:09 
QuestionReading zipped file?? PinmemberRobert Bielik28-Apr-05 3:51 
GeneralProblem --Stream and ZipStream are in same scope Pinmembertakuni3-Feb-05 20:01 
Generalbzip2stream fails with wide file streams PinmemberDenis Dubrov7-Jan-05 10:05 
Questionsupport for seekg/tellg? Pinsussgreglandrum1-Dec-04 12:42 
Generalzipstream cannot unzip the zipped file Pinsusslost in templates20-Oct-04 17:01 
QuestionHow to compress Huge Memory Buffer ? PinmemberWanaDev11-Feb-04 18:15 
GeneralZlib Wrapper and GZip generated buffer on IIS PinsussAnonymous10-Feb-04 17:39 
GeneralCompiling with VC6 PinsussMarlon Gaspar26-Jan-04 9:43 
GeneralRe: Compiling with VC6 PinmemberVattila13-Mar-04 7:04 
GeneralRe: Compiling with VC6 PinmemberVattila15-Mar-04 17:10 
GeneralRe: Compiling with VC6 PinmemberJonathan de Halleux15-Mar-04 21:36 
GeneralRe: Compiling with VC6 PinmemberPatrik Müller24-Aug-04 0:46 
GeneralRe: Compiling with VC6 PinsussAnonymous24-Aug-04 23:13 
GeneralRe: Compiling with VC6 PinmemberPatrik Müller25-Aug-04 22:14 
GeneralRe: Compiling with VC6 PinmemberPatrik Müller25-Aug-04 22:16 
GeneralRe: Compiling with VC6 PinmemberPatrik Müller26-Aug-04 22:38 
GeneralRe: Compiling with VC6 PinmemberJonathan de Halleux27-Aug-04 0:19 
GeneralRe: Compiling with VC6 PinmemberHesham Desuky21-Jun-04 23:47 
GeneralRe: Compiling with VC6 PinsussChristian Steer15-Aug-04 11:55 
GeneralRe: Compiling with VC6 Pinmemberkamarali24-Nov-04 0:07 
GeneralRe: Compiling with VC6 Pinmemberjazzwithshaz13-Jan-05 20:22 
GeneralRe: Compiling with VC6 (download) PinmemberVattila5-Mar-05 15:49 
GeneralRe: Compiling with VC6 (download) PinmemberRobert Bielik18-May-05 23:30 
GeneralRe: Compiling with VC6 (download) PinmemberNandakumar Raghu17-Apr-09 3:00 
GeneralRe: Compiling with VC6 Pinmemberakleinbe13-Sep-05 3:11 
GeneralRe: Compiling with VC6 Pinmemberakleinbe13-Sep-05 3:18 
Questionwrong version? Pinmemberpondor5-Jan-04 17:04 
GeneralProblems with endl PinmemberOwen M. Astley2-Jan-04 3:23 
GeneralRe: Problems with endl PinmemberJonathan de Halleux3-Jan-04 13:17 
Generalspeed Pinmemberishayr22-Oct-03 2:02 
Generalzip to unzip problem Pinmemberyongkieyong20-Oct-03 3:47 
GeneralRe: zip to unzip problem PinsussAnonymous20-Oct-03 11:11 
GeneralRe: zip to unzip problem Pinmemberyongkieyong20-Oct-03 21:53 
GeneralRe: zip to unzip problem PinmemberJonathan de Halleux20-Dec-03 23:58 
GeneralRe: zip to unzip problem PinmemberYongki C. A. Jong25-Dec-03 0:39 
GeneralSTLPort bug ! PinmemberLuc Bergeron9-Oct-03 6:37 
GeneralRe: STLPort bug ! PinsussAnonymous20-Oct-03 11:13 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.1411023.1 | Last Updated 3 Oct 2003
Article Copyright 2003 by Jonathan de Halleux
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid