Click here to Skip to main content
Click here to Skip to main content

Pure C# MiniLZO port

, 22 Dec 2006
Rate this:
Please Sign up or sign in to vote.
Fast stream compression using a ported minilzo for .NET.

Introduction

I was recently in a position where I needed to find a fast streamable compression solution.  The great source by Markus Oberhumer provided the answer. After using the C code for a while, I realized C#/.NET lacked a similar library, short of one I found which was commercial.

As a result, I decided to attempt a minimalistic port of MiniLZO compression and decompression into pure C#. At this point, the code uses unsafe/fixed blocks to best mimic the original minilzo implementation. A pure managed solution may be soon to follow (at the cost of speed).

The code has not been unit tested 100%, but all initial tests have been successful. If anyone finds any bugs, please report them so I may fix it. I also have not done any profiling of the speed yet, if anyone feels upto the task of comparing it against the C implementation.

Use of the code is relatively simple, here is a quick example.

Compression

byte[] original = File.ReadAllBytes("...");
byte[] destination;
MiniLZO.Compress(original, out destination);

Decompression

byte[] destination = ...; /* Resulting buffer from above */
byte[] original = new byte[x];
MiniLZO.Decompress(destination, original);

Note that with compression, the destination buffer will be assigned and trimmed to the correct size. With decompression, the "original" buffer must be preallocated with the correct original size. Also note that my implementation removed all use of goto from the original source, and replaces certain uses with OverflowExceptions.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Astaelan
Web Developer
Canada Canada
Short and simple, I'm a self contracted programmer, my strongest programming skills are in C/C++ and C#/.NET. I have a nack for porting C algorithms to C#.

Comments and Discussions

 
GeneralRe: Another such library... Pinmemberomaurice22-Dec-06 13:43 
GeneralRe: Another such library... PinmemberBehind The Scene22-Dec-06 13:51 
GeneralRe: Another such library... Pinmemberomaurice22-Dec-06 14:00 
GeneralRe: Another such library... PinmemberAstaelan23-Dec-06 5:10 
I don't mean to insult, or demean you, but you do realize "normalizing a memcpy" on 2 different machines is virtually impossible right?
 
You have everything from CPU architecture, cache, clock speed, bus speeds, memory speeds and actual CPU instruction sets affecting the actual result. It is entirely unfair to say "LZF is the winner because memcpy is faster on a different computer". Keep in mind, LZO was designed and profiled on VERY low end hardware, making it even faster on high end hardware.
 
To this end, and to put the discussion to rest, I found some actual numbers showing LZO is leaps and bounds faster than LZF.
 
http://www.quicklz.com/
 
Let's look at the more important numbers involving 68MB of data.
 
LZO Compression = 81.8MB/sec
LZO Decompression = 307MB/sec
 
LZF Compression = 60.9MB/sec
LZF Decompression = 198MB/sec
 
And just for reference, someone who recommended zlib based stuff that's part of .NET now, here's a reference for you:
 
ZLIB Compression = 7.45MB/sec
ZLIB Decompression = 120MB/sec
 
Note that if you check QuickLZ on the page, in this case seems the dominating, but also note that version 1.0 was released in November, and are at 1.10 now, making it really an alpha product by comparison. There may be buffer overflow issues and other security risks still. However, should the product mature nicely, and prove it's stability, it may be worth my time to port a .NET QuickLZ as well in a month or two.
 
What I do want to point out here, is that all around LZO seems about 33% faster than LZF, and has slightly better compression rate too. So, let this be factual information and not some guess by normalizing memcpy's on different hardware.
 
Now, this doesn't accurately reflect our C# ports, but this does show the basic potential of the algorithm and tells me you made an uneducated guess about which is "the winner" algorithm.
 
So ultimately, I stand my ground behind why I ported minilzo. It's faster. Period.
GeneralRe: Another such library... Pinmemberomaurice23-Dec-06 5:32 
GeneralRe: Another such library... PinmemberAstaelan23-Dec-06 5:46 
GeneralRe: Another such library... PinmemberWorldNomad23-Dec-06 4:32 
GeneralRe: Another such library... Pinmemberomaurice23-Dec-06 5:00 
Generalwhy not adding the original size to the header of the buffer PinmemberUnruled Boy5-Nov-06 16:19 
GeneralRe: why not adding the original size to the header of the buffer PinmemberAstaelan6-Nov-06 18:01 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.1411023.1 | Last Updated 22 Dec 2006
Article Copyright 2006 by Astaelan
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid