Need an algorythm to compress a byte array

Question

3.50/5 (2 votes)

See more:

Hello guys
I need to compress data in a byte array and store in database. speed is favored.
I want to compress better than very simple algorythms such as counting repeated bytes.
I want simpler algorythms than lempel-ziv, to compress small count of bytes, say 500 bytes.
Specifically I Want this: 1- compress small arrays, 2-speed, 3-small code, 4-compresion ratio to worth compressing.
Any guidance or article appreciated.
If operating system provides compresion and decompresion, It will be better.
Thanks in advance
mr.abzadeh

Posted 10-Apr-12 14:15pm

mr.abzadeh

Updated 10-Apr-12 21:05pm

v5

Add a Solution

Comments

Sergey Alexandrovich Kryukov 10-Apr-12 21:58pm

"Better than... repeated byte" is understandable, if you mean "run-length encoding", but why "simpler then Zempel-Ziv"? What do you want, simplicity (but who knows what is that?) or speed?

How can you ask what OS provides is you did not share your platform with us? No, please add it in the tags.

--SA

mr.abzadeh 11-Apr-12 3:01am

Thanks for your interest.
You are right, I should specify os, which is windows xp sp2 and above.
And I do not have good familiaity with compresion algorythms,
by counting repeated bytes I mean very simple algorythms, not run-length encoding, what I want is: 1-compress small arrays, 2-speed, 3-small code

enhzflep 11-Apr-12 1:53am

Lzw can be implemented easily in a few hours or less, adding about 10kb to the exe for both compression/decompression. There's also zlib. This will add ~70kb to your exe.

mr.abzadeh 11-Apr-12 3:10am

Thanks, lzw may be acceptable as it addes 10 kb, Does it compress small arrays? Can you offer me a reliable link to download lzw? Do you know another algorythm with smaller code size?

Sergey Alexandrovich Kryukov 12-Apr-12 12:53pm

You really need to try. I would also try Huffman maybe...
--SA

3 solutions

Solution 1

how about huffman encoding http://hbfs.wordpress.com/2011/05/17/huffman-codes/[^]. i have done it.but not sure about the speed of it. don't know whether this helps you.

Posted 10-Apr-12 19:08pm

jk chan

Comments

mr.abzadeh 11-Apr-12 3:13am

Thanks for your guidance.
I need lighter argorythm than huffman encoding, I want this: 1- compress small byte counts in memory, 2- good speed, 3- small code.
Does windows provide compression library?

jk chan 11-Apr-12 3:32am

what is the minimum speed you want to achieve for your chunk(500bytes) ?

mr.abzadeh 11-Apr-12 3:53am

I do not have an absolute minimum size.
I want to log Insert, update, deletes on a database
I want to collect the row's previous and next value, compress them and store in a log table. I can store small record, and compress larger ones, but since there are tens of milions of rows, speed and byte size as small as 100 bytes and even less is desirable,
Thanks

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

CPallini · Accepted Answer · 2012-04-10T21:37:00

Solution 2

The best alogorithm depends on the nature of your data. If they are, for instance, random values then you'll better don't even try to compress them.
If you need a guidance on a compression method then I suggest you the Salomon's book: "Data Compression: The Complete Reference"[^].

On the other hand, if you are searching for a ready-to-use library, then you have several options (they depend also on your favourite development platform, for instance, .NET Framework provides the System.IO.Compression Namespace[^]) and Google is you friend friend[^].

Posted 10-Apr-12 21:37pm

CPallini

Comments

mr.abzadeh 11-Apr-12 3:57am

Thanks for your answer.
I am searching in google know.
Since I code in native cpp, I can not use System.IO.Compression Namespace.
Is there any library for compresion in native cpp?

CPallini 11-Apr-12 4:31am

There are many: Google for, Google for and Google for again!

Espen Harlinn 11-Apr-12 7:41am

5'ed!

VJ Reddy 7-May-12 10:34am

Good answer. 5!

CPallini 7-May-12 12:03pm

Thank you.

Espen Harlinn · Accepted Answer · 2012-04-11T01:45:00

Solution 3

I often use #ziplib [^] because of the wide range of supported formats - including encryption support.

[Update]
When I need something like this in C++ I look to the LZMA SDK[^]

Best regards
Espen Harlinn

Posted 11-Apr-12 1:45am

Espen Harlinn

Updated 11-Apr-12 1:47am

v2

Comments

mr.abzadeh 11-Apr-12 10:33am

Thanks for your reply.
These are good, but they compress large amount of data efficiently. I need an algorythm to compress small amount of data, from 100 bytes to 4k.

Espen Harlinn 11-Apr-12 10:39am

You can always try something simple like:
http://en.wikipedia.org/wiki/Run-length_encoding
or something like http://en.wikipedia.org/wiki/LZ77_and_LZ78 with a predefined dictionary generated from sample data.

The first one is pretty easy to implement while the second one requires something more of an effort.

VJ Reddy 7-May-12 10:33am

Good answer. 5!

Espen Harlinn 7-May-12 10:37am

Thank you, VJ :-D