Rar format and algorithm is proprietary. But you can have unrar code for free (http://www.rarlab.com/rar_add.htm
]), so you might take you chance to get an idea.
Why do you stick to rar, there are other other formats, where you can make your own implementation without restriction.
But after all, why bothering with implementing something that you already have?
Just a note: compression algorithms are really complicated. Here you can read about the probably simplest after RLE: the Huffman code