Click here to Skip to main content
13,795,745 members
Click here to Skip to main content
Add your own
alternative version

Tagged as

Stats

19.8K views
273 downloads
6 bookmarked
Posted 6 Jan 2015
Licenced CPOL

DIFF: Your IRdcLibrary API for Remote Differential Compression

, 21 Mar 2016
Rate this:
Please Sign up or sign in to vote.
A quick library for differential memory/file processing.

 

Introduction

It seems to me that you 've always wanted to create some backup/restore feature with incremental processing of each files. Here is a small library that demonstrates the power of IRdcLibrary COM API, ready to be used.

Background

COM, Win32 and C++, as usual. I don't tolerate more beginners than me here :P.

Using the code

The DIFF library can read/write from files and memory. For reading data, you use Microsoft's IRdcFileReader interface. For your convienience, I provide 2 helper classes in the code that can provide the IRdcFileReader interface:

MemoryRdcFileReader, which can read from memory. The memory must persist while this interface exists (i.e. the interface does not copy the memory). 

Constructor: 

MemoryRdcFileReader(const char* dd,unsigned long long size);

FileRdcFileReader, which can read from an existing HANDLE. The handle must have been opened with GENERIC_READ and must be seekable.

Constructor: 

FileRdcFileReader(HANDLE hF);

 

For writing data, my functions require an instance of a DIFFWRITER class. I also provide a FileDiffWriter (which can provide a DIFFWRITER from a HANDLE which must have been opened with GENERIC_WRITE) and a MemoryDiffWriter which writes to a vector<char>. These helper classes provide the member function GetReader(IRdcFileReader**that returns a MemoryRdcFileReader or a FileRdcFileReader. 

 

Incremental backup

Let's say you have a file named 1.txt in c:\test , and you have backed it in c:\test2. Now let's say that you have modified 1.txt in the original c:\test directory, so c:\test\1.txt is newer. Your incremental backup steps now are (error checking is removed for simplicity):

// Prereqs: file readers for both files. 
// Say that hX1 and hX2 are handles to these two files
CComPtr<IRdcFileReader> fil1;
CComPtr<IRdcFileReader> fil2;
fil1.Attach(new DIFF::FileRdcFileReader(hX1));
fil2.Attach(new DIFF::FileRdcFileReader(hX2));

// Instantiate the library
DIFF::DIFF d;

// Create a signature for both files
// Say that hS1 and hS2 are handles to new files that the signature is to be stored
DIFF::FileDiffWriter sig1(hS1);
DIFF::FileDiffWriter sig2(hS2);
d.GenerateSignature(fil1,sig1);
d.GenerateSignature(fil2,sig1);

// Close handles

This will save in the files the RDC signature for both old 1.txt and new 1.txt. 

Your next step is now to generate a diff between 2 files. My library provides you with 2 options: Either you create a self-contained diff, or a standalone-diff. 

self-contained diff allows you to reconstruct the updated file from the source file offline.
standalone diff is a lot smaller, but it needs parts of the remote file. So if you are to use standalone diffs, you have to implement a IRdcFileReader yourself that downloads the requested portions of the updated file from a server. Since this needs a server, we will focus on self-contained diffs here.

Now, you want to create the diff between the old and the new file.

 CComPtr<IRdcFileReader> xsig1;
 CComPtr<IRdcFileReader> xsig2;
 sig1.GetReader(&xsig1);
 sig2.GetReader(&xsig2);
 DIFF::MemoryDiffWriter diff1_2_sc;
 d.GenerateDiff(xsig1,xsig2,r2,diff1_2_sc));

This generates a self-contained diff (if you pass NULL instead of r2 in the 3rd parameter it generates a standalone diff). Instead of uploading now your new file c:\test\1.txt, you can upload this diff file which is smaller. How smaller it is depends on how much the files differ.

 

Diff Restore

Now say that you have the original (master) backup of 1.txt and you want to restore the newer 1.txt. You need the original 1.txt and a self-contained diff file:

 

// r1 is the reader for the original file.
// xdiff2 is reader for the the diff.
// Say hX3 is a handle to the file we want to reconstruct
DIFF::FileDiffWriter reco1(hX3);
d.Reconstruct(r1,xdiff1,0,reco1);

And that's all. Your newer 1.txt file has been reconstructed by the original file and the diff. If the diff is a standalone diff, you would have to pass a reader to DIFF::Reconstruct's 3rd parameter which would be asked to provide portions of the file.

 

Points of Interest

  • The API requires Vista or newer. 
  • The library runs ONLY in multithreaded COM environement  (CoInitializeEx(0, COINIT_MULTITHREADED);)
  • The constructor throws an exception if the COM object cannot be created.
  • There's a similarity API I must investigate more.
  • The library uses smart pointers, both COM ATL and C++. 

 

Have fun!

 

History

21 - MAR - 2016: Implemented as .h file, with some bugfixes.
21 - OCT - 2015: A few bugfixes and some experimental multiple-sig stuff. 
06 - JAN - 2015: First Release (and Happy NY:))

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Michael Chourdakis
Engineer
Greece Greece
I'm working in C++, PHP , Java, Windows, iOS and Android.

I 've a PhD in Digital Signal Processing and Artificial Intelligence and I specialize in Pro Audio and AI applications.

My home page: http://www.michaelchourdakis.com

You may also be interested in...

Comments and Discussions

 
QuestionOther source of information Pin
hi reddit9-May-16 0:09
professionalhi reddit9-May-16 0:09 
AnswerRe: Other source of information Pin
Michael Chourdakis9-May-16 3:07
mvpMichael Chourdakis9-May-16 3:07 
GeneralRe: Other source of information Pin
hi reddit11-May-16 11:43
professionalhi reddit11-May-16 11:43 
QuestionAny update? Pin
WHAK.com7-Mar-15 0:06
professionalWHAK.com7-Mar-15 0:06 
QuestionUpdated Microsoft Docs on RDC Pin
WHAK.com12-Jan-15 12:41
professionalWHAK.com12-Jan-15 12:41 
AnswerRe: Updated Microsoft Docs on RDC Pin
Michael Chourdakis12-Jan-15 13:04
mvpMichael Chourdakis12-Jan-15 13:04 
QuestionInteresting material Pin
WHAK.com8-Jan-15 8:31
professionalWHAK.com8-Jan-15 8:31 
AnswerRe: Interesting material Pin
Michael Chourdakis8-Jan-15 8:35
mvpMichael Chourdakis8-Jan-15 8:35 
QuestionRDC enabled copy tool Pin
WHAK.com7-Jan-15 9:08
professionalWHAK.com7-Jan-15 9:08 
AnswerRe: RDC enabled copy tool Pin
Michael Chourdakis7-Jan-15 10:14
mvpMichael Chourdakis7-Jan-15 10:14 
AnswerRe: RDC enabled copy tool Pin
Michael Chourdakis21-Mar-16 10:43
mvpMichael Chourdakis21-Mar-16 10:43 
Bugdiff.zip source code Pin
Lucian Todor6-Jan-15 21:51
memberLucian Todor6-Jan-15 21:51 
GeneralRe: diff.zip source code Pin
Michael Chourdakis6-Jan-15 22:25
mvpMichael Chourdakis6-Jan-15 22:25 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Cookies | Terms of Use | Mobile
Web06 | 2.8.181207.3 | Last Updated 21 Mar 2016
Article Copyright 2015 by Michael Chourdakis
Everything else Copyright © CodeProject, 1999-2018
Layout: fixed | fluid