Hashing algorithm md5

Question

4.00/5 (3 votes)

See more:

can any one tell me the fastest way to produce md5 hashes, when a filepath is provided to the method. At present I am using an algorithm which is returning hashed string of 32 bytes. What I think that, if I could get such an algorithm, which can provide hashed string of 16 bytes then the process could be much faster. I am a bit new to it, so may be wrong. Please provide any feedback. This is what I am using.(Got this in an article in Code Project itself).

C#

System.Security.Cryptography.MD5CryptoServiceProvider oMD5Hasher =
                     new System.Security.Cryptography.MD5CryptoServiceProvider();
           try
           {
               oFileStream = GetFileStream(pathName);
               arrbytHashValue = oMD5Hasher.ComputeHash(oFileStream);
               oFileStream.Close();
               strHashData = System.BitConverter.ToString(arrbytHashValue);
               strHashData = strHashData.Replace("-", "");
           }
           catch (System.Exception ex)
           {
           }

Thanks and Regards
Anurag

Posted 18-Nov-10 0:33am

@nuraGGupta@

Updated 18-Nov-10 0:35am

v3

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Manfred Rudolf Bihy · Accepted Answer · 2010-11-18T00:59:00

Solution 1

Hello Anurag,

a MD5 hash function always yields 128 bit which makes for 16 bytes. 16 bytes in hex are 32 characters. A MD5 hash in a string representation cannot be shorter than 32 characters when expressed in hex. Anyhow, the lengthy duration of the caculation in your case is probably due to the file size. The bigger the file the longer it takes to calculate the hash (Did you spot where computeHash is being fed the file input stream?). How big is that file your hashing and how long does it take?

There are other Hash functions that may be more performant but if you can use them or not depends on what your use case is.

Cheers

Manfred

Posted 18-Nov-10 0:59am

Manfred Rudolf Bihy

Comments

@nuraGGupta@ 18-Nov-10 7:23am

Thanks Manfred, actually I was considering the bytes in c-sharp only, where 32 bytes means 32 characters. And yes, you are correct, the bigger the file size, the longer time it takes. Anyways, thanks for the response.

@nuraGGupta@ 18-Nov-10 7:53am

I have to do hashing on the hard disk, or tape drive, or usb drive or any storage device. At a time, my program need to do hashing of 200 gb of data on an average, and it is doing at the rate of 10 gb per hour. I want to make it a bit more faster.

Manfred Rudolf Bihy 18-Nov-10 7:56am

What kind of files are these that get this big and what is your intention of building a MD5 hash on them?

@nuraGGupta@ 18-Nov-10 8:05am

these can be any files, suppose video files etc, and intention is unknown,as I have never asked the client for whom I am working on this. It may be some type of recovery of data thing

Manfred Rudolf Bihy 18-Nov-10 8:05am

But there is always one MD5 hash for each file? What do you do after the MD5 hash is generated?

@nuraGGupta@ 18-Nov-10 8:06am

I need to clearify that this is not for one file, but for all the flies and folders on a drive, upto nth level hierarchy.

@nuraGGupta@ 18-Nov-10 8:08am

I just store it in db for any further usage. If there is always one md5 hash for each file, then how can I fetch it. Can you please elaborate on this?

@nuraGGupta@ 18-Nov-10 8:11am

In the same way, I would need SHA1 hash also for each file. Both are to be fetched and stored in DB.

Manfred Rudolf Bihy 18-Nov-10 8:13am

My question meant if your were building an MD5 hash for each file. Which you already answered with yes.
Program finds files, builds MD5 hash for each file and stores it in a DB but you don't really know what the customer will do with that information. If MD5 hash was a requirement by the customer then I guess your stuck with what you've got right now.

Manfred Rudolf Bihy 18-Nov-10 8:14am

Ok now it's also SHA1 hash. This getting absurd. Sorry. Ask your customer what they are trying to accomplish.

@nuraGGupta@ 18-Nov-10 8:16am

No,My client is technical person, he knows all this, and he is pretty happy with the way the work is going on. He just want me to make any effort if this could be speed-ed up. SO, that's why I was looking for some another method which could speed up the process. It will be OK for him if I present any new idea to him like every file has an MD5 hash and we could somehow fetch it.

Manfred Rudolf Bihy 18-Nov-10 8:21am

For what reason does he need two hashes. If this is a requirement, there has to be a reason for it. And NO, you can't just fetch the MD5 for a file, it has to calculated. If the file is not modified, the MD5 hash will remain the same. After a modification is done on the files content the MD5 hash will change. You might gain a little speed if you used C++ instead of C#.

@nuraGGupta@ 18-Nov-10 8:29am

This I know that if the file content is not changed, then MD5 hash will always be the same and this can be exact reason for fetching it. He will keep the current MD5 and later on, he can check the current MD5 with the latest MD5 of that file to check that whether any changes have been made in the file or not.

Manfred Rudolf Bihy 18-Nov-10 8:59am

That seems like a plausible use case for a MD5 hash. Already know what the SHA1 hash is supposed to be used for? How often is it that the files will be checked? And if it's just checking for modifications why not use the date fields of the file? Does the customer want to make sure the files are not tampered with?

Rasepretrep2 18-Nov-10 11:24am

i think the time-consuming bottleneck in this case for either MD5 or SHA1 is the fact that you have to read the whole amount of data to generate the checksum. The MD5 algorithm ist really pretty fast (mostly only shifts and adds) , the IO time is the limiting factor here.
Unfortunately there is no way around that. You could try to calculate an MD5 sum on only portions of the data to speed up the process for each file but then there might be changes just in the skipped data.

@nuraGGupta@ 18-Nov-10 11:54am

sorry Manfred for replying late, actually that was time for me to pull out off the office and it takes me 3 hrs to drive back home.I dont know that how often files will be checked. I dont know the use of SHA1, and yes the customer may want to make sure that files are not tampered with, thats why he is asking for MD5 hash. In any case these two hashes are anyhow needed.