Click here to Skip to main content
Click here to Skip to main content

CRC32: Generating a checksum for a file

By , 17 Dec 2001
 

Sample Image - crc32.gif

Introduction

Recently I wrote a program in which I wanted to generate a CRC for a given file. I did some checking on the web for sample CRC code, but found very few algorithms to help me. So I decided to learn more about CRCs and write my own code. This article describes what a CRC is, how to generate them, what they can be used for, and lastly source code showing how it's done.

What is a CRC

CRC is an acronym for Cyclic Redundancy Checksum or Cyclic Redundancy Check (depending on who you ask). A CRC is a "digital signature" representing data. The most common CRC is CRC32, in which the "digital signature" is a 32-bit number. The "data" that is being CRC'ed can be any data of any length; from a file, to a string, or even a block of memory. As long as the data can be represented as a series of bytes, it can be CRC'ed. There is no single CRC algorithm, there can be as many algorithms as there are programmers. The ideal CRC algorithm has several characteristics about it. First, if you CRC the same data more than once, you must get the same CRC every time. Secondly, if you CRC two different pieces of data you should get two very different CRC values. If you CRC the same data twice, you get the same digital signature. But if you CRC data that differs (even by a single byte) then you should get two very different digital signatures. With a 32-bit CRC there are over 4 billion possible CRC values. To be exact that's 232 or 4,294,967,296. With that many CRC values it's not difficult for every piece of data being CRC'ed to get a unique CRC value. However, it is possible for spurious hits to happen. In other words two completely different pieces of data can have the same CRC. This is rare, but not so rare that it won't happen.

Why use CRCs

Most of the time CRCs are used to compare data as an integrity check. Suppose there are two files that need to be compared to determine if they are identical. The first file is on Machine A and the other file is on Machine B. Each file is a rather large file (say 500 MB), and there is no network connection between the two machines. How do you compare the two files? The answer is CRC. You CRC each of the two files, which gives you two 32-bit numbers. You then compare those 32-bit numbers to see if they are identical. If the CRC values are different, then you can be 100% guaranteed that the files are not the same. If the CRC values are the same, then you can be 99% sure that the files are the same. Remember, because spurious hits can happen you cannot be positive that the two files are identical. The only way to be positive they are the same is to break down and do a comparison one byte at a time. But CRCs offer a quick way to be reasonably certain that two files are identical.

How to generate CRCs

Generating CRCs is a lot like cryptography in that involves a lot of mathematical theories. Since I don't fully understand it myself, I won't go into a lot of those details here. Instead I'll focus on how to program a CRC algorithm. Once you know how the algorithm works you should be able to write a CRC algorithm in any language on any platform. The first part of generating CRCs is the CRC lookup table. In CRC32 this is a table of 256 specific CRC numbers. These numbers are generated by a polynomial (the computation of these numbers and what polynomial to use are part of that math stuff I'm avoiding). The next part is a CRC lookup function. This function takes two things, a single byte of data to be CRC'ed and the current CRC value. It does a lookup in the CRC table according to the byte provided, and then does some math to apply that lookup value to the given CRC value resulting in a new CRC value. The last piece needed is the actual data that is to be CRC'ed. The CRC algorithm reads the first byte of data and calls the CRC lookup function which returns the CRC value for that single byte. It then calls the CRC lookup function with the next byte of data and passes the previous CRC value. After the second call, the CRC value represents the CRC of the first two bytes. You continuously call the CRC lookup function until all the bytes of the data have been processed. The resulting value is the CRC for the whole data.

Code Details

In this sample program I wanted to show that there are many different ways of generating CRCs. There are over 8 different CRC functions, all based on the above steps for generating CRCs. Each function differs slightly in it's intended use or optimization. There are four main CRC functions, each described below. There are also two separate CRC classes, but more on that later. And lastly there are a few helper functions that CRC strings.

C++ Streams: The first function represents the simplest CRC function. The file is opened using the C++ stream classes (ifstream). This function uses nothing but standard C++ calls, so this function should compile and run using any C++ compiler on any OS.

Win32 I/O: This function is more optimized in that it uses the Win32 API for file I/O; CreateFile, and ReadFile. This will speed up the processing, but by using the Win32 API the code is no longer platform independent.

Filemaps: This function uses memory mapped files to process the file. Filemaps can be used to greatly increase the speed with which files are accessed. They allow the contents of a file to be accessed as if it were in memory. No longer does the programmer need to call ReadFile and WriteFile.

Assembly: The final CRC function is one that is optimized using Intel Assembly. By hand writing the assembly code the algorithm can be optimized for speed, although at the sacrifice of being easy to read and understand.

Those are the four main CRC functions. But there are actually two versions of each function. There are two classes, CCrc32Dynamic and CCrc32Static, each of which have the above four functions for a total of eight. The only difference between the static and dynamic classes is the CRC table. With the static class the CRC table and all the functions in the class are static. The trade off is simple. The static class is simpler to use, but the dynamic class uses memory more efficiently because the CRC table (1K in size) is only allocated when needed.

// Using the static class is as easy as one line of code
dwErrorCode = CCrc32Static::FileCrc32Assembly(m_strFilename, dwCrc32);

// Whereas there is more involved when using the dynamic class
CCrc32Dynamic *pobCrc32Dynamic = new CCrc32Dynamic;
pobCrc32Dynamic->Init();
dwErrorCode = pobCrc32Dynamic->FileCrc32Assembly(m_strFilename, dwCrc32);
pobCrc32Dynamic->Free();
delete pobCrc32Dynamic;

Whenever you calculate a CRC you need to take into account the speed of the algorithm. Generating CRCs for files is both a CPU and a disk intensive task. Here is a table showing the time it took to CRC three different files. The columns are the different file sizes, the rows are the different CRC functions, and the table entries are in seconds. The system these numbers were captured on is a dual Pentium III at 1 GHz with a 10,000 RPM SCSI Ultra160 hard drive.

44 Kb 34 Mb 5 Gb
C++ Streams 0.0013 0.80 125
Win32 I/O 0.0009 0.60 85
Filemaps 0.0010 0.60 87
Assembly 0.0006 0.35 49

As expected the C++ streams is the slowest function followed by the Win32 I/O. However, I was very surprised to see the filemaps were not faster than the Win32 I/O, in fact they are slower. After I thought about it some, I realized memory mapped files are designed to provide fast random access to files. But when you CRC you access the file sequentially. Thus filemaps are not faster, and the extra overhead of creating the "views" of the file are why it's slower. Filemaps do have one advantage that none of the other functions have. Memory mapped files are guaranteed to be able to access files up to the maximum file size in NT which is 264 or 18 exabytes. Although the Win32 I/O may handle files of this size, none of the documentation confirms this. [Note: The largest file I have CRC'ed is 40 GB, which all eight functions successfully CRC'ed, but took over 10 minutes each.]

If anyone who reads this article knows a way to improve the speed even more, please post the code or email me. Especially if you know of a speed improvement for the assembly code. I will bet there are further optimizations that can be made to the assembly code. After all I don't know Intel Assembly very well, therefore I'm sure there are optimizations I don't know about.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Brian Friesen
Web Developer
United States United States
Member
No Biography provided

Sign Up to vote   Poor Excellent
Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

 
You must Sign In to use this message board.
Search this forum  
    Spacing  Noise  Layout  Per page   
Questionhow to compare folders by using CRCmembergreenjade80023 Aug '04 - 11:28 
I am trying to find a C++ source code to compare folders by using CRC. If you know where, please post here. Thanks in advance!

GeneralCRC used with serializationmembertime_error11 Dec '03 - 20:58 
Hi.
I am pretty familiar with the concepts of CRC.
 
I wonder, how do I use it when serializing data to file (I wan´t to be able to validate data in file afterwards when reading it back in to my application).
 
In the serialisation process data is "streamed" to file. First after the file is generated, it would make sence to to the actual CRC check of the file. Another issue is if this does destroy the possibility to use serialization when reading the file back (at leasy some execptions will be thrown).
 
So my question is:
Is there a standard method to use CRC when serializing, so data in the file can be validated afterwards (fx. before reading the file back).
 

 
(ORG. FILE)               -->      (ADD CRC)                  -->   (VALIDATE FILE LATER ON)
|-----------------|               |-----------------|            |-----------------|
|-----------------|               |-----------------|            |-----------------|   CRC == FILECHECK ???
|-----------------|               |-----------------|            |-----------------|
|-----------------|               |-----------------|            |-----------------|
                                                   |-CRC-|
                                                         ^
                                                         |-- This CRC is based on the org. file
 
/Jonas

QuestionCRC32's Question?memberdavid75329 Aug '03 - 23:35 
I check several books that mentioned about CRC principle.
The basic theory is like as below description in coding.
====================================
for(index = 0; index <= x; index++)
{
crc = crc & 0x80000000 ? (crc << 1) ^ 0x04C11DB7 : crc << 1;
}
=====================
But, In the most of CRC32's implementation.
It used a look up table.
And both results are difference.
Why??
I am curious in CRC32's implementation method and theory.
Who is the inventor?
Why the result doesn't correspond to the basic theory?
Where can I get any related paper?
Your kind reply will be highly appreciated.
 

 

 

QuestionGetting consistent CRCs??memberJohnDoeNumber230 Jul '03 - 7:53 
First, let me say this article was a big help. It had just the right amount of info for a person who knows very little about CRCs and simply needs the basics. However, I have a question/problem. I am working on a project that I need to have the same CRC each time it is compiled. I compile my project under VisualStudio 6.0 on WinNT and generate a CRC on the executable I just created. Then, I rebuild the entire project (using the exact same files and compiler) and do a CRC on the new executable. The two CRCs are different, even though the exes are the same. I have come to the conclusion that there is some sort of timestamp telling when the file was created. Does anyone know of an algorithm that compensates for the timestamp and ignores it? Does that make sense? If not, ask and I can clarify.
Thanks
AnswerRe: Getting consistent CRCs??memberBrian Friesen30 Jul '03 - 9:47 
How do you know the two files are the same? Did you do a bit-wise comparision (e.g. "fc /b"). Even though the source code didn't change, everytime a project is compiled, Visual Studio does not generate the exact same binary. You'd think it would, but it doesn't. I'm not sure why (maybe someone else knows), there might be an internally compiled timestamp which would explain the difference. But my program does not use the file system times. That's most likely why you're getting different CRCs, because the files are indeed different.
 
That having been said, I think there is a flaw in the way you're trying to use CRCs. Are you saying if you were to make actual changes to the code you still want to get back the same CRC? To try and force one file to have to same CRC as another file is a very difficult task, nearly impossible. The whole idea behind CRCs is the slightest difference in a file results in a significant change in the CRC value.

GeneralRe: Getting consistent CRCs??memberJohnDoeNumber230 Jul '03 - 10:25 
I opened the files in a Hex Editor, looking for differences, and noticed that there were 3 locations that a pair of entries were different. The locations of the differences are always the same, so I'm fairly sure that there is some sort of timestamp applied to the file. However, I haven't found if there is a way to control this from Visual Studio and was hoping someone might know where I could find information about this. The reason I need the CRC is to prove to certain authorities that I can recompile exactly the same executable now and six months down the road, with the exact same files (We're under verions control management).
GeneralRe: Getting consistent CRCs??memberWanderley Caloni25 May '05 - 8:01 
Take a look in the appendix of the PE Specification http://www.cs.ucsb.edu/~nomed/docs/pecoff.html#_Toc83091247.
 
--------------------
Wanderley Caloni Jr.
 
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GCS/IT d++(--) s+: a- C++ L E- W++ K- w++ PS
PE Y+ PGP+ t+ X+ R tv b+ DI++ D+ G e h- r y+
------END GEEK CODE BLOCK------
AnswerRe: Getting consistent CRCs??memberHPSI3 Aug '03 - 0:05 
You are correct in your conclusion. The PE file format contains multiple timestamps. The resources are also timestamped. To "prove" that two exe's are identical, you will have to parse the PE file format, crc'ing only the code sections.
 
Have you considered crc'ing the source code instead?
 

HPS HwndSpy
- GUI developer's aid to visually
locate and inspect windows. For the month of August
only, use coupon code CP-81239 for 30% off.

GeneralRe: Getting consistent CRCs??memberJohn Simmons / outlaw programmer31 Oct '03 - 2:59 
Does your hwndspy app work on full-screen direct3d apps?

 
------- signature starts
 
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001
 
"You won't like me when I'm angry..." - Dr. Bruce Banner
 
Please review the Legal Disclaimer in my bio.
 
------- signature ends
AnswerRe: Getting consistent CRCs??memberDoug Gale9 Aug '04 - 4:38 
Yes, it is in the IMAGE_FILE_HEADER.
 
Look at your WinNT.h file and file IMAGE_FILE_HEADER. Look at the TimeDateStamp field.
 
The timestamp is completely ignored by windows and it is safe to null it out.
 
To find the file header in your exe, read the IMAGE_DOS_HEADER located at the beginning of the file.
 
The e_lfanew tells you the file offset of the IMAGE_FILE_HEADER. Seek there and read the IMAGE_FILE_HEADER. Zero out the TimeDateStamp field, seek back to e_lfanew, and write the IMAGE_FILE_HEADER.
 
Totally safe. Smile | :)

GeneralCRC32 to GUIDsussAnonymous24 Jun '03 - 8:27 
"{03544472-641E-4B7B-8AEF-214C8DB9037E}",
"{FB4A21E3-48F0-46FF-98BB-410914CE9C6A}"
 
And many other pairs of very different GUID strings has the same CRC32 value
GeneralRe: CRC32 to GUIDmemberBrian Friesen24 Jun '03 - 12:32 
As I said in my article it's possible for spurious hits to happen. CRCs do not guarantee uniqueness, but are a pretty good indication of it.
GeneralWon't compile in VS 2003 due to iostream changesmemberJ Cardinal3 May '03 - 7:08 
The old iostream library has now been removed from VS 2003 (it was flagged as deprecated in vs 2002).
 
This means that the FileCrc32Streams function will not compile, however if the FileCrc32Streams function and header definition are commented out, the rest will compile fine so no problem if you don't use FileCrc32Streams.
 
I really like this article and found it very useful, I encourage you to take a look at re-writing FileCrc32Streams to use the new Standard C++ iostream library.
 
See here for the changes:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vccore/html/_core_differences_in_iostream_implementation.asp[^]
GeneralRe: Won't compile in VS 2003 due to iostream changesmemberBrian Friesen4 May '03 - 18:10 
The code I provided was never meant to be used as an SDK if you will. The article is meant to teach people about CRCs and give them code showing them how it's done. It is up to the other developers to use my code as a guide, or copy and paste the necessary sections into their projects. Thus I doubt I'll take the time to update the code to the new compiler.
GeneralError with streams functions in VC7memberJean-Michel Reghem23 Jan '03 - 22:50 
As #include is deprecated in VC7, the code compile, but if you use it in a MFC application, it couldn't link because you have a "delete" function redefinition in the .lib containaing file stream functions ...
The solution to use it in VC7 with MFC is to comment
//#include
and to comment FileCrc32Streams function in both .cpp and .h files...
GeneralRe: Error with streams functions in VC7memberJean-Michel Reghem23 Jan '03 - 22:52 
sorry ... i post the message without check the box "Display this message as-is (no HTML)" ... and thus, it is not complete ... Here is the message:
 
As #include <fstream.h> is deprecated in VC7, the code compile, but if you use it in a MFC application, it couldn't link because you have a "delete" function redefinition in the .lib containaing file stream functions ...
The solution to use it in VC7 with MFC is to comment
//#include <fstream.h>
and to comment FileCrc32Streams function in both .cpp and .h files...
GeneralRe: Better solutionmemberAldamo22 Jul '03 - 0:24 
Another solution of this problem (in case you don't know):
 
In the new Standard C++ iostream library
"open" functions do not take a third parameter (the protection parameter),
and some elements of the old iostream library are not elements
of the new iostream library, among them is "nocreate".
 
First of all, you must change the

#include <fstream.h>

to:
#include <fstream>

Then add the string:

using namespace std;

You should also make some changes in the function "open" to
comply with the new stream library implementation of VC7,
and it will just work fine.
 
Here is the original function in the sample code
(files Crc32Static.cpp and Crc32Dynamic.cpp), which caused compile errors:

file.open(szFilename, ios::in | ios::nocreate | ios::binary, filebuf::sh_read);

and here is the "modernized" one:

file.open(szFilename, ios::in | ios::binary);

Now it should compile without errors.
Generalvery nice code :) but...sussAnonymous29 Oct '02 - 14:34 
are you planning to port this to c#?
GeneralRe: very nice code :) but...memberBrian Friesen30 Oct '02 - 13:52 
Sorry, but I have no plans to convert the code to VB, Java, J#, nor C#. I've provided the code and the techniques, I'll let someone else do the conversion.
GeneralAssembly optimizationssussAnonymous18 Oct '02 - 15:15 
You might find this interesting for the assembly optimizations
 
Asm optimized crc32
 
//Ante, ante.c@runbox.com
GeneralRe: Assembly optimizationsmemberjvene15 May '03 - 0:02 
Hey. Nice stuff, found it useful.
 
I decided to try my hand at the assembler, poked around the link listed by Ante above. My result is below. In my case, I "init" the CRC in a separate call, this routine works a series of buffers as they're received until I reach the end. My table is a global variable, not a static member of the object, but otherwise it's like yours. I noted that on the site Ante links to, they claimed 158 Mbytes per second on a T'bird 1.4 Ghz. On my 1.5Ghz AMD XP I get 275 Mbytes per second, not sure if the T'bird should be that far behind or not. For me, the punch came from loading a quad word at a time, not a byte at a time.
 
__asm
{
mov eax, this // Load 'this'
mov ecx, [eax]this.CurrentCRC // Load running CRC from 'this'
 
mov edi, offset Crc32Table // Load the CRC32 table
 
mov esi, buf // Load buffer
 
xor ebx, ebx // zero ebx, used to process bytes, forming
// index into crc table
 
mov eax, len // get length
mov edx, eax
and eax, 3 // calc remainder after division by 4
push eax // preserve the remainder for later
 
shr edx, 2 // div by 4, calculating total quadwords
jz crc32tail // if zero, prepare for a tiny bit of work
 

crc32loop:
 
mov eax, [esi] // grab a quadword from buf

mov bl, al // form index entry, starting with a byte from buf
// part one of 4 in the quadword
xor bl, cl // xor against current CRC
shr ecx, 8 // shift CRC
 
xor ecx, [edi + ebx * 4] // xor CRC with the table's entry
 
// part two of 4 in the quadword
mov bl, ah // grab another byte of buf
 
xor bl, cl // xor against current CRC
shr ecx, 8 // shift CRC
 
xor ecx, [edi + ebx * 4] // xor CRC with table's entry
 
shr eax, 16 // shift the buf data two bytes down
// part three of 4 in the quadword
mov bl, al // grab another byte of buf
 
xor bl, cl // xor against current CRC
shr ecx, 8 // shift CRC
 
xor ecx, [edi + ebx * 4] // xor CRC with table's entry
 
// part four of 4 in the quadword
mov bl, ah // grab another byte of buf
 
xor bl, cl // xor against current CRC
shr ecx, 8 // shift CRC
 
xor ecx, [edi + ebx * 4] // xor CRC with table's entry
 

add esi, 4 // Advance the source pointer one quadword
dec edx // counting quadwords
jnz crc32loop // if more quadwords, loop

crc32tail:
 
pop edx // retreive the remainder of quadwords
cmp edx, 0 // check to see if it's zero
je crc32end
 
crc32tinyloop:
 
mov bl, byte ptr [esi] // grab one byte from buf

xor bl, cl // xor against current crc
shr ecx, 8 // shift crc
 
xor ecx, [edi + ebx * 4] // xor crc with table's entry
inc esi // increment buf pointer
dec edx // dec count
jnz crc32tinyloop // loop if not zero
 
crc32end:
 
mov eax, this
mov [eax]this.CurrentCRC, ecx // write to CurrentCRC

GeneralRe: Assembly optimizationsmemberjvene16 May '03 - 19:35 
Hahaha
 
I've been reading way to much on the Opteron!
 
Sorry guys.
 
My post kept refering to quadwords in that should have read dwords.
 
Same theory, just half the size.
 
Now, on the Opteron - once I get one - this should really zing right through it.

GeneralRe: Assembly optimizationssussV.Dmitrovsky10 Nov '03 - 6:02 
// stdafx.h
#pragma once
#include <iostream>
#include <tchar.h>
#include <windows.h>
----------------------------------------------
<pre>
// crc32.cpp
#include "stdafx.h"
 
#define	POLY	0xEDB88320
#define	SEED	0xFFFFFFFF
 
DWORD CRCTable[256];
int _tmain(int argc, _TCHAR* argv[])
{	DWORD crc32;
	/* The following __asm block needs some comments.
	**
	** One quote from "AMD Athlon™ Processor x86 Code Optimization Guide", 22007H/0—June 2000.
	**
	** "Loops can be unrolled completely, if all of the following conditions are true:
	**	* The loop is in a frequently executed piece of code.
	**	* The loop count is known at compile time.
	**	* The loop body, once unrolled, is less than 100 instructions, which is approximately 400 bytes of code."
	**
	** As far as all conditions are met, the loop for char's bits is completely unrolled.
	**
	** This block seems to be the fastest procedure for CRCTable filling (less then 5200 CPU clocks on AMD Athlon XP™,
	** ~21 clocks for 36 instruction in the loop only!!!).
	*/
	__asm								; CRCTable filling.
	{	xor ebx, ebx					;  bl - CRCTable entry index. Init with 0.
		mov ecx, POLY					; Load polynom into ecx for speed
		CRCTableLoop:					; Head of CRC table calculation main loop.
			mov eax, ebx				; Load index into eax.
 
			xor edx, edx				; edx = 0
			shr eax, 1					; Carry Falg = LSB of eax.
			cmovc edx, ecx				; edx = (eax & 1) ? POLY : 0
			xor eax, edx				; eax = (eax & 1) ? eax ^ POLY : eax
			; The same with other bits
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
			xor edx, edx
			shr eax, 1
			cmovc edx, ecx
			xor eax, edx
 
			mov CRCTable[4*ebx], eax	; Fill the current table entry.
			inc bl							; Move to the next table entry.
		jnz CRCTableLoop					; If index < 256 Then continue CRCTable values
											; Else Table is full.
	}
	printf("CRC32\t\tFileSize\tFileName\n");
	for(int i = 1; i < argc; ++i)
	{	HANDLE hFile = CreateFile(argv[i], GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
		if(hFile == INVALID_HANDLE_VALUE)
		{	printf("Error opening\t%s\n", argv[i]);
			continue;
		}
		HANDLE hMap = CreateFileMapping(hFile, NULL, PAGE_READONLY, 0, 0, NULL);
		if(hMap == NULL)
		{	CloseHandle(hFile);
			printf("Error creating map\t%s\n", argv[i]);
			continue;
		}
		LPVOID pBuffer = MapViewOfFile(hMap, FILE_MAP_READ, 0, 0, 0);
		if(pBuffer == NULL)
		{	CloseHandle(hMap);
			CloseHandle(hFile);
			printf("Error MapViewOfFile\t%s\n", argv[i]);
			continue;
		}
		DWORD dwSize = GetFileSize(hFile, NULL);
		__asm
		{	mov		esi, pBuffer				; esi = buffer pointer.
			mov		ecx, dwSize					; ecs = buffer size.
			mov		eax, SEED					; Init CRC.
			CalcCRC:							; Head of buffer CRC calculation main loop.
				movzx	ebx, byte ptr [esi]		; bl = next char, other ebx bits = 0
				xor		bl, al					;  /Calculate
				shr		eax, 8					; | current
				mov		ebx, CRCTable[4*ebx]	;  \CRC value.
				xor		eax, ebx				; eax = current CRC value.
				inc		esi						; Move to the next char.
				dec		ecx						; Decrement of remaining bytes counter.
			jnz	CalcCRC							; IF counter > 0 THEN continue buffer CRC calculation.
			not		eax							; eax ^= 0xFFFFFFFF
			mov		crc32, eax					; save CRC
		}
		DWORD t3 = GetTickCount();
 
		printf("0x%-08X\t%-8d\t%s\n", crc32, dwSize, argv[i]);
 
		UnmapViewOfFile(pBuffer);
		CloseHandle(hMap);
		CloseHandle(hFile);
	}
	return 0;
}

GeneralRe: Assembly optimizationsmemberDoug Gale9 Aug '04 - 3:24 
Here's a loop I came up with:
 
#pragma warning(push)
#pragma warning(disable : 4035)
unsigned long Crc32_Asm(char *pBuf, unsigned nBytes,
unsigned *pCrcTbl, unsigned nCrc)
{
__asm {
mov edi,pBuf
mov ecx,nBytes
mov eax,nCrc
mov ebx,pCrcTbl
add edi,ecx
neg ecx
 
mov edx,eax
again:
and edx,0x000000ff
shr eax,8
movzx esi,byte ptr [edi+ecx]
xor edx,esi
mov esi,[ebx+edx*4]
xor eax,esi
inc ecx
mov edx,eax
jnz again
}
}
#pragma warning(pop)
 

Generalcrc == 0memberswinefeaster21 Apr '02 - 17:28 
Hey there,
 
I'm using your awesome crc generator. I have a question though: is it possible that a crc would be returns as 0, and how probable is this? I am assuming that I don't have a crc if it is equal to 0, so I'm wondering how safe is this of an assumption.
 
Cheers,
 
swinefeaster
 
Check out Aephid Photokeeper, the powerful digital
photo album solution at www.aephid.com.

GeneralRe: crc == 0memberBrian Friesen21 Apr '02 - 18:44 
A CRC of 0 is entirely possible. The most likely case for a CRC of 0 is in the case of a zero byte file. But it is also possible for a non-zero byte file to have a CRC of 0. Basically all CRC32s from 0 to 0xFFFFFFFF are possible. Thus you cannot rely on the CRC return value to determine if the CRC was successful. That's way all of my CRC functions take the CRC as a parameter and return an error code from the function. You must use that error code to check for errors, not the CRC value.
GeneralRe: crc == 0memberswinefeaster21 Apr '02 - 18:49 
Thanks!
 
Check out Aephid Photokeeper, the powerful digital
photo album solution at www.aephid.com.

GeneralRe: crc == 0memberswinefeaster22 Apr '02 - 20:15 
What if I were to wrap your crc generator such that a crc or 0 is never returned? As in, if for example:
 

if(errorCode == NO_ERROR)
{
// Successful.
 
if(Crc == 0)
{
// Never return 0 as a crc.
Crc = 1;
}
}
else
{
Crc = 0;
}

 
Would this be a fairly reasonable thing to do? Obviously, any files with the real crc of 0 or 1 would be regarded as having the same crc, but this would still be a very small likelyhood, no?
 
What do yo think?
 
swine
 

 
Check out Aephid Photokeeper, the powerful digital
photo album solution at www.aephid.com.

GeneralRe: crc == 0sussAnonymous14 Sep '02 - 7:23 
why would you wanna do that ?
 
besides it wouldnt be a real crc32 checksum then.
GeneralRe: crc == 0memberDoug Gale9 Aug '04 - 1:41 
The original strength of the checksum is 1 in 4294967296. If you removed one of the possibilities, it would increase to 1 in 4294967295.
 
4294967295 / 4294967296 is 99.99999998% its original effectiveness.
 
BUT, why would you do that? How do you "not have a checksum"? Are you trying to cheat and steal a value as a "failure code". Smile | :) DON'T do that. Do this:
 
int SomeFunc(int someParam, unsigned *pRetCRC)
{
    // whatever...

    if (errorCode != NO_ERROR)
        return SOME_ERROR_VALUE;
 
    *pRetCRC = nTheCrc;
 
    return SOME_SUCCESS_VALUE;
}
 
and call it like this:
 
unsigned nMyCrc;
if (SomeFunc(nSomeVar, &nMyCrc) == SOME_ERROR_VALUE)
    return SOME_ERROR_VALUE;
 
// my crc is in nMyCrc here. SomeFunc filled it in.

GeneralRe: crc == 0memberDoug Gale9 Aug '04 - 1:27 
swinefeaster wrote:
is it possible that a crc would be returns as 0, and how probable is this?
 
Quite possible.
 
There is a 1 in 2**32 chance
 
or
 
1 in 4,294,967,296
 
chance of getting a zero crc. Four billion is NOT that big.

Generalcrc32 in a png filememberAnonymous7 Apr '02 - 5:49 
What's special with the crc32 algo used in the png files? Can somebody show me som code for this, which doesn't use a LUT?!?
GeneralRe: crc32 in a png filememberMike Nordell20 Jun '02 - 13:00 
What's special with the crc32 algo used in the png files?
 
???
Special? There's nothing special with the CRC32 usind in PNG, it's just the standard CRC-32 polynomial. What made you think there was anything special about it?
 
Can somebody show me som code for this, which doesn't use a LUT?!?
 
You almost always use a lookup table. The difference might be that some implementations provide it explicitly while others generate it "on the fly". For an example of the latter you might want to Google just one before asking. Try "CRC-32 generate".
GeneralPerformance improvementmemberAnonymous31 Mar '02 - 5:17 
On NT platforms it may be possible to increase the performance very slightly by using 2 buffers & overlapped IO, so that the OS is loading your next buffer whilst you are calculating the CRC of the current buffer. Does anyone with a good grasp of overlapped IO care to try it?
 
Thanks for the absolutely excellent sample Brian! Smile | :)
GeneralRe: Performance improvementmemberBrian Friesen5 Apr '02 - 13:46 
Thanks for the idea. I don't know why I didn't use overlapped IO in the first place. The results of my tests are overlapped IO is and isn't faster. Let me explain. I created several new CRC32 functions that were identical to the assembly functions, except the file reading was done using overlapped IO. When I ran the tests sometimes overlapped IO is faster, but most of the time it's not. The results were all over the place, especially in the "static" class. Sometimes the overlapped IO function was faster than assembly function, but most of the time it was slower. The results in the "dynamic" class were more consistent. The assembly function was always faster than the overlapped IO. To be honest, I don't even know why there is a speed difference between the dynamic and static classes in the first place. However, one thing is true from my testing, the dynamic assembly function is still the fastest. Faster than overlapped IO and definitely faster than filemaps.
 
Again, these test results come as a surprise to me. I originally expected the filemap function to be the fastest, but that proved wrong do to the sequential versus random file access. But I would have expected overlapped IO to be faster. The only reason a programmer would want to use overlapped IO is because there is a lot of processing that could be done while the file is being read. My guess right now is the assembly CRC code is so efficient that it completes the CRC'ing of the previous buffer so fast, that it ends up waiting for the next buffer just as if it were synchronous IO. But the extra overhead associated with overlapped IO (creation of multiple buffers and copying the data from one buffer to the other) is enough to cause the code to run slower.
 
Nonetheless, I'll try and get my overlapped IO code posted just in case someone else can tweak the code so that it runs faster.

GeneralRe: Performance improvementmemberxayide16 Nov '06 - 23:54 
Hi,
 
I am building an SFV-checker in MFC and found a small ASM routine to calculate the CRC32 just like yours. I am trying to get overlapped I/O but havent got that working correctly as beeing faster. Would you mind pasting your code. Maybe I can do some improvments. The code I have today is fairly fast Intel but very slow on AMD. Still I/O seems to be the bottleneck rather than the CPU.

QuestionDoing a check sum on resource files?memberAnonymous24 Feb '02 - 8:08 
What if I want to run a check sum on one of my internal resource files? To add a bit of protection to my own software? How could i use this to do that?
GeneralJust what I needed!memberswinefeaster14 Jan '02 - 16:49 
Thankx!
 
Check out Aephid Photokeeper, the powerful digital
photo album solution at www.aephid.com.

GeneralCRC resourcesmemberVernon31 Dec '01 - 9:32 
I found a great resource at: http://www.ross.net/crc/ See the "Painless Guide". Math explanation given in simplified form. C code included.
 
Vernon Sauder
GeneralQuestion about assemblymemberRick York17 Dec '01 - 19:20 
Could you elaborate a bit on what was done in the assembly version of the algorithm ? I am kind of curious.
 
The math is actually rather interesting. It involves an ex-or for each byte followed by an and, shift, and another ex-or for each bit of each byte. This makes it very computationally expensive. This is one reason that lookup tables were generated for it. I have worked with the CRC-16 algorithm but not CRC-32. I suppose that they are not significantly different. Oddly enough, I have run across a few PLC communication protocols that rely on CRC-16 for error checking.
 
Another odd story: back in the early 80s DEC VAXs were popular for factory automation applications. We developed a driver for the RTU protocol that utilized CRC-16 for error checking. This would absolutely swamp a VAX-750. We had a service rep come out and modify the wire-wrapped motherboard and install new firmware to support a new CRC-16 instruction performed by the CPU hardware. This sped things up quite a bit but I was shocked by what he had to go through. It was something else seeing a CPU that occupied a motherboard approximately 50x50cm and it was wire wrapped. OMG | :OMG:
 
Those were the old days. Not necessarily good, just old. Big Grin | :-D

GeneralRe: Question about assemblymemberTim Smith18 Dec '01 - 2:01 
He just used an assembler version of the table lookup.
 
And yes, 32 bit CRCs are just like 16 bit CRCs excluding the size of the polynomial and thus the results.
 
Tim Smith
Descartes Systems Sciences, Inc.
GeneralRe: Question about assemblymemberRick York18 Dec '01 - 5:36 
The number EX-ORed with each bit shift and mask is 0xA001 for CRC16. What is for CRC32 ?

GeneralRe: Question about assemblymemberTim Smith18 Dec '01 - 10:45 
Well, all CRCs are based off of the same theory of polynomial arithmetic (most of which I don't understand.) The size of the polynomial is where the name comes from. However, early on people started calling very specific CRCs based on the polynomial size. Thus we have CRCs called CRC16 and CRC32. These names don't even come close to describing all the elements of a CRC.
 
Here is all the different parameters for a CRC:
 
1. What is the polynomial?
2. Are the input bytes reflected?
3. What is the initial value of the CRC?
4. Is the final value XORed?
5. Is the polynomial reversed?
 
1. The polynomial specifies the mask of the CRC and the size. A polynomial such as x^16 + x^12 + x^5 + 1 has a mask of 0x1021. The X^16 isn't included in the mask. A 32 bit CRC would have an x^32 element.
 
2. When each bit is processed in a CRC, you can either start from the high bit and move to the low bit, or the low bit and move to the high. The latter case is know as reflected. Now, the slow way of doing a reflected CRC is to reflect the input values as they are being processed. The fast way is to reflected the whole bloody algorithm including the polynomial mask. Thus making the reflected CRC just as fast as the non-reflected. In my experience, it is more common to find reflected algorithms than non-reflected. If your CRC is being shifted left, that is a non-reflected CRC. If your CRC is being shifted to the right, that is a reflected CRC.
 
3. The initial value of a CRC is very important. This value is usually 0xFFFF for a 16 bit crc and 0xFFFFFFFF for a 32 bit crc. Some CRCs use a 0 initial value which isn't a very good idea since you can't tell the difference between a stream of 11 bytes of 0 and 12 bytes of zero.
 
4. Many algorithms XOR the final value with 0xFFFF for a 16 bit crc and 0xFFFFFFFF for a 32 bit crc. This is purely up to the whim of the programmer.
 
5. Well, some moron had to do it. They used a non-reversed polynomial mask in a reflected algorithm or they used a reversed polynomial mask in a non-reflected algorithm. The good news is that a reversed poly seems to work just as well.
 
So, to answer your question, which CRC32 are you talking about? I have seen two or three.
 
Tim Smith
Descartes Systems Sciences, Inc.
GeneralRe: Question about assemblymemberTim Smith18 Dec '01 - 10:48 
FYI:
 
A 6 bit CRC or 67 bit CRC is perfectly valid.
 
Tim Smith
Descartes Systems Sciences, Inc.
GeneralRe: Question about assemblymemberRick York18 Dec '01 - 12:19 
Thanks, that is some interesting info. The specific CRC16 I worked with was used as a checksum in the RTU communications protocol. I have seen this same protocol used with several PLCs and RTUs over the years. Regarding the issues you mentioned :
 
- it uses 0xFFFF as an initial value.
- it does right shifting so it is reflected.
- if the low order bit is one prior to the shift, the value is xor'ed with 0xA001
- there is no final xor done.
 
I had no specific CRC32 in mind. I was just curious about what the value of the xor mask was. I looked through the ziplib code that I have and I noticed that they use a lookup table also for their CRC32 so the xor mask is built in.
 
Anyway, thanks again.

GeneralRe: Question about assemblymemberTim Smith18 Dec '01 - 14:05 
Looking at the source, it seems it is a reflected algorithm.
 
0x04C11DB7 is the non-reflected mask, thus a polynomial of:
 
x^32+x^27+x^24+x^23+x^17+x^13+x^12+x^11+x^9+x^8+x^6+x^5+x^3+x^2+x^1 (nasty looking, ain't it)
 
There is also a final XOR.

 
Tim Smith
Descartes Systems Sciences, Inc.
GeneralRe: Question about assemblymemberBrian Friesen18 Dec '01 - 13:30 
Well I don't know really how to elaborate on what the assembly code does. The assembly code is pretty well documented as to what each line does.
 
To see a C++ version of the assembly code, compare the two functions FileCrc32Win32 and FileCrc32Assembly. The functions are identical except inside the while loop. The first function has a for loop inside of which it calls the CalcCrc32 function. The assembly function does this exact same logic as this except it does it in hand written assembly. The 'crc32loop' marks the beginning of the loop, which replaces the for loop. The next 8 lines of assembly do the same computation as the CalcCrc32 function. The assembly code above and below the 'crc32loop' is housekeeping if you will, getting set up for the loop.

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Permalink | Advertise | Privacy | Mobile
Web01 | 2.6.130516.1 | Last Updated 18 Dec 2001
Article Copyright 2001 by Brian Friesen
Everything else Copyright © CodeProject, 1999-2013
Terms of Use
Layout: fixed | fluid