|
|
Comments and Discussions
|
|
 |

|
Jeffrey,
We have a task we are working upon related to performing a MD5 hash verification upon a Windows XP operating system image. This is a new area to us and we are battling a bit. Also, I am not the top guy here on this but I think I can decribe the issue and maybe if needed I can pass this off to answer any retrun questions.
After finding your article, which is excellent, I would like to ask you a basic question pertaining to this sample application code in your article that you might easily be able to answer.
We develop in VB .Net and have started writing code to try and perform the MD5 verification task but have hit a problem with open/locked files used by the Windows XP OS.
Briefly, we are required to do the following. Booot up the PC into XP. Our application then must automatically do a MD5 verification against the contents of the C: drive (XP Operating System) and D: (Just some program files for an application)
We are hitting a wall with trying to access some of the Windows XP files as they are already open when we are running our application, etc.
Just a really simple question, can using a shell extension DLL such as the approach you have taken bypass this issue? We just need to do a MD5 verification on our XP OS, thus should there be an entirely different approach than what we are doing you might have a comment.
Thanks for any insight
Thanks,
Jamey
|
|
|
|
|

|
I've been trying to use this program and even crypto++ by itself to try and generate md5 checksums of files for internal release management purposes. Some of the files in question have Asian Characters in the names and may be in folders with completely Asian character names. So far, I haven't found anything that works. I've tried the standard WINAPI conversion functions. They allow crypto++ to work with characters such as tildes (~) in Spanish file names, but it still dies when it hits the first file with Asian characters in the path. Do you have any suggestions?
|
|
|
|

|
Hi Paul,
I believe the problem is in Microsoft's STL implementation. I've suffered a lot of hardships, especially when trying to do simple things such as write out a wstring using a wstream.
> Do you have any suggestions?
Use the Win32 API to open the file rather than a FileSource. Take the file, map a view into memory (it now looks like a BYTE* and with a size) with Win32's CreateFile/CreateFileMapping/MapViewOfFile. Then send it into Crypto++.
Unfortunately, I use the STL implemntation because of Crypto++'s affinity towards it. Other than that, I try to stay away from STL.
Jeff
|
|
|
|

|
Thanks for the speedy reply and the great suggestions.
|
|
|
|

|
Ok, I experience the same thing. So now I know what to do, thought it's with the file extension dll. I'll try your suggestion now.
|
|
|
|

|
???????? ??? ???????????????????????? ????????
|
|
|
|

|
Hi abdelfattah,
abdelfattah wrote: ???????? ??? ???????????????????????? ????????
Unfortunately, I do not read arabic (and CP does not display it well).
Jeff
|
|
|
|

|
I'm having problems registering "CtxCreateHash.dll" and "CtxVerifyHash.dll" with the system. I get the following error for CtxCreateHash.dll:
---------------------------
RegSvr32
---------------------------
LoadLibrary("C:\Windows\System32\CtxCreateHash.dll") failed - This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.
---------------------------
OK
---------------------------
and for CtxVerifyHash.dll:
---------------------------
RegSvr32
---------------------------
LoadLibrary("C:\Windows\System32\CtxVerifyHash.dll") failed - This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem.
---------------------------
OK
---------------------------
Any ideas what should I do? My machine is running Win XP Pro SP2, and it is running under administrator privileges when I try to register the DLLs.
|
|
|
|
|

|
Hi,
I have written a simple code to add a menu ...
But my code after dllmain and dllGetClassObject goes out(As I know It must go to
Initialize )
I have checked my code and yours...They are the same...(VS2005)
Please Help me.
Thank you.
Every new thing you learn,Gives you a new personality.
|
|
|
|

|
Hi Solariu,
> ...goes out
???
Jeff
|
|
|
|

|
Hi
Excuse me ... My English is not good,
I mean After dllGetClassObject it doesn't do anything...
Every new thing you learn,Gives you a new personality.
|
|
|
|

|
Hi, i'm trying to get working this shell extension. "Create Checksums" works well, but "Verify" option never appears
I use Windows XP SP2 in spanish, but my colleague uses XP in english and the results are the same.
Any ideas?
Julio Daniel Moreyra
|
|
|
|
|

|
Hi Jeffrey, so no chances to create and verify in the same machine?
Well, if somebody can contribute with a program that can verify ...
Julio Daniel Moreyra
|
|
|
|

|
Hi Julio,
Julio Moreyra wrote: no chances to create and verify in the same machine
It will be fixed. I've been out of town on a customer's site for the last few weeks. Also my laptop does not have Visual Studio and WinDbg installed, so I can't work on it on the rare occasion I have some time at night.
On the good side, the last weekend I spent at my home, I verified the WinDbg setup, so I should be ready to go... (see An Analysis of the Windows PE Checksum Algorithm[^]).
So, in closing, I need more time in a day
Jeff
|
|
|
|
|

|
For example:
when right click blank place of desktop or an opened explorer, a menu will appear.
it contain "view"
"Arrange Icons By"
"Paste"
"Paste Shortcut"
"New"
"Properties" and ...
now i want to insert a menu like "Paste", how can do it?
I know that how to insert a menu to right click menu of a file or folder as you introduced.
string keyname = "Reg Paste";
string guid = "{" + typeof(PasteMenu).GUID.ToString() + "}";
RegistryKey rk, rk2;
rk = Registry.ClassesRoot.OpenSubKey(@"xmlfile\shellex\ContextMenuHandlers", true);
rk2 = rk.OpenSubKey(keyname);
if (rk2 == null)
rk2 = rk.CreateSubKey(keyname);
rk2.SetValue("", guid);
rk2.Close();
rk.Close();
But how to registry to blank place of window?
Thanks very much if you can give me same advice!
take it easy & keep it simple....
|
|
|
|

|
Hi Hays,
Take a look at Michaels Articles - they are much more comprehensive than this with respect to shell programming.
Jeff
|
|
|
|

|
You have written an excellent series of cryptography related articles, so I think you are the best person to come to with this problem.
I am trying to reverse engineer the index.dat files used by internet explorers cache manager. As you may know all your history, cookies and offline content are stored in these files. Extracting the data is quite easy, but I am looking to do a little more than that.
Anyway, the files contain a hash table which IE uses for fast lookup. Each table row has a DWORD hash and a DWORD index. The hash is giving me a headache since I have so far been unsuccesful in reproducing it. Here are a few samples:
0: 73A7A400 10: D83D7500 20: 668A2540 30: 82E183C0
1: A7D808C0 11: 348B4DC0 21: FFE4CF40 31: 71DEEC00
2: AD9C5680 12: CA96CF40 22: 24B68100 32: 114A0B80
3: F4DA5FC0 13: 7BD6FD40 23: 34A14800 33: 01421C40
4: 4EA85FC0 14: C94CEF80 24: D8252AC0 34: 4F5CDA40
5: 6F40B200 15: 7509CD80 25: 92C1A440 35: DEE1BCC0
6: 526FB400 16: 77128380 26: 4601BF00 36: 8AF84BC0
7: DA715A40 17: DB5EAA80 27: 55018DC0 37: CD960100
8: 04E3AEC0 18: 46D03340 28: E9998EC0 38: 839C2600
9: 7FB4E800 19: B6849080 29: A4203E00 39: 94B5DF40
As you can see it's a 32 bit value. You may also notice the last byte of the hash always contains similar numbers, I'm hoping this will help in determining which algorithm was used. The last byte values are usually x00, x40, x80, xC0, there are some less common values of 0x04, x05 and x08. Could they be flags?
I'm guessing the url entered into the browser is what's being hashed, though without knowing the algorithm used it's uncertain.
Probably of no use, but it may help in identifying the algo:
The actual index.dat file is actually a memory mapped file, several processes are able to read and write to the file at any given time. So I am thinking that the hash table (or rather a copy) is not stored in the processes memory, which would make finding an entry in the table just a little bit more difficult. Since the files were designed with speed in mind, I'm going to assume a B-tree style search is used, could the last byte be related to a B-tree at all? Also, the hash entries are not stored in a linear form, they appear to be entered into the table (which has a fixed size) at random locations. It's highly improbable that it is random, it's something I'm looking into at the moment.
Also, many of the wininet functions state that entered url strings should not contain escape characters. So I'm going to assume that if a lookup table is being used to generate the hash, it will be of a size equal to allowed characters[^] for a url, possibly smaller is converted to upper/lowercase case first.
Anyway, if you could provide me with any help, pointers, suggestions or thoughts on this I would be very grateful.
|
|
|
|

|
Hi WalderMort,
It's been years since I used SoftIce. I would no longer recognizes mammon's style of writing
Shooting from the hip, I would try a CRC32 or LookUp3[^] as starters. LookUp3 is Collison Resistant and Fast, but it can be reversed. It works well for hash keys. I've read (in the past - I don't have any M$ citations) Microsoft will play a little truncation game with MD5. So I would not be suprised if 73A7A400 (entry 0) is not a substring of an MD5 hash of a file.
In Office Passwords[^], John Kuslich states:
The game Office plays with password protection is as follows:
Learn about structured storages. You will need to know this to be able to
access the "1Table" structure in the Excel file. The 1Table structure
contains three 16 byte numbers. The first is a random salt. The second is
a an MD5 hashed nonce encrypted using RC4 with a key (K). This encrypted
hash is stored as a second 16 byte number in 1Table. The nonce is encrypted
using key (K) and MD5 hashed then stored as a third 16 byte number in
1Table.
The key (K) is calculated in the following way:
The password (expressed in unicode) is MD5 hashed. The first five bytes of
the password hash are put into an array with the 1st 16 byte number (the
salt) stored in the 1Table structure. The salt is repeatedly concatenated
with the password and then padded according to the MD5 algorithm. The MD5
hash is taken. The first five bytes of this hash are saved and then padded
and MD5 hashed again. The first five bytes of this hash along with a counter
byte become the RC4 key for encrypting/decrypting the document. The counter
periodically re-keys the RC4 engine by incrementing the counter byte modulo
8.
This key is first MD5 hashed before RC4 key scheduling. This hash is the key
(K).
This sounds correct. I don't recall what the salt is, but if memory serves me, it is a phrase. Again, this is just to give you an idea...
I'm a bit tired now, so I'm going to have to read the post again in the morning when I am fresh. Hopefully, I will have a better answer for you.
Jeff
|
|
|
|

|
One other thing before I sign off tonight. If the algorithm for the autocomplete (when typing a URL) is similar to the system you are trying to crack, have a look at Algorithms[^] by Rivest, et al, Chapter 34 - String Matching. It may be a something as simple as a modulo system (2^24 = 0x1000000, roll back to the next prime). Read about the Rabin-Karp algorithm (aka Rabin-Karp Matcher). Also look at the Knuth-Morris-Pratt algorithm, and the Boyer-Moore algorithm (Moore is from Intel - Boyer may be Joanne Boyer who cracked LCGs). You probably recognise the rest of the suspects.
The Boyer-Moore algorithm is considered as the most efficient string-matching algorithm in usual applications.
Jeff
|
|
|
|

|
Thanks for all the suggestions and the links. It's been a long time since I dealt with hashing algorithms, other than the standard libraries and simple hashing of strings/files etc. I must say I have never seen an algorithm producing results like these. That last byte is pretty much consistent. From this I can think of 2 possibilites:
1. The hash is actualy 24 bits occupying 3 WORDS, the last byte is a flag of some sort.
2. The hash is larger than 32 bits, the last byte indicates a truncation.
Before I wipe the dust of SoftIce, I will read up on these algorithms you have pointed out. Thanks for the help
|
|
|
|

|
It occured to me that this has been reversed. If you can find Fravia's old essays...
Working (deeply) from memory one file is current 'recent' - for example, today or 1 week's MRU, the other is older. Whack IE's history while watching what goes on with Russinovick's FileMon and RegMon.
The fellow who reversed it was pissed that the information was being saved.
Jeff
|
|
|
|
|
|

|
Hi WlderMort,
WalderMort wrote: break out SoftIce and start going through Wininet.dll
I'd probably disassemble before inspecting a live target. The strings of the file names should jump out like a sore thumb. It will at least confirm you have zeroed in on the proper tasrget. Sourcer, WDASM, etc...
Jeff
|
|
|
|

|
Well, I have found the loop which creates the hash. I was surprised to see a 256 char lookup table. The hash is created from this table as you would expect, but for some as yet unknown reason the last byte is modified after the hash is created. My knowledge of asm is limited at best, I am currently trying to convert this loop to C code.
|
|
|
|

|
Found it:
DWORD HashKey( LPCSTR lpszKey )
{
static const unsigned char lookupTable[256] =
{
0x01, 0x0E, 0x6E, 0x19, 0x61, 0xAE, 0x84, 0x77,
0x8A, 0xAA, 0x7D, 0x76, 0x1B, 0xE9, 0x8C, 0x33,
0x57, 0xC5, 0xB1, 0x6B, 0xEA, 0xA9, 0x38, 0x44,
0x1E, 0x07, 0xAD, 0x49, 0xBC, 0x28, 0x24, 0x41,
0x31, 0xD5, 0x68, 0xBE, 0x39, 0xD3, 0x94, 0xDF,
0x30, 0x73, 0x0F, 0x02, 0x43, 0xBA, 0xD2, 0x1C,
0x0C, 0xB5, 0x67, 0x46, 0x16, 0x3A, 0x4B, 0x4E,
0xB7, 0xA7, 0xEE, 0x9D, 0x7C, 0x93, 0xAC, 0x90,
0xB0, 0xA1, 0x8D, 0x56, 0x3C, 0x42, 0x80, 0x53,
0x9C, 0xF1, 0x4F, 0x2E, 0xA8, 0xC6, 0x29, 0xFE,
0xB2, 0x55, 0xFD, 0xED, 0xFA, 0x9A, 0x85, 0x58,
0x23, 0xCE, 0x5F, 0x74, 0xFC, 0xC0, 0x36, 0xDD,
0x66, 0xDA, 0xFF, 0xF0, 0x52, 0x6A, 0x9E, 0xC9,
0x3D, 0x03, 0x59, 0x09, 0x2A, 0x9B, 0x9F, 0x5D,
0xA6, 0x50, 0x32, 0x22, 0xAF, 0xC3, 0x64, 0x63,
0x1A, 0x96, 0x10, 0x91, 0x04, 0x21, 0x08, 0xBD,
0x79, 0x40, 0x4D, 0x48, 0xD0, 0xF5, 0x82, 0x7A,
0x8F, 0x37, 0x69, 0x86, 0x1D, 0xA4, 0xB9, 0xC2,
0xC1, 0xEF, 0x65, 0xF2, 0x05, 0xAB, 0x7E, 0x0B,
0x4A, 0x3B, 0x89, 0xE4, 0x6C, 0xBF, 0xE8, 0x8B,
0x06, 0x18, 0x51, 0x14, 0x7F, 0x11, 0x5B, 0x5C,
0xFB, 0x97, 0xE1, 0xCF, 0x15, 0x62, 0x71, 0x70,
0x54, 0xE2, 0x12, 0xD6, 0xC7, 0xBB, 0x0D, 0x20,
0x5E, 0xDC, 0xE0, 0xD4, 0xF7, 0xCC, 0xC4, 0x2B,
0xF9, 0xEC, 0x2D, 0xF4, 0x6F, 0xB6, 0x99, 0x88,
0x81, 0x5A, 0xD9, 0xCA, 0x13, 0xA5, 0xE7, 0x47,
0xE6, 0x8E, 0x60, 0xE3, 0x3E, 0xB3, 0xF6, 0x72,
0xA2, 0x35, 0xA0, 0xD7, 0xCD, 0xB4, 0x2F, 0x6D,
0x2C, 0x26, 0x1F, 0x95, 0x87, 0x00, 0xD8, 0x34,
0x3F, 0x17, 0x25, 0x45, 0x27, 0x75, 0x92, 0xB8,
0xA3, 0xC8, 0xDE, 0xEB, 0xF8, 0xF3, 0xDB, 0x0A,
0x98, 0x83, 0x7B, 0xE5, 0xCB, 0x4C, 0x78, 0xD1
};
BYTE hash[4];
DWORD i;
for ( UINT i = 0; i < 4; ++i )
hash[i] = lookupTable[*lpszKey + i];
for ( lpszKey++; *lpszKey && ((lpszKey[0] != '/') || (lpszKey[1] != 0)); lpszKey++ )
{
for ( UINT i = 0; i < 4; i++ )
hash[i] = lookupTable[*lpszKey ^ hash[i]];
}
return *(DWORD *)hash;
}
The strange thing is the above code is so similar to that of WINE's wininet.c which I have known about for quite some time. I origionaly thought they had produced their own algorithm since their function never returned the same results. But, their function had a bug. The first loop was like this:
for ( UINT i = 0; i < 4; ++i )
hash[i] = lookupTable[i];
Notice that they don't take into account the value of the char when indexing the look up table.
Now I just need to figure out why wininet is modifying that last byte.
|
|
|
|

|
WalderMort wrote: My knowledge of asm is limited at best
If I can help you with the disassembly, let me know.
The last byte sounds like it is mutually exclusive:
PAGE_AVALABLE = 0x01;
PAGE_UNAVAILABLE = 0x02;
PAGE_... = 0x04;
Jeff
|
|
|
|

|
I origionaly thought that, but the bits are too far apart. I'm working on the assumption that the position within the hash table is directly related to the hash. It seems a little strange that the hash table always has a fixed size of 64 blocks of 7, the hash is always divisible by 64 and pairs are entered into a block in a seemingly random fashion. If it is related, it would allow for a very fast lookup.
|
|
|
|

|
As a follow-up and a thanks for the help, I have managed to solve it. The algorithm is calculated from the hash I posted earlier. After the hash is created, a bit of math is done to calculate the offset within the hash table. This allows for a fast lookup since a linear search is only required across 7 elements. The only thing I am unsure about now is how wininet handles collisions. I'm guessing that the url string would again be compared to the entered string, but right now I am a little too drunk to care. Suffice to say, for now I am able to both read and write to the damned files!
|
|
|
|

|
Hi WalderMort,
> I have managed to solve it.
I hope you write an article. If not, send over the code (off site). I'd like to look at their implementation. I will not disclose.
> how wininet handles collisions...
I would assume that if position i = hash ( URL ), and i is occupied, try i++ % x (x = 7?). I can get back into a manual later if you'd like. That algorithm works well on larger sets, but I'm not sure about smaller ones. Then again M$ may have thrown the Computer Science out the window.
Jeff
|
|
|
|

|
I probably will write an article, but either way I will send you the set of classes.
Thinking about the collision, there are 7 slots available for each hash in each table. 1 table has a max of 448 slots, after they are filled another table is created allowing for a further 7 hashes. Or it could be that as soon as the 7 slots are filled a new table is created regardless. I should look into this a little more maybe. Either way, if there is a collision, does it really matter if the data is overwritten? If it was, if and when an attempt to retrieve the data fails, the browser should download a fresh copy.
To be honest, there is too much asm in the dll, and trying to collide more than 7 urls is a challenge in itself. I will wait for the bug reports to come in.
There is just one more thing bugging me about these hashes. When they are stored, they are alligned to decimal 64. So for example the url "http://www.catch22.net/" ( a random url I have been working with throughout this ) would hash to 44AE7139, it would be changed to 44AE7100 before any lookup or being stored. Within the lookup loop, the hash is retrieved and again aligned to 64. But the stored hash is 44AE7104. So something is being added to the hash. It's not such a big deal at the moment, but it could be something important. ( BTW, this particular hash was stored in the 2nd slot of the 57th group ).
|
|
|
|

|
> there are 7 slots available for each hash
> in each table. 1 table has a max of 448 slots,
> after they are filled another table is created
> allowing for a further 7 hashes.
Interesting way of doing things. A linked list of vectors (or a vector of vectors)...
> Either way, if there is a collision...
Good point, but I don't think that is IE's behavior. IE appears to honor the setting in Options.
> To be honest, there is too much asm in the dll
I go crossed-eyed at times myself.
> they are alligned to decimal 64
<#pragma pack( 8 )>: probably a peephole optimization based on their computed hash size (optimized for tables rather than machine word size).
Jeff
|
|
|
|
|
|

|
Hi Coolboy,
My pleasure. The real credit goes to Michael Dunn... I attached some basic logic demonstrated on the command line.
Jeff
|
|
|
|

|
I am also learn a lot from Michael Dunn~~
|
|
|
|
 |
|
|
General News Suggestion Question Bug Answer Joke Rant Admin
|
Create a File Checksum Shell Menu Extension using ATL and Crypto++
| Type | Article |
| Licence | LGPL3 |
| First Posted | 15 Dec 2006 |
| Views | 149,213 |
| Downloads | 3,785 |
| Bookmarked | 104 times |
|
|