Win32 API hooking: Another reason why it might not work

Introduction

Like (many) other people, I've been quite fascinated by the possibilities given by Win32 hooking. After reading API hooking revealed, I decided the best way to do it was to rewrite the Import Address Table (IAT) in the application I targeted. This technique is called IAT-patching. It worked quite well for some time, but then, my program failed to hook properly on a number of target applications, such as the MSN Messenger or the latest versions of Emule. I first suspected some kind of protection within these applications, but since Emule is open source, I could download the code and see there was no such thing that should have made IAT-patching fail. Then, an in-depth look at how the function I targeted was called from MSN and Emule allowed me to understand where the problem came from: the targeted function, imported from a standard DLL, was not loaded into these executables in the standard way, based upon the IAT (and which is described by Matt Pietrek in MSDN). I'll describe here how the loader deals with these functions, and why this undocumented feature of the Portable Executable format makes IAT-patching powerless.

Background

Since my goal here is to show why IAT-patching does not always work, it would be quite useful to know how IAT is supposed to work. The two articles I mentioned in the introduction are the best places for a starter.

The standard way of calling an imported function

When a function is imported into an executable, the loader stores the actual address of this function into the IAT before any call to this function is made. In an application that calls its imported functions in a standard way, here is what you get when you make such a call after the loader has done its job:

00412CB7   CALL DWORD PTR DS:[0042C544]

The four bytes at DS:[0042C544] are inside the IAT, they store the actual address (that is 0x77D5C411) of the function I intend to hook (in this case: SetMenuItemInfoW from User32.dll). Rewriting the IAT basically means I'll change the address written at DS:[0042C544] and write the address of my replacement function instead.

How SetMenuItemInfoW is called in Emule

The problem is: SetMenuItemInfoW is not called this way in Emule. The first thing to say about SetMenuItemInfoW is that you won't even find it in the IAT if you use programs such as PEView or DUMPBIN. And still, it is used by emule.exe. Here's the assembler code for Emule:

00568A65   CALL DWORD PTR DS:[0074BE08]

This assembler line is the line that corresponds to the C++ line that calls SetMenuItemInfoW, but the address stored at DS:[0074BE08] is not the address of SetMenuItemInfoW, and the four bytes at 0074BE08 are not inside the IAT: there's no way you could change the value there. The address pointed to by PTR DS:[0074BE08] is 005D4F70, and it is inside emule.exe itself. Here is what we have at 005D4F70:

As you can see, instead of calling SetMenuItemInfoW, the code calls a function located at 005D5529. I'll call it the "loader function" in this article. This function takes five arguments, three of them are interesting to describe:

• Arg3=0074BE08 is the address used just before (at 00568A65: CALL DWORD PTR DS:[0074BE08]). (It is cast here as the address of an ASCII string, this is due to a little problem with my assembler debugger.)
• Arg2 is the name of the function originally called: "SetMenuItemInfoW".
• Arg1 is the name of the DLL that exports SetMenuItemInfoW, "user32.dll".

(You can see that this function is also called for other imported functions such as EnumFontsW or GetDateFormatW...).

Immediately after the call to the loader function, there is a line of code (at 005D4F8F) that calls the same address as the line at 00568A65 : JMP DWORD PTR DS:[0074BE08]. Remember the address stored at DS:[0074BE08] is the address of the loader function. That is, once the loader function has returned, this JMP instruction will cause the loader function to be called again, in an infinite cycle.

I won't give too much useless information as to what the loader function does. Here is the most important: it first calls LoadLibrary to get a handle to the required DLL, and then it looks up the name of the function originally called inside the Export Address Table of that DLL in order to calculate the actual address of this function (note: the loader function does not use GetProcAddress).

Once the loader function has returned, the line that follows it is modified: it is still JMP DWORD PTR DS:[0074BE08], but the location pointed to by DS:[0074BE08] is no longer 005D4F70 but the actual address of SetMenuItemInfoW (= 77D5C411). So, the JMP will not call the loader function again, but SetMenuItemInfoW instead.

Now, we can jump to SetMenuItemInfoW for good.

As you noticed, the first time, at 00568A65, we "called" the address DS:[0074BE08] (CALL DWORD PTR ...), and the second time, at 005D4F8F, we "jumped" (JMP DWORD PTR ...). This way, when SetMenuItemInfoW returns, it will not return to the line where it was actually called (with a JMP) but to the original CALL line.

Of course, since the address in the JMP and in the CALL are the same, the next time we call SetMenuItemInfoW, at 00568A65, we will call the actual function not the protocol we had to go through for the first call.

The next schemas will illustrate this discussion:

• before the first call:

  

• right after the loader function has returned:

  

• all the subsequent calls:

  

What are the consequences for Win32 hooking?

Since this mechanism does not use IAT, there is no way you can intercept calls to SetMenuItemInfoW by altering the IAT. And there is no way to say where to find the four bytes that will be altered by the loader function: those four bytes are inside the executable, exactly where the imported function is called.

There are still two possibilities that could allow you to hook such a function.

The first one is similar to IAT altering: but instead of rewriting entries in the IAT, you have to rewrite entries in the Export Address Table (EAT) of the DLL that hosts the targeted function. But this has to be done before any call to the targeted function is made; once the first call is made, the loader function has already looked up the EAT and written the actual address. So the trick is to alter the EAT so that the loader function will read the address of your own function.

The second possibility consists in changing the first few bytes at the beginning of the function you want to hook, the new bytes will be a JMP to your own function.

This solution is in fact the most efficient since it enables you to bypass not only the protocol described in this article but also other forms of protection against Win32-hooking (use of GetProcAddress, encrypted executables ...). But it goes beyond the scope of this article. Anyway, you'll find all the necessary information here.

Conclusion

Going through this protocol is a lengthy process, so it is implemented only if the targeted function is called from a single location in the executable. Calling the loader function only once is similar to processing the IAT: both protocols must load the DLL and calculate the actual address of the function. But, while the IAT must be processed when the executable is launched, the process described here happens only when the function is called for the first time, causing the executable to launch faster. It is thus similar to the delay-import function mechanism.