Hooks and DLLs

• Download demo project - 22 Kb

Introduction

There is a lot of confusion about how to set up and use global hook functions. This essay attempts to clear up some of these issues.

It may be worth pointing out that Flounders disapprove of hooks in general, but these hooks are felt to be acceptable.

Note that none of the problems described below occur if you are simply hooking operations from your own process. This only happens if you want to get events system-wide.

The key problem here is address space. When a global DLL executes, it executes in the context of the process whose event it is hooking. This means that the addresses it sees even for its own variables are addresses in the context of the target process. Since this is a DLL, it has a private copy of its data for every process that is using it, which means that any values you set in variables global in the DLL (such as those declared at file-level) are private variables, and will not inherit anything from the original DLL's context. They are going to be initialized anew, meaning, typically, they will be zero.

A recent post even suggested the notion of storing a callback address in the DLL. This is impossible. Well, it is not impossible to store it, but it is impossible to use it. What you've stored is a bunch of bits. Even if you follow the instructions below to create a shared memory variable that is visible to all instances of the DLL, the bunch of bits (which you think is an address) is actually meaningful as an address only in the context of the process that stored it. For all other processes, this is merely a bunch of bits, and if you try to use it as an address, you will call some address in the process whose event is being intercepted, which is completely useless. It will most likely just cause the app to crash. 

This concept of separate address spaces is a hard concept to grasp. Let me use a picture to illustrate it.

What we have here are three processes. Your process is shown on the left. The DLL has code, data, and a shared segment, which we'll talk about how to do later. Now when the hook DLL executes to intercept an event for Process A, it is mapped into Process A's address space as shown. The code is shared, so the addresses in Process A refer to the same pages as the addresses in Your Process.  Coincidentally, they happen to be relocated into Process A at the same virtual addresses, meaning the addresses process A sees. Process A also gets its very own private copy of the data segment, so anything in "Data" that Process A sees is completely private to Process A, and cannot affect any other process (or be affected by another process!). However, the trick that makes this all work is the shared data segment, shown here in red. The pages referred to by Your Process are exactly the same memory pages referred to in Process A. Note that coincidentally, these pages happen to appear in Process A's address space in exactly the same virtual addresses as in Your Process. If you were sitting debugging Your Process and Process A concurrently (which you can do with two copies of VC++ running!), if you looked at &something that was in the shared data segment, and looked at it in Your Process and then at the same &something in Process A, you would see exactly the same data, even at the same address. If you used the debugger to change, or watched the program change, the value of something, you could go over to the other process, examine it, and see that the new value appeared there as well.

But here's the kicker: the same address is a coincidence. It is absolutely, positively, not guaranteed. Take a look at Process B. When the event is hooked in Process B, the DLL is mapped in. But the addresses it occupied in Your Process and Process A are not available in Process B's address space. So all that happens is the code is relocated into a different address in Process B. The code is happy; it actually doesn't care what address it is executing at. The data addresses are adjusted to refer to the new position of the data, and even the shared data is mapped into a different set of addresses, so it is referenced differently. If you were running the debugger in Process B and looked at &something in the shared area, you would find that the address of something was different, but the contents of something would be the same; making a change in the contents in Your Process or Process A would immediately make the change visible in Process B, even though Process B sees it at a different address. It is the same physical memory location. Virtual memory is a mapping between the addresses you see as a programmer and the physical pages of memory that actually comprise your computer.

While I've referred to the similar placement as a coincidence, the "coincidence" is a bit contrived; Windows attempts whenever possible to map DLLs into the same virtual location as other instances of the same DLL. It tries. It may not be able to succeed. 

If you know a little bit (enough to be dangerous), you can say, Aha! I can rebase my DLL so that it loads at an address that does not conflict, and I'll be able to ignore this feature. This is a prime example of a little knowledge being a dangerous thing. You cannot guarantee this will work in every possible executable that can ever run on your computer! Because this is a global hook DLL, it can be invoked for Word, Excel, Visio, VC++, and six thousand applications you've never heard of, but you might run someday or your customers might run. So forget it. Don't try rebasing. You will lose, eventually. Usually at the worst possible time, with your most important customer (for example, the magazine reviewer of your product, or your very best dollar-amount customer who is already nervous about other bugs you may have had...)  Assume that the shared data segment is "moveable". If you didn't understand this paragraph, you don't know enough to be dangerous, and you can ignore it.

There are other implications to this relocation. In the DLL, if you had stored a pointer to a callback function in Your Process, it is meaningless for the DLL to execute it in Process A or Process B. The address will cause a control transfer to the location it designates, all right, but that transfer will happen into Process A or Process B's address space, which is pretty useless, not to mention almost certainly fatal.

It also means you can't use any MFC in your DLL. It can't be an MFC DLL, or an MFC Extension DLL. Why? Because it would call MFC functions. Where are they? Well, they're in your address space. Not in the address space of Process A, which is written in Visual Basic, or Process B, which is written in Java. So you have to write a straight-C DLL, and I also recommend ignoring the entire C runtime library. You should only use the API. Use lstrcpy instead of strcpy or tcscpy, use lstrcmp instead of strcmp or tcscmp, and so on. 

There are many solutions to how your DLL communicates to its controlling server. One solution is to use ::PostMessage or ::SendMessage (note that I refer here to the raw API calls, not MFC calls!) Whenever it is possible to use ::PostMessage, use it in preference to ::SendMessage, because you can get nasty deadlocks. If Your Process stops, eventually, every other process in the system will stop because everyone is blocked on a ::SendMessage that will never return, and you've just taken the entire system down, with potential serious lossage of data in what the user sees as critical applications. This is Most Decidedly Not A Good Thing.

You can also use queues of information in the shared memory area, but I'm going to consider that topic outside the scope of this essay.

In the ::SendMessage or ::PostMessage, you cannot pass back a pointer (we'll ignore the issue of passing back relative pointers into the shared memory area; that's also outside the scope of this essay). This is because any pointer you can generate is either going to be referring to an address in the DLL (as relocated into the hooked process) or an address in the hooked process (Process A or Process B) and hence is going to be completely useless in Your Process. You can only pass back address-space- independent information in the WPARAM or LPARAM.

I strongly suggest using Registered Window Messages for this purpose (see my essay on Message Management). You can use the ON_REGISTERED_MESSAGE macro in the MESSAGE_MAP of the window you send or post the message to.

Getting the HWND of that window in is now the major requirement. Fortunately, this is easy.

The first thing you have to do is create the shared data segment. This is done by using the #pragma data_seg declaration. Pick some nice mnemonic data segment name (it must be no more than 8 characters in length). Just to emphasize the name is arbitrary, I've used my own name here. I've found that in teaching, if I use nice names like .SHARE or .SHR or .SHRDATA, students assume that the name has significance. It doesn't.

#pragma data_seg(".JOE")
HANDLE hWnd = NULL;
#pragma dta_seg()
#pragma comment(linker, "/section:.JOE,rws")

Any variables you declare in the scope of the #pragma that names a data segment will be assigned to the data segment, providing they are initialized. If you fail to have an initializer, the variables will be assigned to the default data segment and the #pragma has no effect.

It appears at the moment that this precludes using arrays of C++ objects in the shared data segment, because you cannot initialize a C++ array of user-defined objects (their default constructors are supposed to do this). This appears to be a fundamental limitation, an interaction between formal C++ requirements and the Microsoft extensions that require initializers be present.

The #pragma comment causes the linker to have the command line switch shown added to the link step. You could go into the VC++ Project | Settings and change the linker command line, but this is hard to remember to do if you are moving the code around (and the usual failure is to forget to change the settings to All Configurations and thus debug happily, but have it fail in the Release configuration. So I find it best to put the command directly in the source file. Note that the text that follows must conform to the syntax for the linker command switch. This means you must not have any spaces in the text shown, or the linker will not parse it properly.

You typically provide some mechanism to set the window handle, for example

void SetWindow(HWND w)
   {
    hWnd = w;
   }

although this is often combined with setting the hook itself as I will show below.

Sample: A Mouse Hook

header file (myhook.h)

The functions setMyHook and clearMyHook must be declared here, but this is explained in my essay on The Ultimate DLL Header File.

#define UWM_MOUSEHOOK_MSG \
	_T("UMW_MOUSEHOOK-" \
        "{B30856F0-D3DD-11d4-A00B-006067718D04}")

source file (myhook.cpp)

#include "stdafx.h"

#include "myhook.h"


#pragma data_seg(".JOE")
HWND hWndServer = NULL;
#pragma data_seg()
#pragma comment("linker, /section:.JOE,rws")

HINSTANCE hInstance;
UINT HWM_MOUSEHOOK;
HHOOK hook;

// Forward declaration

static LRESULT CALLBACK msghook(int nCode, WPARAM wParam, LPARAM lParam);

/****************************************************************
*                               DllMain
* Inputs:
*       HINSTANCE hInst: Instance handle for the DLL
*       DWORD Reason: Reason for call
*       LPVOID reserved: ignored
* Result: BOOL
*       TRUE if successful
*       FALSE if there was an error (never returned)
* Effect:
*       Initializes the DLL.
****************************************************************/

BOOL DllMain(HINSTANCE hInst, DWORD Reason, LPVOID reserved)
{
 switch(Reason)
   { /* reason */
    //**********************************************

    // PROCESS_ATTACH

    //**********************************************

    case DLL_PROCESS_ATTACH:
       // Save the instance handle because we need it to set the hook later

       hInstance = hInst;
       // This code initializes the hook notification message

       UWM_MOUSEHOOK = RegisterWindowMessage(UWM_MOUSEHOOK_MSG);
       return TRUE;

    //**********************************************

    // PROCESS_DETACH

    //**********************************************

    case DLL_PROCESS_DETACH:
       // If the server has not unhooked the hook, unhook it as we unload

       if(hWndServer != NULL)
          clearMyHook(hWndServer);
       return TRUE;
   } /* reason */

/****************************************************************
*                               setMyHook
* Inputs:
*       HWND hWnd: Window whose hook is to be set
* Result: BOOL
*       TRUE if the hook is properly set
*       FALSE if there was an error, such as the hook already 
*             being set
* Effect:
*       Sets the hook for the specified window.
*       This sets a message-intercept hook (WH_GETMESSAGE)
*       If the setting is successful, the hWnd is set as the
*       server window.
****************************************************************/

__declspec(dllexport) BOOL WINAPI setMyHook(HWND hWnd)
  {
   if(hWndServer != NULL)
      return FALSE;
   hook = SetWindowsHookEx(
                           WH_GETMESSAGE,
                           (HOOKPROC)msghook,
                           hInstance,
                           0);
   if(hook != NULL)
     { /* success */
      hWndServer = hWnd;
      return TRUE;
     } /* success */
   return FALSE;
  } // SetMyHook

/****************************************************************
*                             clearMyHook
* Inputs:
*       HWND hWnd: Window whose hook is to be cleared
* Result: BOOL
*       TRUE if the hook is properly unhooked
*       FALSE if you gave the wrong parameter
* Effect:
*       Removes the hook that has been set.
****************************************************************/
__declspec(dllexport) BOOL clearMyHook(HWND hWnd)
   {
    if(hWnd != hWndServer)
       return FALSE;
    BOOL unhooked = UnhookWindowsHookEx(hook);
    if(unhooked)
       hWndServer = NULL;
    return unhooked;
   }

/****************************************************************
*                              msghook
* Inputs:
*       int nCode: Code value
*       WPARAM wParam: parameter
*       LPARAM lParam: parameter
* Result: LRESULT
*
* Effect:
*       If the message is a mouse-move message, posts it back to
*       the server window with the mouse coordinates
* Notes:
*       This must be a CALLBACK function or it will not work!
****************************************************************/

static LRESULT CALLBACK msghook(int nCode, WPARAM wParam, LPARAM lParam)
   {
    // If the value of nCode is < 0, just pass it on and return 0

    // this is required by the specification of hook handlers

    if(nCode < 0)
      { /* pass it on */
       CallNextHookEx(hook, nCode,
                   wParam, lParam);
       return 0;
      } /* pass it on */

    // Read the documentation to discover what WPARAM and LPARAM

    // mean. For a WH_MESSAGE hook, LPARAM is specified as being

    // a pointer to a MSG structure, so the code below makes that

    // structure available


    LPMSG msg = (LPMSG)lParam;

    // If it is a mouse-move message, either in the client area or

    // the non-client area, we want to notify the parent that it has

    // occurred. Note the use of PostMessage instead of SendMessage

    if(msg->message == WM_MOUSEMOVE ||
       msg->message == WM_NCMOUSEMOVE)
      PostMessage(hWndServer,
                  UWM_MOUSEMOVE,   
                  0, 0); 

    // Pass the message on to the next hook

    return CallNextHookEx(hook, nCode, 
                       wParam, lParam);
   } // msghook

The server application

In the header file, add this to the protected section of the class:

afx_msg LRESULT OnMyMouseMove(WPARAM,LPARAM);

In the application file, add this at the front of the file somewhere:

UINT UWM_MOUSEMOVE = ::RegisterWindowMessage(UWM_MOUSEMOVE_MSG);

In the MESSAGE_MAP, add the following line outside the magic //{AFX_MSG comments:

ON_REGISTERED_MESSAGE(UWM_MOUSEMOVE, OnMyMouseMove)

In your application file, add the following function:

LRESULT CMyClass::OnMyMouseMove(WPARAM, LPARAM)
   {
    // ...do stuff here

    return 0;
   }

I've written a little sample application to show this, but since I was bored doing a global hook function for the n+1^st time, I gave it a nice user interface. The cat looks out the window and watches the mouse. But be careful! Get close enough to the cat and it will grab the mouse!

You can download this project and build it. The real key is the DLL subproject; the rest is decorative fluff that uses it.

There are several other techniques shown in this example, including various drawing techniques, the use of ClipCursor and SetCapture, region selection, screen updating, etc., so for beginning programmers in various aspects of Windows programming this has other value besides demonstrating the use of the hook function.

The views expressed in these essays are those of the author, and in no way represent, nor are they endorsed by, Microsoft.

Send mail to newcomer@flounder.com with questions or comments about this article.
Copyright © 2001 All Rights Reserved
www.flounder.com/mvp_tips.htm