Owner drawn menus in two lines of code

• Download source files - 12.5 Kb
• Download demo project - 30.6 Kb

Table of contents

• Quick start guide
• Introduction
• Getting started
• Problems, problems
• Multiple WM_INITMENUPOPUP messages
• Shell file context menus - e.g., a right click in the Save as dialog box
• Other considerations
• TPM_NONOTIFY style
• Intercepting the OS
• Compiler problems
• Version history

Quick start guide

To get started using this with your application, follow these steps:

• Add the files ODMenu.h/.cpp, EnumerateLoadedModules.h/cpp to your application.
• Add these two lines to your CMainFrame class:

  #include "ODMenu.h"
      CODMenu m_ownerMenu; // member variable

• Add PSAPI.lib to your application's link information.

  

• Rebuild your application.

Introduction

I was looking through an old project of mine which was being reworked, and was not completely happy about how it implemented owner drawn menus. At the time of development, I was not very experienced in this area and made use of Brent Corkum's Cool Owner Drawn Menus with Bitmaps - Version 3.03 class. After fixing a few (minor) problems and having a lot of trouble with dynamic menu items, I got the system working. But it's an ugly area (and prone to errors on changes) of the application where few dare to venture now. *shudder*

I have also previously developed a plug-in owner menu handler for my plug-in series of articles. Although my application uses a very old variation of this technology, it would be far too much work to convert it across to run with this new method, so I thought "What if I modify my owner draw menu method so it hooks all the windows and does all the work, that way I can revert all my application menu code to CMenu?". So, with the bit between my teeth, I set off on my new journey.

Getting started

The first stage in the project was to learn about Windows hooks. This is an area that I had never covered before, so it was going to be a learning experience. First stop was the MSDN and later on the CodeProject VC++ forum, when I needed help on target areas which were not working correctly, or as hoped.

After an initial foray into the documentation, I thought that I needed to hook just my main application UI thread using the WH_CALLWNDPROC hook type. From there, I could intercept the three messages WM_INITMENUPOPUP, WM_MEASUREITEM and WM_DRAWITEM, and map them across to my original plug-in owner drawn menu code.

void CODMenu::InstallHook()
{
    m_hookHandle = SetWindowsHookEx(
            WH_CALLWNDPROC, 
            (HOOKPROC)CODMenu::HookFunction, 
            AfxGetResourceHandle(), 
            GetCurrentThreadId());
}

void CODMenu::UninstallHook()
{
    VERIFY(::UnhookWindowsHookEx(m_hookHandle));
}

And I thought "Voila".

But life is not that easy and neither is Windows.

The first problem that was obvious (to me) is that the WH_CALLWNDPROC hook gets called before the actual target window processes the message. If you look at the code, the OnInitMenuPopup() handler looks like this:

void CODMenu::OnInitMenuPopup(HMENU hMenu, UINT nIndex, BOOL bSysMenu)
{
    UNREFERENCED_PARAMETER(nIndex);
    // iterate any menu about to be displayed and make sure
    // all the items have the ownerdrawn style set
    // We receive a WM_INITMENUPOPUP as each menu is displayed, even if the user
    // switches menus or brings up a sub menu. This means we only need to
    // set the style at the current popup level.
    // we also set the user item data to the index into the menu to allow
    // us to measure/draw the item correctly later
    //
    if (hMenu != NULL)
    {
        m_bSysMenu = (bSysMenu != FALSE);
        m_menuBeingProcessed = hMenu; // only valid for measure item calls
        int itemCount = ::GetMenuItemCount(hMenu);
        for (int item = 0; item < itemCount; item++)
        {
            // make sure we do not change the state of the menu items as
            // we set the owner drawn style
            MENUITEMINFO    itemInfo;
        
            memset(&itemInfo, 0, sizeof(MENUITEMINFO));
            itemInfo.cbSize = sizeof(MENUITEMINFO);
        
            itemInfo.fMask = MIIM_STATE | MIIM_TYPE;
            VERIFY(::GetMenuItemInfo(hMenu,
                    item, 
                    TRUE,       // by position
                    &itemInfo));
            int itemID = ::GetMenuItemID(hMenu, item);
            if ((itemInfo.fType & MFT_SEPARATOR) == 0)
            {
                ::ModifyMenu(hMenu,
                        item, 
                        itemInfo.fState | MF_BYPOSITION | MF_OWNERDRAW, 
                        itemID,
                        (LPCTSTR)item);
            }
            else
            {
                ::ModifyMenu(hMenu,
                        item, 
                        MF_BYPOSITION | MF_OWNERDRAW | MF_SEPARATOR, 
                        itemID,
                        (LPCTSTR)item);
            }
        }
    }
}

This code adds the owner draw style MF_OWNERDRAW to every menu item so that we can get the WM_MEASUREITEM and WM_DRAWITEM messages sent to us. Notice also that it makes sure that we do not clear the MF_SEPARATOR style, as doing so would make a separator selectable through the keyboard (but not the mouse). As this happens before the standard MFC code processes the message (due to the hook style), it means that any item that gets added during the processing of the WM_INITMENUPOPUP message will not have the owner drawn style set on it. This would lead to a combination of ownerdrawn and non-ownerdrawn items together in the same menu. This would not look good.

Typically, you see this for a Recent file list in the File menu or Open document list in the Window menu.

It's a simple fix for our problem, we just switch to the WH_CALLWNDPROCRET hook method, as this gets called after the application has completed processing the message.

So that's the first problem sorted. The next was that no matter what I did, the WM_MEASUREITEM handler, although called, would never set up the actual menu item widths/heights, as the WH_CALLWNDPROC/WM_CALLWNDPROCRET hook functions are designed to make sure that you cannot modify the message and any of its parameters. The LPMEASUREITEMSTRUCT pointer I was getting pointed to a different object, which means all my menu items were not getting the correct width/height, so my menu items defaulted to 12 * 16 pixels in size. It did not look good.

Problems, problems

So I went back to the MSDN docs to try and find a hook function that lets me modify the message parameters. The only one that seemed to allow me to do this and get the message types I wanted was the WH_GETMESSAGE hook. So I setup a second application UI thread hook using this one. I then added the correct mapping for WM_MEASUREITEM and WM_DRAWITEM. I was expecting spectacular results. I didn't expect Windows to throw a spanner in the works.

This hook got me just about every message except the two I wanted. Never received, Nada, Ziltch. Sob sob.

At this point, I thought my super method was dead. No matter which hook types I tried, none worked. I resorted to the VC boards and started asking what appeared to be difficult questions. I got a few responses, one of which read like this:

Hooks are generally crap. You are better off subclassing the window like this:

oldWndProc = (WNDPROC)SetWindowLong(hwnd, GWL_WNDPROC, (LONG)NewWndProc);

If I use this method, I would, in effect, subclass the window that owned the menu. I could do this for long enough to handle the measure/draw calls for the menu, and then un-subclass it once the menu was no longer shown. Doing it this way would stop us having to keep handle maps of HWND to old WNDPROC pointers. As you can only have one menu showing at a time in an application, this would not be a problem.

So after a quick bit of work, my hook function looked like this:

LRESULT CALLBACK CODMenu::HookFunction(int code, WPARAM wParam, LPARAM lParam)
{
    CWPRETSTRUCT * cwpretStruct = (CWPRETSTRUCT*)(lParam);
    
    ASSERT(m_activeObject != NULL);
    switch (cwpretStruct->message)
    {
    case WM_INITMENUPOPUP:
        {
            TRACE("WM_INITMENUPOPUP\n");
            m_activeObject->OnInitMenuPopup(
                    (HMENU)cwpretStruct->wParam, 
                    LOWORD(cwpretStruct->lParam), 
                    (BOOL)HIWORD(cwpretStruct->lParam));
            if (m_oldWndProc == NULL)
            {
                // hook the window message queue so
                // we can handle the WM_DRAWITEM/WM_MEASUREITEM messages
                m_oldWndProc = (WNDPROC)SetWindowLong(
                        cwpretStruct->hwnd, 
                        GWL_WNDPROC, 
                        (LONG)MenuWndProc);
                TRACE("Replaced WndProc\n");
            }
            m_activeMenuLayers++;
        }
        break;
    case WM_UNINITMENUPOPUP:
        {
            TRACE("WM_UNINITMENUPOPUP\n");
            m_activeMenuLayers--;
            if (m_activeMenuLayers == 0)
            {
                // restore the old wndProc
                m_oldWndProc = (WNDPROC)SetWindowLong(
                        cwpretStruct->hwnd, 
                        GWL_WNDPROC, (
                        LONG)m_oldWndProc);
                m_oldWndProc = NULL;
                TRACE("Restored WndProc\n");
            }
        }
        break;
    }
    return ::CallNextHookEx(m_hookHandle, code, wParam, lParam);
}

An area of interest is the use of the counter m_activeMenuLayers. We need this as we do not want to end up multi-subclassing a window which just has a menu with many popup layers present. For every extra popup that the user goes to, we just count it. We receive one¹ WM_INITMENUPOPUP message for each popup level shown, and one¹ WM_UNINITMENUPOPUP for each one hidden. We just need to keep track of how many popup layers are currently in use. When the count goes to 0, we can un-subclass the window.

¹ I have since found this to be incorrect. Some menus do not follow this rule, with you receiving two (or more) WM_INITMENUPOPUP messages for the first menu layer to a single WM_UNINITMENUPOPUP message. This would cause menu code to fail to un-subclass the window due to thinking it still had active menu layers being shown. This would cause very bad screen redraw problems and possible crash very badly.

Multiple WM_INITMENUPOPUP messages

To solve this multiple WM_INITMENUPOPUP message receiving problem, I first had to identify a case where this happens every time a menu is displayed. I eventually tracked one down to the standard MFC Save as dialog box. If you clicked on the drop arrow to change the method of filename display in the list control, a small popup menu is displayed. I would receive two WM_INITMENUPOPUP messages for this menu, after which no menu in the application would draw correctly. I also noticed another problem in this dialog box - if you right click a file to get a context menu, Windows provides a shell menu which comes with pre-setup icons in them. This menu would not render correctly at all and generally caused a GPF².

² I will show how I solved this problem in the next section.

My initial look at this multiple message problem did not seem to offer any hope of a work around. The code needs to subclass the window, but there seemed to be no way to tell the difference between a valid WM_INITMENUPOPUP message and an invalid one. As I was using a variation of this code in an application that is due to be released, it was considered very high priority, and if no solution could be found, all the OD menu code would just have been disabled for such a release. This seemed the most likely course of action until I noticed a small detail. The HMENU handle that came through in both of the init messages were different, and they were also not the HMENU of any contained popup menus.

Once I had realized this, I could then setup a test for bogus messages. I just need to build a list of all the valid HMENU handles for the current and sub menus of the menu being displayed, and if it's not in the list, it's a bogus message!

void CODMenu::BuildSubMenuList(HMENU hMenu)
{
    m_subMenus.clear();
    // iterate the menu and get the HMENUs for all popups upder the current one
    AddToList(hMenu);
}

void CODMenu::AddToList(HMENU hMenu)
{
    int itemCount = ::GetMenuItemCount(hMenu);
    for (int item = 0; item < itemCount; item++)
    {
        MENUITEMINFO    itemInfo;    
        memset(&itemInfo, 0, sizeof(MENUITEMINFO));
        itemInfo.cbSize = sizeof(MENUITEMINFO);
    
        itemInfo.fMask = MIIM_SUBMENU;
        ::GetMenuItemInfo(hMenu, item, TRUE, &itemInfo);
        
        if (itemInfo.hSubMenu != NULL)
        {
            // do a recursive call on this popup menu
            m_subMenus.push_back(itemInfo.hSubMenu);
            // recursive call to get any sub menus under this one
            AddToList(itemInfo.hSubMenu);
        }
    }
}

bool CODMenu::SubMenuPresentInList(HMENU hMenu) const
{
    bool found = false;

    for (size_t index = 0 ; index < m_subMenus.size() && !found ; ++index)
    {
        if (m_subMenus[index] == hMenu)
        {
            // its present in the list, ok to go!
            found = true;
        }
    }
    return found;
}

So three new private functions were added to the class, one to build the list, another to recurse any given menu, and one to check whether a given HMENU was present in the list. This allowed the code to correctly check for the bad messages and solved the first of the problems.

Shell file context menus - e.g., a right click in the Save as dialog box

As you can see in the above picture, a shell context menu contains icons supplied by other applications and shell plug-ins. When I first encountered problems with the ODMenu code drawing this menu, I tried to get it to render this menu correctly. There were several issues:

• Separators were drawn as menu items - This was because the menu gave separators non-zero ID numbers. The ODMenu code was checking the ID number and only rendering it as a separator if it was 0. This could be easily fixed by getting the style of the menu item and checking for the MFT_SEPARATOR flag. This would get these to render correctly.
• I could not guarantee that there was an ASCII version of the menu text - The shell runs in UNICODE. The code was using the ANSI version of the ::GetMenuString function and then converting the text to UNICODE internally. This caused us to get corrupted or no text at all for menu items. I converted all the code to use UNICODE internally. This solved 70% of the problems with menu text. Some menu items seem to use callbacks to get menu text, and I was unable to get this, so no text was being rendered. This is especially true for the Send to menu items.
• System supplied icons - The system is supplying the icons beside many of the menu items, also menu IDs would also randomly match those in use in the application and show incorrect icons. I initially tried to render the system supplied icons and I also renumbered my menu IDs so they did not clash with the system menu. The MENUITEMINFO object should be able to return the HBITMAP of the icon to render next to the menu item, but this needs you to compile the code with WINVER defined as 0x0500. Doing so gives you many warnings from the compiler about such code not being allowed to be released to the world at large. Without it, you do not get the HBITMAP you need; with it, you cannot release the code. I could find no way to solve this problem.

Having identified and only partially solved these problems for the shell menu, it was decided internally that the best way to deal with this is to exclude all shell menus from being made ownerdrawn. So a way has to be found to identify these types of menus. The only way I can see this being done, is that you get separators with non-zero menu IDs! So, we just need to scan a menu to see if it has these. If it does, we ignore it and do not make it ownerdrawn.

bool CODMenu::IgnoreThisMenu(HMENU hMenu)
{
    // if we receive a menu which has separators
    // which have a non-zero value, then its
    // a menu we should not make ownerdrawm
    // determine whether this is such a menu
    bool ignoreMenu = false;

    int itemCount = ::GetMenuItemCount(hMenu);
    for (int item = 0; item < itemCount && !ignoreMenu; item++)
    {
        MENUITEMINFO    itemInfo;    
        memset(&itemInfo, 0, sizeof(MENUITEMINFO));
        itemInfo.cbSize = sizeof(MENUITEMINFO);
    
        itemInfo.fMask = MIIM_TYPE | MIIM_ID | MIIM_SUBMENU;
        ::GetMenuItemInfo(hMenu, item, TRUE, &itemInfo);
        
        int itemID = itemInfo.wID;

        if ((itemInfo.fType & MFT_SEPARATOR) != 0 && itemID != 0)
        {
            // this is a menu type we need to ignore
            ignoreMenu = true;
        }
        if (itemInfo.hSubMenu != NULL)
        {
            // do a recursive call on this popup menu
            ignoreMenu = IgnoreThisMenu(itemInfo.hSubMenu);
        }
    }
    return ignoreMenu;
}

With all the changes made to the hook function, it now looks like this:

LRESULT CALLBACK CODMenu::HookFunction(int code, WPARAM wParam, LPARAM lParam)
{
    CWPRETSTRUCT * cwpretStruct = (CWPRETSTRUCT*)(lParam);
    
    ASSERT(m_activeObject != NULL);
    switch (cwpretStruct->message)
    {
    case WM_INITMENUPOPUP:
        {
            if (!m_ignoredMenu)
            {
                m_ignoredMenu = IgnoreThisMenu((HMENU)cwpretStruct->wParam);
                if (!m_ignoredMenu)
                {
                    m_activeObject->OnInitMenuPopup(
                            (HMENU)cwpretStruct->wParam, 
                            LOWORD(cwpretStruct->lParam), 
                            (BOOL)HIWORD(cwpretStruct->lParam));
                    if (m_oldWndProc == NULL)
                    {
                        // hook the window message queue so we can handle
                        // the WM_DRAWITEM/WM_MEASUREITEM messages
                        m_oldWndProc = (WNDPROC)SetWindowLong(
                            cwpretStruct->hwnd, GWL_WNDPROC, (LONG)MenuWndProc);
                        m_activeObject->BuildSubMenuList(
                            (HMENU)cwpretStruct->wParam);
                    }
                }
            }
            // make sure its not a bogus WM_INITMENUPOPUP message
            if (m_activeMenuLayers == 0 || 
                m_activeObject->SubMenuPresentInList((HMENU)cwpretStruct->wParam))
            {
                // count the active layer
                m_activeMenuLayers++;
            }
        }
        break;
    case WM_UNINITMENUPOPUP:
        {
            m_activeMenuLayers--;
            if (m_activeMenuLayers == 0)
            {
                // restore the old wndProc
                if (!m_ignoredMenu)
                {
                    m_oldWndProc = (WNDPROC)SetWindowLong(
                        cwpretStruct->hwnd, GWL_WNDPROC, (LONG)m_oldWndProc);
                    m_oldWndProc = NULL;
                }
                m_ignoredMenu = false;
            }
        }
        break;
    }
    return ::CallNextHookEx(m_hookHandle, code, wParam, lParam);
}

Other considerations

As I want this class to be really easy to use, I do not want the user to have to worry about loading in all the toolbar resources used to get the correct images for the owner drawn menu. I needed to get the code to go get this itself. Now, I do not really know what the target application environment is going to be like. I had to make the following assumptions:

• The application may be made up of an EXE and additional DLLs. (This is like my work project.)
• All the DLLs will already have been loaded by the time we need to enumerate the toolbars.

So with these two requirements in mind, I set off to develop a helper class which will get me a list of all the loaded modules for the current application. This turns out to be fairly easy, as I can make use of standard system functionality provided by the PSAPI.DLL. There may be one issue with this in that it may not be supported on older OS versions. In such cases, there is a different API available, something like TOOLWIN32 (I forgot the actual name).

But for my purposes, PSAPI will do the job.

class EnumerateLoadedModules  
{
public:
    EnumerateLoadedModules();
    ~EnumerateLoadedModules();

    int Count() const;
    HMODULE GetModuleHandle(int index) const;
    CString GetModuleFilename(int index) const;

private:
    void Enumerate();
    std::vector<HMODULE>    m_loadedModuleHandles;
    std::vector<CString>    m_loadedModuleFilenames;
};

// EnumerateLoadedModules.cpp: implementation
//            of the EnumerateLoadedModules class.
//
/////////////////////////////////////////////////////////

#include "stdafx.h"
#include "EnumerateLoadedModules.h"
#include <Psapi.h>

/////////////////////////////////////////////////////////
// Construction/Destruction
/////////////////////////////////////////////////////////

EnumerateLoadedModules::EnumerateLoadedModules()
{
    Enumerate();
}

EnumerateLoadedModules::~EnumerateLoadedModules()
{

}

void EnumerateLoadedModules::Enumerate()
{
    DWORD currentProcessId = GetCurrentProcessId();
    HANDLE hProcess = OpenProcess(
            PROCESS_QUERY_INFORMATION | PROCESS_VM_READ,
            FALSE,
            currentProcessId);
    
    m_loadedModuleHandles.clear();
    m_loadedModuleFilenames.clear();

    if (hProcess)
    {
        HMODULE hMod;
        DWORD cbNeeded;
        if (EnumProcessModules( hProcess, &hMod, sizeof(hMod), &cbNeeded))
        {
            int numModules = cbNeeded / sizeof(HMODULE);
            if (numModules > 0)
            {
                HMODULE * modules = new HMODULE[numModules];
                EnumProcessModules(
                        hProcess, 
                        modules, 
                        sizeof(HMODULE) * numModules, 
                        &cbNeeded);
                for (int moduleIndex = 0 ; moduleIndex < numModules ; ++moduleIndex)
                {
                    char moduleFilename[MAX_PATH];

                    m_loadedModuleHandles.push_back(modules[moduleIndex]);
                    if (GetModuleFileNameEx(
                            hProcess, 
                            modules[moduleIndex], 
                            moduleFilename, 
                            MAX_PATH))
                    {
                        m_loadedModuleFilenames.push_back(moduleFilename);
                        TRACE("%s\n", moduleFilename);
                    }
                }
                delete []modules;
            }
        }
        CloseHandle(hProcess);
    }
}

int EnumerateLoadedModules::Count() const
{
    return m_loadedModuleHandles.size();
}

HMODULE EnumerateLoadedModules::GetModuleHandle(int index) const
{
    ASSERT(index >= 0 && index < m_loadedModuleHandles.size());
    return m_loadedModuleHandles[index];
}

CString EnumerateLoadedModules::GetModuleFilename(int index) const
{
    ASSERT(index >= 0 && index < m_loadedModuleHandles.size());
    return m_loadedModuleFilenames[index];
}

It's a small helper class, which the ODMenu code makes use of in its constructor. Once we have a list of the modules, we can enumerate the individual toolbars in that module and load them if they are the correct size.

void CODMenu::EnumerateAndLoadToolbars()
{
    // load all the toolbars from all loaded modules (exe files/dll files)
    EnumerateLoadedModules modules;

    for (int moduleIndex = 0 ; moduleIndex < modules.Count() ; ++moduleIndex)
    {
        TRACE("Enumerating file %s\n", modules.GetModuleFilename(moduleIndex));
        EnumResourceNames(
                modules.GetModuleHandle(moduleIndex), 
                RT_TOOLBAR,
                (ENUMRESNAMEPROC)EnumResNameProc, 
                0);
    }

    // we now have all the toolbars loaded into the main image list m_buttonImages
    // generate the disabled versions of the images used
    CBitmap disabledImage;
    CWindowDC dc(NULL);

    dc.SaveDC();
    disabledImage.CreateCompatibleBitmap(&dc, m_iconX, m_iconY);
    dc.SelectObject(&disabledImage);
    for(int image = 0 ; image < m_buttonImages.GetImageCount() ; image++)
    {
        CBitmap bmp;
        GetBitmapFromImageList(&dc, &m_buttonImages, image, bmp);
        DitherBlt3(&dc, bmp, ::GetSysColor(COLOR_3DFACE));
        m_disabledImages.Add(&bmp, ::GetSysColor(COLOR_3DFACE));
    }
    dc.RestoreDC(-1);
}

So with this done, we have all the information we need.

TPM_NONOTIFY style

While doing some further testing of the class, I noticed that when a menu is displayed using the TPM_NONOTIFY style, we do not get the WM_INITMENUPOPUP or the WM_UNINITMENUPOPUP messages. This means that any context menu used by any controls, such as edit controls, can not have its menus made ownerdrawn. This gives the interface an inconsistent feel to the user. So I set out to resolve this.

The first attempt was using the window hook method already in use. We could trap the WM_CREATE message for the window class #32768 which is the standard class name used for most menu types (but not all). The problem with this approach is that you have a HWND and not a HMENU handle. I tried to find a way to go from the menu window handle to its menu handle, but there was no way I could traverse from one to the other. Question in forums and Google searches returned deafening silences. I did find one series of posts on the subject, but that had not resolved the issue either. I gave up this method as a dead end.

So, how was I to proceed? The only thing that occurred to me was that the HMENU handle will be available on any call to TrackPopupMenu/TrackPopupMenuEx. So, could I intercept these OS calls and effectively grab the HMENU handle there? I thought it might work.

Intercepting the OS

Well, a quick search of the Internet came up with a very suitable technique which would allow me to intercept the OS call, do my extra work to the menu, and pass the call back on to the OS function. This intercept method uses what is called a trampoline technique. It works by finding the target function in memory, changing the access rights to that area of memory so we can modify the code. Horror - Self modifying code! We then copy the first five (or more, depending on the function) bytes into the trampoline function and insert a relative jump instruction, which makes the start of the OS code jump to our function.

As long as our intercept function has the same calling conventions and parameter list as the OS function, we can do what we want and then call the trampoline function, which in effect runs those first five instruction bytes and jumps back to the OS code for it to continue.

Compiler problems

This technique looked like it should work, but it failed every time I intercepted the OS call. After looking at the assembler source for my functions which were being generated, it looked like the compiler was generating incorrect code. If you took the address of the trampoline function and looked at the assembly source, it started with a relative jump to the real function! Why? It's the actual function start address, why have this extra jump stage?

Well, this means I had to translate my trampoline target address to the target of the real jump instruction. The problem was caused by the trampoline function having to be a minimum of 10 bytes in length. This is for the five original (or more) instruction bytes of the OS function being intercepted and the relative jump instruction tacked to the end to jump back to the real OS code (five bytes). As the compiler was giving me "bogus" function starts (five bytes in length) which jumped to the real function, if I did the trampoline copy to that location, I would end up trashing the following five bytes for whatever function that was.

With this translate code added to the InterceptAPI function, we use this code to intercept the OS calls:

void CODMenu::InstallHook()
{
    m_hookHandle = SetWindowsHookEx(
        WH_CALLWNDPROCRET,
        (HOOKPROC)CODMenu::HookFunction,
        AfxGetResourceHandle(),
        GetCurrentThreadId());
    InterceptAPI(
        AfxGetResourceHandle(),
        "User32.dll",                            // module function is in
        "TrackPopupMenu",                        // function to intercept
        (DWORD)CODMenu::TrackPopupMenu,          // our intercept function
        (DWORD)CODMenu::TrampolineTrackPopupMenu,// trampoline function
        5);                                      // target bytes to copy
    InterceptAPI(
        AfxGetResourceHandle(), 
        "User32.dll", 
        "TrackPopupMenuEx", 
        (DWORD)CODMenu::TrackPopupMenuEx, 
        (DWORD)CODMenu::TrampolineTrackPopupMenuEx, 
        5);
}

BOOL CODMenu::InterceptAPI(
        HMODULE hLocalModule, 
        const char* c_szDllName, 
        const char* c_szApiName, 
        DWORD dwReplaced, 
        DWORD dwTrampoline, 
        int offset)
{
    int i;
    DWORD dwOldProtect;
    DWORD dwAddressToIntercept = (DWORD)GetProcAddress(
            GetModuleHandle((char*)c_szDllName), 
            (char*)c_szApiName);

    BYTE *pbTargetCode = (BYTE *) dwAddressToIntercept;
    BYTE *pbReplaced = (BYTE *) dwReplaced;
    BYTE *pbTrampoline = (BYTE *) dwTrampoline;

    // Change the protection of the trampoline region
    // so that we can overwrite the first 5 + offset bytes.
    if (*pbTrampoline == 0xe9)
    {
        // target function starts with an relative jump
        // change trampoline to the target of the jump
        pbTrampoline++;
        int * pbOffset = (int*)pbTrampoline;
        pbTrampoline += *pbOffset + 4;
    }
    VirtualProtect((void *) pbTrampoline, 5+offset, PAGE_WRITECOPY, &dwOldProtect);
    for (i=0;i<offset;i++)
        *pbTrampoline++ = *pbTargetCode++;
    pbTargetCode = (BYTE *) dwAddressToIntercept;

    // Insert unconditional jump in the trampoline.
    *pbTrampoline++ = 0xE9;        // jump rel32
    *((signed int *)(pbTrampoline)) = (pbTargetCode+offset) - (pbTrampoline + 4);
    VirtualProtect((void *) dwTrampoline, 5+offset, PAGE_EXECUTE, &dwOldProtect);
    
    // Overwrite the first 5 bytes of the target function
    VirtualProtect((void *) dwAddressToIntercept, 5, PAGE_WRITECOPY, &dwOldProtect);

    // check to see whether we need to translate the pbReplaced pointer
    if (*pbReplaced == 0xe9)
    {
        // target function starts with an relative jump
        // change to target of the jump
        pbReplaced++;
        int * pbOffset = (int*)pbReplaced;
        pbReplaced += *pbOffset + 4;
    }
    *pbTargetCode++ = 0xE9;        // jump rel32
    *((signed int *)(pbTargetCode)) = pbReplaced - (pbTargetCode +4);
    VirtualProtect((void *) dwAddressToIntercept, 5, PAGE_EXECUTE, &dwOldProtect);
    
    // Flush the instruction cache to make sure 
    // the modified code is executed.
    FlushInstructionCache(GetCurrentProcess(), NULL, NULL);
    return TRUE;
}

So, I only needed the code that was to be added to my new intercept functions:

BOOL WINAPI CODMenu::TrackPopupMenu(
        HMENU hMenu,         // handle to shortcut menu
        UINT uFlags,         // options
        int x,               // horizontal position
        int y,               // vertical position
        int nReserved,       // reserved, must be zero
        HWND hWnd,           // handle to owner window
        CONST RECT *prcRect  // ignored
)
{
    bool hooked = false;
    if (uFlags & TPM_NONOTIFY)
    {
        m_activeObject->m_menuBeingProcessed = hMenu;
        CMenu menu;
        menu.Attach(hMenu);
        m_activeObject->OnInitMenuPopup(&menu, 0, FALSE);
        menu.Detach();
        // hook the window message queue
        // so we can handle the WM_DRAWITEM/WM_MEASUREITEM messages
        m_oldWndProc = (WNDPROC)SetWindowLong(hWnd, 
                                GWL_WNDPROC, (LONG)MenuWndProc);
        hooked = true;
    }
    BOOL ret = TrampolineTrackPopupMenu(hMenu, uFlags, 
                      x, y, nReserved, hWnd, prcRect);
    if (hooked)
    {
        // restore the old wndProc
        m_oldWndProc = (WNDPROC)SetWindowLong(hWnd, 
                                GWL_WNDPROC, (LONG)m_oldWndProc);
        m_oldWndProc = NULL;
    }
    return ret;
}

BOOL WINAPI CODMenu::TrackPopupMenuEx(
        HMENU hMenu,       // handle to shortcut menu
        UINT fuFlags,      // options
        int x,             // horizontal position
        int y,             // vertical position
        HWND hwnd,         // handle to window
        LPTPMPARAMS lptpm  // area not to overlap
)
{
    bool hooked = false;
    if (fuFlags & TPM_NONOTIFY)
    {
        m_activeObject->m_menuBeingProcessed = hMenu;
        CMenu menu;
        menu.Attach(hMenu);
        m_activeObject->OnInitMenuPopup(&menu, 0, FALSE);
        menu.Detach();
        // hook the window message queue so we can handle
        // the WM_DRAWITEM/WM_MEASUREITEM messages
        m_oldWndProc = (WNDPROC)SetWindowLong(hwnd, 
                        GWL_WNDPROC, (LONG)MenuWndProc);
        hooked = true;
    }
    BOOL ret = TrampolineTrackPopupMenuEx(hMenu, fuFlags, x, y, hwnd, lptpm);
    if (hooked)
    {
        // restore the old wndProc
        m_oldWndProc = (WNDPROC)SetWindowLong(hwnd, 
                                GWL_WNDPROC, (LONG)m_oldWndProc);
        m_oldWndProc = NULL;
    }
    return ret;
}

And it worked! The context menus for controls such as CEdit were made ownerdrawn. This can be seen by running the demo application and using the context menu for the edit control in the About box.

History

V1.2 1^st February 2005

• Icon transparency issue depending on OS version.
• Bogus WM_INITMENUPOPUP messages.
• Shell context menus.
• Possible incorrect processing of draw/measure messages which were not targeted at menus.
• General code cleanup and rationalization.
• Now only supports XP drawing style - you will have to add in other drawing styles yourself (sorry).
• Image index problems on duplicate button IDs.

V1.1 29^th November 2004

• Possible crash when calling target window's real WndProc function fixed.
• Upgraded to allow context menus to be intercepted and made owner drawn.
• Fixed the missing ClientToScreen function in the About box (Thanks Dido2k).
• Fixed the separators being selectable through the keyboard bug (Thanks to David Simmonds).

V1.0 2^nd November 2004

• The initial release of the code.