SetWindowsHookEx fails inside MFC DLL code.

Question

5.00/5 (1 vote)

See more:

In the MFC code for Visual Studio 6.0 there is a function

C++

void AFXAPI AfxHookWindowCreate(CWnd* pWnd)

and it installs a windows hook to catch window creation.

C++

if (pThreadState->m_hHookOldCbtFilter == NULL)
{
    pThreadState->m_hHookOldCbtFilter = ::SetWindowsHookEx(WH_CBT,
        _AfxCbtFilterHook, NULL, ::GetCurrentThreadId());
    if (pThreadState->m_hHookOldCbtFilter == NULL)
        AfxThrowMemoryException();
}

Regrettably, even the debug version of the code throws the memory exception instead of printing out the last error code.

This is happening so close to application startup, and cannott be replicated as a problem if the application is being debugged. The circumstances to make it happen are so bizarre, that I really think just knowing the error code could help here.

Has anyone else run into the problem with this failing?

Any ideas on how to 'intercept' the hook installation and maybe find out what went wrong?

Posted 22-Jul-11 11:30am

Blake Miller

Updated 25-Apr-16 5:45am

Richard MacCutchan

v2

Add a Solution

3 solutions

Solution 1

According to the documentation[^] AfxThrowMemoryException() occurs when you have run out of memory, so it looks like you have a bug somewhere that is consuming all your system resources.

Posted 22-Jul-11 23:58pm

Richard MacCutchan

Solution 3

Thanks Roger Allen.

I used this:
HookImportFunctionByName v1.0[^]

and the trick was to set the module not to the EXE but to the module handle for MFC42?.DLL (obtained with GetModuleHandle() call) and to hook the function "SetWindowsHookExA" since I am building ANSI.

I can see the calls in the call stack to the interceptor function.

S_MySetWindowsHookEx(int 5, long (int, unsigned int, long)* 0x5f42f417 _AfxCbtFilterHook(int, unsigned int, long), HINSTANCE__ * 0x00000000, unsigned long 5204) line 140
AfxHookWindowCreate(CWnd * 0x003c7c30 {CMainFrame hWnd=0x00000000}) line 633 + 22 bytes
CWnd::CreateEx(unsigned long 0, const char * 0x001567d8, const char * 0x003c7a0c, unsigned long 13598720, int -2147483648, int -2147483648, int -2147483648, int -2147483648, HWND__ * 0x00000000, HMENU__ * 0x07e00667, void * 0x0012f8e8) line 705
CFrameWnd::Create(const char * 0x001567d8, const char * 0x003c7a0c, unsigned long 13598720, const tagRECT & {top=-2147483648 bottom=0 left=-2147483648 right=0}, CWnd * 0x00000000 {CWnd hWnd=???}, const char * 0x00000080, unsigned long 0, CCreateContext * 0x0012f8e8) line 568 + 79 bytes
CFrameWnd::LoadFrame(unsigned int 128, unsigned long 13598720, CWnd * 0x00000000 {CWnd hWnd=???}, CCreateContext * 0x0012f8e8) line 699 + 45 bytes
CDocTemplate::CreateNewFrame(CDocument * 0x003c6330 {CAlmPrtDoc}, CFrameWnd * 0x00000000 {CFrameWnd hWnd=???}) line 279 + 32 bytes
CSingleDocTemplate::OpenDocumentFile(const char * 0x0012fba8, int 1) line 132 + 17 bytes
CDocManager::OpenDocumentFile(const char * 0x003c624c) line 953
CWinApp::OpenDocumentFile(const char * 0x003c624c) line 93
CWinApp::ProcessShellCommand(CCommandLineInfo & {CCommandLineInfo}) line 40 + 26 bytes

Now I can capture the error code.

Posted 3-Aug-11 11:57am

Blake Miller

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Roger Allen · Accepted Answer · 2011-07-26T06:15:00

Solution 2

You could try using a trapoline function to intercept the call to SetWindowsHookEx

See http://research.microsoft.com/pubs/68568/huntusenixnt99.pdf[^]

Posted 26-Jul-11 6:15am

Roger Allen