Memory(-Leak) and Exception Trace (CRT and COM Leaks)

• Download project (with source) - 26.1 Kb
• Download MFC project (with source) - 48 Kb
• Download MemLeakAnalyse-Tool (with source) - 46.2 Kb

Introduction

With this utility you can simply find memory leaks in your program (CRT and COM-Leaks!). Each leak is displayed with the callstack (including the source line) of the allocation. So, you can easily find leaks, while using the STL. It will also write a file with the callstack if your application crashes (it can also handle stack-overflows!). It almost has no runtime-overhead (runtime-cost). And the best: it is free (GNU Lesser General Public License).

Finding memory leaks

It is easy to implement this in your existing VC code:

1. Add Stackwalker.cpp and Stackwalker.h to your project.
2. Include Stackwalker.h in your main source file.
3. Call InitAllocCheck() right after the beginning of your main.
4. Call DeInitAllocCheck() just before the end of your main (here all the leaks will be reported).

All the leaks will be listed in the file YouAppName.exe.mem.log in the application directory (only in debug builds; it is deactivated for release builds). This will also activate exception-handling by default (release and debug builds).

Only use exception-handling

If you only want to use exception handing, you need to do the following:

1. Add Stackwalker.cpp and Stackwalker.h to your project.
2. Include Stackwalker.h in your main source file.
3. Call OnlyInstallUnhandeldExceptionFilter() right after the beginning of your main.

If an exception occurs, it will write a file with the callstack in the application directory with the name YouAppName.exe.exp.log.

Example

A simple example is given below:

#include <windows.h>

#include "Stackwalker.h"


void main()
{
  // Uncomment the following if you only

  // need the UnhandledException-Filter

  // (to log unhandled exceptions)

  // then you can remove the "(De)InitAllocCheck" lines

  //OnlyInstallUnhandeldExceptionFilter();


  InitAllocCheck();

  // This shows how the mem-leak function works

  char *pTest1 = new char[100];

  // This shows a COM-Leak

  CoTaskMemAlloc(120);

  // This shows the exception handling

  // and log-file writing for an exception:

  // If you want to try it, please comment it out...

  //char *p = NULL;

  //*p = 'A'; // BANG!


  DeInitAllocCheck();
}

If you execute this example, you will get a file Appication-Name.exe.mem.log with the following content:

##### Memory Report ########################################
11/07/02 09:43:56

##### Leaks: ###############################################
RequestID:           42, Removed: 0, Size:          100
1: 11/07/02 09:43:56
1: f:\vs70builds\9466\vc\crtbld\crt\src\dbgheap.c(359)
                              +30 bytes (_heap_alloc_dbg)
1: f:\vs70builds\9466\vc\crtbld\crt\src\dbgheap.c(260)
                              +21 bytes (_nh_malloc_dbg)
1: f:\vs70builds\9466\vc\crtbld\crt\src\dbgheap.c(139) +21 bytes (malloc)
1: f:\vs70builds\9466\vc\crtbld\crt\src\newop.cpp(12) +9 bytes (operator new)
1: d:\privat\memory_and_exception_trace\
            memory_and_exception_trace\main.cpp(9) +7 bytes (main)
1: f:\vs70builds\9466\vc\crtbld\crt\src\crt0.c(259)
                               +25 bytes (mainCRTStartup)

**** Number of leaks: 1

##### COM-Leaks: ###############################################
(shortened)
**** Number of leaks: 1

Explanation

Now, I will explain the Memory-Report-File:

RequestID:           42, Removed: 0, Size:          100

This line is the beginning of one leak. If you have more than one leak, then each leak will start with a RequestID.

• RequestID

  For CRT: This is the RequestID which is passed to the AllocHook. This ID clearly identifies an allocation. The CRT just increments this number for each allocation. You can also use this number with the _CrtSetBreakAlloc function.

  For COM: This is the address of the allocated memory.

• Removed

  In a memory leak dump this must always be 0 (false).

• Size

  This is the size of the allocated memory block.

1: f:\vs70builds\9466\vc\crtbld\crt\src\dbgheap.c(359)
                                   +30 bytes (_heap_alloc_dbg)

This is an actual stack entry. The stack is shown from the last function on the top going through each callee until the end of the stack is reached.

• 1:

  This number is incremented for each complete callstack. You can ignore this.

• f:\vs70builds\9466\vc\crtbld\crt\src\dbgheap.c

  The actual filename.

• (359)

  The line number inside the file.

• +30 bytes

  This is the offset from this line in bytes (if a line produces more than one assembler instruction).

• (_heap_alloc_dbg)

  The name of the function.

More options by calling InitAllocCheck

InitAllocCheck has three parameters:

Log-output with more info

You can also get an output with more info about each stack entry. For this you have to call InitAllocCheck with the first parameter set to ACOutput_Advanced. If you execute the following sample you will get a file Appication-Name.exe.mem.log with more info:

#include <windows.h>

#include "Stackwalker.h"


void main()
{
  InitAllocCheck(ACOutput_Advanced);
  // This shows how the mem-leak function works

  char *pTest1 = new char[100];
  DeInitAllocCheck();
}

And here is the (shortened) output:

##### Memory Report ########################################
11/04/02 09:04:04

##### Leaks: ###############################################
RequestID:           45, Removed: 0, Size:          100
1: 11/04/02 09:04:04
// ...
1:   5     main +49 bytes
1:     Decl: main
1:     Line: d:\privat\memory_and_exception_trace\main.cpp(27) +7 bytes
1:     Mod:  Memory_and_Exception_Trace, base: 00400000h

1:   6     mainCRTStartup +363 bytes
1:     Decl: mainCRTStartup
1:     Line: f:\vs70builds\9466\vc\crtbld\crt\src\crt0.c(259) +25 bytes
1:     Mod:  Memory_and_Exception_Trace, base: 00400000h

1:   7     _BaseProcessStart@4 +35 bytes
1:     Decl: _BaseProcessStart@4
1:     Mod:  kernel32, base: 77e40000h

**** Number of leaks: 1
// ...

Explanation

Here, I will explain the Memory-Report-File:

RequestID:           45, Removed: 0, Size:          100

This line is the same as above:

1:   5     main +49 bytes

• 1:

  This number is incremented for each complete callstack. You can ignore this.

• 5

  This is the depth of the callstack. This number is incremented for each stack entry. The stack is shown from the last function on the top (number 0) going through each callee until the end of the stack is reached.

• main +49 bytes

  The number of bytes from the beginning of this function, where the instruction for this callstack is stored.

1:     Decl: main
1:     Line: d:\privat\memory_and_exception_trace\main.cpp(27) +7 bytes
1:     Mod:  Memory_and_Exception_Trace, base: 00400000h

• Decl: main

  This is the declaration of the function.

• Line: ....xyz.cpp(27) +7 bytes

  This shows the actual line (in brackets) of the callstack (here: line 27). In addition, it gives the offset from this line in bytes (if a line produces more than one assembler instruction).

• Mod: Memory_and_Exception_Trace

  The name of the module (EXE, DLL, OCX, a.s.o.).

• base: 00400000h

  The base address of this module.

XML output

If you set the first parameter to ACOutput_XML an XML file will be produced. It has the following contents:

<MEMREPORT date="11/08/02" time="10:43:47">
  <LEAK requestID="47" size="100">
    <!-- shortened -->
    <STACKENTRY decl="main" decl_offset="+100"
                    srcfile="d:\...\main.cpp" line="16"
      line_offset="+7" module="Memory_and_Exception_Trace" base="00400000"/>
    <STACKENTRY decl="mainCRTStartup" decl_offset="+363"
                    srcfile="f:\...\crt0.c" line="259"
      line_offset="+25" module="Memory_and_Exception_Trace" base="00400000"/>
  </LEAK>
</MEMREPORT>

It is pretty self explaining if you take a look at the "advanced log output".

Mem-leak-analyse tool

If you are using the XML-output format then you can use my MemLeakTool to display the leaks in a sorted order (sorted by callstack). Just select the "xml-leak"-File and press "Read". The callstack will be displayed in a TreeView. If you select a node, the source code will be shown in the right part (if it could be found).

Information: This program requires .NET Framework 1.0!

A word on leaks

You should be aware, that some leaks might be the result of other leaks. For example the following code throws two leaks, but if you remove the "originator" of the leaks, the other leak will also disappear. For example:

#include <windows.h>

#include <stdlib.h>

#include "stackwalker.h"

class MyTest
{
  public:
    MyTest(const char *szName)
    {
      // The following is the second resulting leak

      m_pszName = strdup(szName);
    }
    ~MyTest()
    {
      if (m_pszName != NULL)
        free(m_pszName);
      m_pszName = NULL;
    }
  protected:
    char *m_pszName;
};

void main()
{
  InitAllocCheck();

  // This is the "main" leak

  MyTest *pTest = new MyTest("This is an example");

  DeInitAllocCheck();
}

How it works (CRT)

The basis of the memory leak logger is a Hashtable with information about all the allocated memory (including callstack). Basically _CrtSetAllocHook is called to hook all the memory allocations / frees. Therefore only C/C++ allocations are logged. On every allocation a portion of the callstack and the Instruction-Pointer is caught and stored in the Hashtable, with some other information about the allocation.

If the application calls DeinitAllocCheck then the Hashtable will be iterated and the (saved) callstack of all the entries will be listed in the file. For this we provide a pointer to our ProcessMemoryRoutine function to the StackWalk function.

In detail

Hashtable

The Hashtable contains by default 1024 entries. You can change this value if you are doing many allocations and want to reduce the collisions. Just change the ALLOC_HASH_ENTRIES define.

As hash-key, the lRequestID for each allocation is used. This ID is passed to the AllocHook function (at least for allocs). If it is not passed (for example, for freeing), then an (valid) address is passed. By having this address it is also possible to get the lRequestID, by looking into the _CrtMemBlockHeader of the allocated block.

For hashing, a very simple and fast hash-function is used:

static inline ULONG AllocHashFunction(long lRequestID) {
  return lRequestID % ALLOC_HASH_ENTRIES;
}  // AllocHashFunction

Insert an allocation into the Hashtable

If a new allocation should be inserted into the Hashtable, first a thread context for the actual thread is made by calling GetThreadContext. This function requires a "real" thread handle and not a pseudo handle which is returned by GetCurrentThred. So, for this I have to create a "real" handle by calling DuplicateHandle.

Actually, I only need the current Ebp and Eip registers. This could also be done by just reading the registers with the inline assembler. Now having the registers, I read the memory at the specified address. For Eip, I only need to read 4 bytes. I do not know why StackWalk needs to read the Eip values, but if the values could not be read from StackWalk it fails to build the callstack. The real important part is the callstack which is stored in the memory pointing from Ebp (or Esp).

At the moment, I just try to read 0x500 bytes by calling the ReadProcessMemory function. I don't read the complete stack, because it might use too much memory for many allocations. So, I reduced the maximum size to 0x500. If you need a deeper callstack, you can change the MAX_ESP_LEN_BUF define.

If the callstack is not 0x500 bytes deep, then the ReadProcessMemory will fail with ERROR_PARTIAL_COPY. If this happens, I need to ask how many are possible, to read without any error. For this I need to query this value by calling VirtualQuery. Then, I try to read as many bytes as possible.

Having the callstack I can simply insert the entry into the Hashtable. If the given hash-entry is already occupied, I make a linked list and append this entry to the end.

Building the leak-list

If you call DeInitAllocCheck I simply walk through the Hashtable and output every entry which was not freed. For this I call StackWalk with a pointer to my own memory-reading-function (ReadProcMemoryFromHash). This function is called from the internals of StackWalk. If it is called it looks up the Hashtable for the given lRequestID and returns the memory which was stored in the Hashtable. The lRequestID is passed in the hProcess parameter of the StackWalk function (as stated in the documentation of StackWalk).

Ignoring allocations

Allocations/frees for _CRT_BLOCK are ignored (for more info, see here). This is because CRT dynamically allocates some memory for "special purposes". The tool also checks the _CRTDBG_ALLOC_MEM_DF flag of the _crtDbgFlag variable. If it is off, then all the allocations are ignored. For more details see _CrtSetDbgFlag.

How it works (COM)

To track COM-memory-leaks, you have to provide an IMallocSpy interface. This interface must be registered with CoRegisterMallocSpy. After that the (own) IMallocSpy instance is called for every memory (re)allocation/free. So, you can track all memory actions.

The storage of the callstack is done in the same way as for CRT-allocs (in a Hashtable). So, for more info please read the CRT-section.

A word on COM-leaks

Actually, there is nothing to say, but...

If you are using MSXML 3 or 4 implementation, you have to be aware of the fact that this parser uses a "smart" pseudo-garbage collector. This means that they allocate memory and will not free it after it is used! So, you may see some leaks which are only "cached memory". If you first call CoUninitialize, then all the cached memory will be freed and the "real" COM-leaks will be reported.

For more info see: Understanding the MSXML Garbage Collection Mechanism.

MFC usage

The problem with MFC is that the derived CWinApp class is instantiated by the C-runtime, because it is a global variable. The easiest solution to implement the Leak-Finder is to declare the following static struct inside your MainApp.cpp.

You also have to add stackwalk.cpp and stackwalk.h to your project. You also need to add the #include <stdafx.h> on top of the stackwalk.cpp file (if you use precompiled headers). A sample of an MFC application is also available from the top of this article:

static struct _test
{
  _test()
  {
    InitAllocCheck();
  }

  ~_test()
  {
    DeInitAllocCheck();
  }
} _myLeakFinder;

Temporarily disable logging (only CRT)

When you don't want to log a special allocation of your application (for whatever reason; MFC does this often), then you can simply deactivate it by disabling the CRT flag _CRTDBG_ALLOC_MEM_DF with the _CrtSetDbgFlag function. Here is an example of how you can do this:

#include "Stackwalker.h"


#include <crtdbg.h>


bool EnableMemoryTracking(bool bTrack)
{
  int nOldState = _CrtSetDbgFlag(_CRTDBG_REPORT_FLAG);
  if (bTrack)
    _CrtSetDbgFlag(nOldState | _CRTDBG_ALLOC_MEM_DF);
  else
    _CrtSetDbgFlag(nOldState & ~_CRTDBG_ALLOC_MEM_DF);
  return nOldState & _CRTDBG_ALLOC_MEM_DF;
}

void main()
{
  InitAllocCheck();

  // The following will be logged

  char *pTest1 = new char[100];

  EnableMemoryTracking(false);  // disable logging

  // The following will NOT be logged

  char *pTest2 = new char[200];
  EnableMemoryTracking(true);  // enable logging


  // The following will be logged

  char *pTest3 = new char[300];

  DeInitAllocCheck();
}

Unhandled exceptions

There are three ways to use this tool for unhandled exceptions.

Simple using

If you just call InitAllocCheck with no parameters or with the second parameter set to TRUE, then an unhandled exception filter will be installed. If an unhandled exception occurs, a log file with the callstack will be written, a dialog box with the exception message will be displayed, and the application will be terminated with FatalAppExit.

Second simple using

If you don't want the AllocCheck-overhead (the (small) overhead is only present in debug builds), you can simply call OnlyInstallUnhandeldExceptionFilter. This will install the UnhandledExceptionFilter which writes a log file if an (unhandled) exception occurs. The log file will be stored in the application directory with the name YouAppName.exe.exp.log:

int main()
{
  OnlyInstallUnhandeldExceptionFilter();

  // do your main code here...

}

Advanced using

You can write your own exception filter and just call StackwalkFilter to produce the callstack. Then you can do whatever you want. Here is a small example:

static LONG __stdcall MyUnhandlerExceptionFilter(EXCEPTION_POINTERS* pExPtrs)
{
   LONG lRet;
   lRet = StackwalkFilter(pExPtrs,
             EXCEPTION_EXECUTE_HANDLER, _T("\\exception.log"));
   TCHAR lString[500];
   _stprintf(lString,
      _T("*** Unhandled Exception!\n")
      _T("   ExpCode: 0x%8.8X\n")
      _T("   ExpFlags: %d\n")
      _T("   ExpAddress: 0x%8.8X\n")
      _T("   Please report!"),
      pExPtrs->ExceptionRecord->ExceptionCode,
      pExPtrs->ExceptionRecord->ExceptionFlags,
      pExPtrs->ExceptionRecord->ExceptionAddress);
   FatalAppExit(-1,lString);
   return lRet;
}

int main()
{
  InitAllocCheck(ACOutput_Advanced, FALSE);
  SetUnhandledExceptionFilter(MyUnhandlerExceptionFilter);

  // do some stuff...


  DeInitAlloocCheck();
}

Common mistakes

One of the most common mistakes while using this tool is that you statically instantiate classes in your main function. The problem is that the destructor of the class is called after the call to DeInitAllocCheck. If some memory was allocated inside this class, this memory will appear as a leak. For example:

#include <windows.h>

#include "Stackwalker.h"

#include <string>


void main()
{
  InitAllocCheck();
  std::string szTemp;
  szTemp = "This is a really long string";
  DeInitAllocCheck();
}

There are two solutions for this. You can start a block after the call to InitAllocCheck and end it before the call to DeInitAllocCheck. With this you can be sure that the destructors are called before the leak file is produced. For example:

#include <windows.h>

#include "Stackwalker.h"

#include <string>


void main()
{
  InitAllocCheck();
  {
    std::string szTemp;
    szTemp = "This is a really long string";
  }
  DeInitAllocCheck();
}

The second solution is to use the same technique that is used for MFC applications (see above).

Visual Studio 7 and Win2K / NT

I found a problem with the executables built with VS7 and run on Win2K or NT. The problem is due to an old version of dbghelp.dll. The PDB files generated from VS7 are in a newer format (DIA). It appears that the VS installations do not update dbghelp.dll on Win2K. So the original version (5.0.*) is still on the system and will be used. But with this version it is not possible to read the new PDB format. So, no callstack can be displayed.

To get it to work you have to do the following

Download the latest Debugging Tools for Windows (which includes dbghelp.dll). You have to install it to get the files. But you only need the dbghelp.dll! Now we have another problem. The installer does not replace the original dbghelp.dll. So we need to copy the dbghelp.dll in our EXE dir. Now to make sure the right version is loaded you have to put a file with the name appname.local in your EXE dir (please replace appname with the EXE name (without extension)). Now it should also work on WinNT/2K.

Known issues

• The memory leak works correctly only if the lRequestID does not wrap (32-bit value). If the value wraps around, then it is not possible to clearly assign a given lRequestID to a previous allocation, because it is possible that this ID was used twice (or even more). But this happens only with VC7, because VC6 has a bug in the C-runtime which will call _DbgBreak if the lRequestID wraps (if no _CrtBreakAlloc is used).
• If you compile with "Detect 64-bit portability issues (/Wp64)" option on VC7, it will generate a warning.
• If you use this tool in managed C++ it will not correctly display the callstack for managed code.
• For some reason, the COM-alloc-callstack cannot display the stack-entry which really calls the CoTaskMemAlloc function. Only the upper stack entry can be shown.

References

• Most of the Stackwalker-functions are from Felix Kasza, thanks to him!
• Bugslayer, Feb. 1999: Very good article and source
• Bugslayer, Oct. 1997: Annihilating Bugs in an Application Near You
• Writing Debug Hook Functions
• MS Book: Debugging applications
• How to Use Dh.exe to Troubleshoot User-Mode Memory Leaks
• Debugging Tools for Windows (latest dbghelp.dll)
• dbghelp.dll redistributable for NT
• IMallocSpy-Interface
• Understanding the MSXML Garbage Collection Mechanism
• Umdhtools.exe: How to Use Umdh.exe to Find Memory Leaks.

History

• 4^th Nov, 2002
  • Initial revision.
• 5^th Nov, 2002
  • Updated to work with MFC, "Temporarily disable logging", "MFC usage" and "References" added.
• 6^th Nov, 2002
  • "Explanation" for leak-file, "A word on leaks" and "How it works: In detail" added.
• 7^th Nov, 2002
  • Simple leak output implemented.
  • Explaining the simple leak output.
  • "Common mistakes" added.
• 8^th Nov, 2002
  • Explanation of the InitAllocCheck parameters added.
  • XML-output added, "Soon to come" and "History" added.
• 13^th Nov, 2002
  • Updated the source to compile again under VC6 (SymDIA undefined, will be corrected later).
• 21^st Nov, 2002
  • Updated the source to compile again with UNICODE-builds.
  • #include "stdafx.h" instead of #include <stdafx.h> in MFC-demo.
• 6^th Dec, 2002
  • Updated to work correctly with UNICODE.
  • Added MemLeakAnalyse tool; "Soon to come" removed.
• 19^th Dec, 2002
  • Now using only the dbghelp.dll. NT users have to install the dbghelp.dll redistributable for NT (see references above).
  • Added comments on how to use it from VS7 on NT/W2K...
• 8^th Jan, 2003
  • Major update: Added support for COM-leaks.
  • Small updates to MemLeakAnalyse tool; removed a few small bugs.
• 9^th Jan, 2003
  • Removed the handle-leaks inside the IMallocSpy interface.
• 23^rd Aug, 2003
  • Updated a few links.
• 28^th Aug, 2003
  • Updated to the newer version of dbghelp.dll (StackWalk64).
  • Added OnlyInstallUnhandeldExceptionFilter function; License clarification: LGPL.
• 3^rd Sep, 2003
  • Updated the source to catch "stack-overflows".
  • Removed some "memory leak inside the memory leak finder"; Hashtable-size is now a prime number.
• 12^th Sep, 2003
  • Fixed a bug in PreRealloc; thanks to Christoph Weber.
  • License changed to "zlib/libpng license".
  • AnalyseTool: added command line support (you can specify the XML file in the command line).
  • AnalyseTool: support for additional search paths for looking up the source-file.
• 19^th Oct, 2005
  • Changed GetThreadContext to my own function, because of changes in XP-SP2.
  • Changed the license to LGPL.
• 21^st Nov, 2005
  • Working around a bug in the newest dbghelp.dll (6.5.3.7/8).