Click here to Skip to main content
Click here to Skip to main content

How to use memory dump to trace bug

, 9 Sep 2003
Rate this:
Please Sign up or sign in to vote.
How to use memory dump to trace bug

Introduction

Our software just published version 1.0 . My clients often encountered exceptions and blue screens that caused the program to terminate. Most of time, we can not reproduce the bug in our systems. So catching and analyzing a memory dump file is important. This article teaches you how to use hands-on tools to catch and read memory dump files.

The hands-on tools are :-

  • Dr. Watson, which is included by almost all the windows systems.
  • VC6 and VC7 compiler, we need to recompile the project to generate Map files and COM files for tracing the bug.
  • Dumpchk can check dump file, before you are going to send it.
  • Windbg and kd can give you detailed information about the spot when exception occurred. Symbol files are always necessary for debugging dump files.

Details

Before reproducing the bug, Dr. Watson must be installed and running for srwtsn32.log to be created. We also need to configure the project to generate debug information, MAPFILE and COD file, then recompile the project.

Configuring Dr. Watson

To start Dr. Watson and set it as default debugger, go to command line, type:

drwtsn32 –I 

If you want to change the settings you can run drwtsn32 at command line any time. Default settings is ok. Then if a program crashes, Dr. Watson will write drwtsn32.log and dump memory into user.dmp.

Reconfiguring Project Settings in Visual C++ environment

You can prepare the source code to be ready for Dr. Watson and memory dump. You need to take three simple step as following:

  1. You need to Generate debug symbols for the Win32 Release version of your .EXE and/ or .DLL files in settings configuration for the project, for the linker, Open "Project Settings" window( click Settings item in Project menu), select Link tab, check Generate Mapfile, and check Generate debug info. For the compiler, check Debug Info: Program Database, under General in C/C++ tab.
  2. Creating and keeping a copy of your application's .MAP and .COD intermediate files for further reference. Under Link tab, in Project Options editbox, add switch "/mapinfo:exports" ( switch /mapinfo:exports causes the linker to describe the binary's exported functions.) Add switch "/mapinfo:lines" (The switch /mapinfo:lines will add line numbers in mapfile). A .COD file contains a mixture of source code, assembly code and memory addresses, and it will help you to locate the offending line of code within a function that has been identified via the .MAP file. Under General section in C/C++ tab, add switch "/FAcs" in project options.
  3. Specifying a fixed load base hexadecimal address for your .EXE and/or .DLL files. Under Output in the Link tab, input a hexadecimal address in Base address, such as 0x602f0000

Examples

Use the Visual C++ application wizard to create MFC win32 exe file names "MapDemo". Add following lines in InitInstance function in CMapDemoApp class.

 char * pch1 = 0; 
 char * pch2 = 0; 
 pch1 = (char *)malloc( 20 ); 
 pch2 = (char *)malloc( 20 ); 
 pch1 = pch2; free( pch1 ); 
 free( pch2 ); 

This code will eventually fail because the application will try to release the same block of memory twice. Run the program and it will crash. Open the file drwtsn32.log:

*----> Module List <----*
(0000000000400000 - 000000000040b000: 
    C:\WORK\CODEPROJ\MapDemo\Debug\MapDemo.exe
(0000000010200000 - 000000001026c000: C:\WINDOWS\System32\MSVCRTD.dll
(000000005f400000 - 000000005f4e5000: C:\WINDOWS\System32\MFC42D.DLL
(00000000605d0000 - 00000000605d8000: C:\WINDOWS\System32\mslbui.dll
(0000000070a70000 - 0000000070ad4000: C:\WINDOWS\system32\SHLWAPI.dll
(0000000074720000 - 0000000074764000: C:\WINDOWS\System32\MSCTF.dll
(0000000077120000 - 00000000771ab000: C:\WINDOWS\system32\OLEAUT32.DLL
(00000000771b0000 - 00000000772d1000: C:\WINDOWS\system32\OLE32.DLL
(0000000077c00000 - 0000000077c07000: C:\WINDOWS\system32\VERSION.dll
(0000000077c10000 - 0000000077c63000: C:\WINDOWS\system32\msvcrt.dll
(0000000077c70000 - 0000000077cb0000: C:\WINDOWS\system32\GDI32.dll
(0000000077d40000 - 0000000077dc6000: C:\WINDOWS\system32\USER32.dll
(0000000077dd0000 - 0000000077e5d000: C:\WINDOWS\system32\ADVAPI32.dll
(0000000077e60000 - 0000000077f46000: C:\WINDOWS\system32\kernel32.dll
(0000000077f50000 - 0000000077ff7000: C:\WINDOWS\System32\ntdll.dll
(0000000078000000 - 000000007807f000: C:\WINDOWS\system32\RPCRT4.dll 
*----> State Dump for Thread Id 0x3274 <----* 
eax=00000001 ebx=7ffdf000 ecx=10261558 edx=00010000 
esi=0012febc edi=7ffdf000
eip=10213638 esp=0012fe5c ebp=0012fe6c iopl=0         
nv up ei pl zr na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000            
efl=00000246 
function: MSVCRTD!_free_dbg_lk
        10213616 7521             jnz     MSVCRTD!_free_dbg_lk+0xc9 
                                          (10213639)
        10213618 6814732510       push    0x10257314
        1021361d 6a00             push    0x0
        1021361f 6814040000       push    0x414
        10213624 68f4712510       push    0x102571f4
        10213629 6a02             push    0x2
        1021362b e810180000       call    MSVCRTD!_CrtDbgReport (10214e40)
        10213630 83c414           add     esp,0x14
        10213633 83f801           cmp     eax,0x1
        10213636 7501             jnz     MSVCRTD!_free_dbg_lk+0xc9 
                                           (10213639)
FAULT ->10213638 cc               int     3
        10213639 33c0             xor     eax,eax
        1021363b 85c0             test    eax,eax
        1021363d 75c9             jnz     MSVCRTD!_free_dbg_lk+0x98 
                                           (10213608)
        1021363f 8b4d08           mov     ecx,[ebp+0x8]
        10213642 83e920           sub     ecx,0x20
        10213645 894dfc           mov     [ebp-0x4],ecx
        10213648 8b55fc           mov     edx,[ebp-0x4]
        1021364b 8b4214           mov     eax,[edx+0x14]
        1021364e 25ffff0000       and     eax,0xffff
        10213653 83f804           cmp     eax,0x4 
*----> Stack Back Trace <----*
*** WARNING: Unable to verify checksum for 
  C:\WORK\CODEPROJ\MapDemo\Debug\MapDemo.exe
*** ERROR: Symbol file could not be found.  Defaulted to 
  export symbols for C:\WINDOWS\system32\kernel32.dll - 
WARNING: Stack unwind information not available. Following 
  frames may be wrong.
ChildEBP RetAddr  Args to Child              
0012fe6c 10213541 00324708 00000001 7ffdf000 MSVCRTD!_free_dbg_lk+0xc8
0012fea0 102134ce 00324708 00000001 0012fee8 MSVCRTD!_free_dbg+0x41
0012feb0 00401360 00324708 7ffde000 00324238 MSVCRTD!free+0xe
0012fee8 5f4335c3 7ffdf000 7ffde000 7ffdf000 
    MapDemo!CMapDemoApp__InitInstance+0x132
0012ff08 00402248 00400000 00000000 001423ac MFC42D!AfxWinMain+0x83
0012ff20 00402153 00400000 00000000 001423ac MapDemo!WinMain+0x18
0012ffc0 77e814c7 7ffdf000 7ffde000 7ffdf000 
    MapDemo!WinMainCRTStartup+0x1b3
0012fff0 00000000 00401fa0 00000000 78746341 
    kernel32!GetCurrentDirectoryW+0x44 

Because the code was fault in the memory free function "_free_dbg_lk", it is called by function Initinstance(), so we search the stack-track-back part, on the fourth line, it indicates that the sub function was called by InitInstance function at the offset 0x132.

So, look at the .COD intermediate file, "MapDemo.COD", search for the module "InitInstance()", the offset address at the beginning of the module is 0x00ae, add the address of (0x132+0xae = 0x01e0 ) , so we know the program was crashed at line 79 in source code file "MapDemo.cpp".


Another Example - How to trace fault into DLL file( in – process, and out – process DLL file).

Use Application Wizard in Visual Studio to generate a DLL library project MapDemoDll, create a export function DllSquareRoot(), and put a fault code in that function :-

 char* pChar = NULL;
 *pChar = 'a'; // This is fault line 70 

The program will throw an exception that trying to access to a invalid memory address(0x0000). Use Application Wizard to generate an client program calling the DllSquareRoot function in MapDemoDll.dll. Before running the exe program with the DLL library, we need to set the MapDemoDll’s project’s settings. Generate MAP, COD file, specify the base address for loading at 0x60f20000. Compile the project and run the program.

Program crashed and screen displayed the message like that "DemoDllClient.exe has generated errors and will be closed by windows. You will need to restart the program." Following is the drwsnt32.log generated by Dr. Watson.

State Dump for Thread Id 0x7b0 
eax=00000000 ebx=005e4b68 ecx=00135dd0 edx=60f35918 
esi=0012f27c edi=0012f26c
eip=60f21320 esp=0012f204 ebp=0012f26c iopl=0         
nv up ei pl zr na po nc
cs=001b  ss=0023  ds=0023  es=0023  fs=0038  gs=0000           
efl=00000246 
function: DllSquareRoot
        60f212f9 8d7da4           lea     edi,[ebp+0xa4]         
                ss:00c8c83e=????????
        60f212fc b917000000       mov     ecx,0x17
        60f21301 b8cccccccc       mov     eax,0xcccccccc
        60f21306 f3ab             rep     stosd                  
                es:0012f26c=0012f2cc
        60f21308 e8f3020000       call    AfxGetStaticModuleState 
                (60f21600)
        60f2130d 50               push    eax
        60f2130e 8d4df8           lea     ecx,[ebp+0xf8]         
                ss:00c8c83e=????????
        60f21311 e8a4010000  call 
                AFX_MAINTAIN_STATE2::AFX_MAINTAIN_STATE2 (60f214ba)
        60f21316 c745f400000000   mov   dword ptr [ebp+0xf4],0x0 
                ss:00c8c83e=????????
        60f2131d 8b45f4           mov     eax,[ebp+0xf4]         
                ss:00c8c83e=????????
FAULT ->60f21320 c60078           mov     byte ptr [eax],0x78          
                ds:00000000=??
        60f21323 dd4508          fld qword ptr [ebp+0x8] 
                ss:00c8c83e=????????????????
        60f21326 dc1d4841f360                            
                ds:60f34148=0000000000000000
                                  fcomp   qword ptr [_real (60f34148)]
        60f2132c dfe0             fstsw
        60f2132e f6c401           test    ah,0x1
        60f21331 7520             jnz     _enc$textbss$begin+0x5f43 
               (60f29e53)
        60f21333 8b4d0c           mov     ecx,[ebp+0xc]          
               ss:00c8c83e=????????
        60f21336 51               push    ecx
        60f21337 8b5508           mov     edx,[ebp+0x8]          
               ss:00c8c83e=????????
        60f2133a 52               push    edx
        60f2133b e848060000       call    sqrt (60f21988)
        60f21340 83c408           add     esp,0x8 

The program crashed at address 0x60f21230, here is part of the mapfile

Address Publics by Value Rva+Base Lib:Object 
 0001:00000050 ?_GetBaseMessageMap@CMapDemoDllApp@@KGPBUAFX_MSGMAP@@XZ 
        60f21050 f MapDemoDll.obj
 0001:00000080 ?GetMessageMap@CMapDemoDllApp@@MBEPBUAFX_MSGMAP@@XZ 
        60f21080 f MapDemoDll.obj
 0001:000000c0 ??0CMapDemoDllApp@@QAE@XZ 60f210c0 f MapDemoDll.obj
 0001:00000120 ??_GCMapDemoDllApp@@UAEPAXI@Z 60f21120 f i MapDemoDll.obj
 0001:00000120 ??_ECMapDemoDllApp@@UAEPAXI@Z 60f21120 f i MapDemoDll.obj
 0001:00000190 ??1CMapDemoDllApp@@UAE@XZ 60f21190 f i MapDemoDll.obj
 0001:000002f0 _DllSquareRoot 60f212f0 f MapDemoDll.obj
 0001:000003b2 ?WinHelpA@CWinApp@@UAEXKI@Z 60f213b2 
      f mfc42d:MFC42D.DLL
 0001:000003b8 ?OnDDECommand@CWinApp@@UAEHPAD@Z 60f213b8 
      f mfc42d:MFC42D.DLL
 0001:000003be ?DoWaitCursor@CWinApp@@UAEXH@Z 60f213be 
      f mfc42d:MFC42D.DLL
 0001:000003c4 ?DoMessageBox@CWinApp@@UAEHPBDII@Z 60f213c4 
      f mfc42d:MFC42D.DLL
 0001:000003ca ?SaveAllModified@CWinApp@@UAEHXZ 60f213ca 
      f mfc42d:MFC42D.DLL
 0001:000003d0 ?InitApplication@CWinApp@@UAEHXZ 60f213d0 
      f mfc42d:MFC42D.DLL 

The biggest function address that just less than the crashing address 0x60f21320 is 0x60f212f0, which belongs to DllSquareRoot function in MapDemoDll object. We also can follow that calculation: crash_address - preferred_load_address - 0x1000. Why do we need to subtract 0x1000, because the first part the binary is the Portable Executable (PE), which is 0x1000 bytes long. So the calculation is: 0x60f21320 – 0x60f20000 – 0x1000 = 0x320

Line numbers for .\Debug\MapDemoDll.obj 
(D:\work\MapDemoDll\MapDemoDll.cpp) segment .text 
    44 0001:00000050    44 0001:00000080    
    55 0001:000000c0    58 0001:000000f0
    63 0001:00000220    66 0001:000002f0    
    67 0001:00000308    69 0001:00000316
    70 0001:0000031d    72 0001:00000323    
    73 0001:00000333    74 0001:00000353
    75 0001:00000361    76 0001:0000037a 

Above is the information section of MAPFILE, like that: 44 0001:00000050, first number is line number, the second number is the offset from the beginning of the code section in which this line occurred. So we will know the program was crashed at line 70 in MapDemoDll.cpp. Which tried to write a number into an invalid address.

How to read and debug by memory dump file

  1. Get the symbol table or symbol server address. To prepare for debugging the dump file, you should get the Microsoft symbol server address or download the symbols from Microsoft (http://www.microsoft.com/whdc/ddk/debugging/symbolpkg.mspx#Windows%20symbol%20packages)
  2. Download the debugging tools for Windows. Those debugging tools can read the windows memory dump and simulate the original memory environment.
  3. Need to install the support tools from original installation CD. Before using memory dump to debug, we need to check if it is not corrupted by dumpchk.exe, which is in the windows installation CD. Go to command line and input: Dumpchk -a [dump file]
  4. If the memory dump file passes the check, now we can use windbg or kd to debug the memory dump. After the installation is complete, start a command prompt, change to the path where the debugging tools were installed. Use the one of following command to load the dump file into debugger. Windbg.exe is a windows-based debugger, kd.exe is a command line debugger.
windbg -y SymbolPath -i ImagePath -z DumpFilePath
kd -y SymbolPath -i ImagePath -z DumpFilePath

The SymbolPath is the path of symbols, it is either a local path where the symbol files were downloaded, or the server path which is an URL. ImagePath is the path of windows image files, which are contained in the /I386 folder on the WINDOWS. Installation CD-ROM. DumpPath is the path and file name for the dump that you are examining.

Example

kd -y SRV*f:\symbols*http://msdl.microsoft.com/download/symbols 
   -i e:\i386 -z f:\dumps\minidump.dmp
kd -y f:\symbols -i e:\i386 -z "f:\my debug files\minidump.dmp"
windbg -y f:\symbols -i e:\i386 -z "f:\dumps\minidump.dmp"

Both of the debuggers accept commands which gather information from dump file, such as:

The !analyze -show command displays the Stop error code 
    (also known as the bug check code) and its parameters. 
The !analyze -v command displays verbose output. 
The !drivers command displays a list of the drivers that 
    were loaded on the computer when the problem occurred.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Woodykk
Web Developer
United States United States
I am a shameless software engineer and developer.
work-manic, sleepless animal, a crazy MCP, MCSE, MCDBA and SCJP.

Comments and Discussions

 
GeneralArticle needs some more clarity Pinmemberjomercepj15-Oct-07 2:46 
GeneralTypo in memory address PinmemberSlackerMonkey17-Jan-07 8:13 
Questionhow to create .MAP and .COD intermediate files in VS.NET PinsussAnonymous13-Oct-05 6:04 
GeneralMultithread app PinmemberDalcio Tito16-Jun-04 6:16 
GeneralRe: Multithread app PinmemberArunKumar Yalamarthy20-Jun-05 5:47 
GeneralAnother similar article PinsussAnonymous15-Mar-04 22:36 
GeneralCorporate Error Reporting PinmemberAnthonyJ21-Sep-03 4:25 
GeneralGood work! Pinmembersrinivas vaithianathan16-Sep-03 19:59 
Generaldebugging skills Pinmemberumeca7415-Sep-03 21:01 
GeneralRe: debugging skills Pinsussbseager16-Sep-03 11:43 
GeneralNice ... Fix Formatting PinmemberOkeno Palmer10-Sep-03 3:47 
GeneralRe: Nice ... Fix Formatting PineditorNishant S10-Sep-03 4:38 
GeneralRe: Nice ... Fix Formatting PinmemberOkeno Palmer10-Sep-03 7:09 
GeneralRe: Nice ... Fix Formatting PinmemberA. Riazi11-Sep-03 0:06 
GeneralRe: Nice ... Fix Formatting Pinmembermoomoo10-Sep-03 4:50 
GeneralGood Job! PinmemberKevinHall10-Sep-03 3:41 
General! PinmemberA. Riazi10-Sep-03 2:06 
GeneralRe: ! PineditorNishant S10-Sep-03 4:36 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web03 | 2.8.140721.1 | Last Updated 10 Sep 2003
Article Copyright 2003 by Woodykk
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid