Get Started: Debugging Memory Related Issues in .Net Application Using WinDBG and SOS

1. Introduction

What Do I Mean By Bad Shaped Application?

Why WinDBG + SOS?

2. Things to Understand Before Using WinDBG and SOS

What is WinDBG?

What WinDBG Can Do?

What is SOS.dll (Son of Strike)?

Debugging Essentials

How to Configure WinDBG?

WinDBG and SOS Commands

Executing Batch Commands

CPU Registers

3. Capturing Dumps

4. Debugging with WinDBG and SOS

Debugging Live Targets

Debugging Dump files (Postmortem Debugging)

5. Examples

6. Points of Interest

7. History

1. Introduction

Visual studio inbuilt debugger is a well known debugger, which offers both managed and mixed debugging options but at times when we need to debug memory related issues such as to optimize application performance, solve crashes or out of memory exceptions, then we really need to have a more powerful debugger, that can give detailed insight of objects on heap and GC.

There are number of profiling tools available in the market to analyze hangs or crashes like .Net memory profiler and Ants profiler; I have mostly heard of these two. These tools might fail and become helpless in situations where application is not in a shape (in bad shape) to be profiled.

What Do I Mean By Bad Shaped Application?

For e.g. Suppose an application has so many objects containing self-references resulting in the value of performance counter “.NET CLR Memory->% Time spent in GC” while profiling reaching up to 80% while it is running normally without being profiled. The application could probably crash and cause to hang the profiler tool itself.

I had faced the similar situation while trying to analyze cause of OOM exception in my win form application a few days back. Then, I chose to have WinDBG + SOS as my .Net profiler. Why??? This would be answered itself while we reach the end of this article.

Why WinDBG + SOS?

At first glance, WinDBG + SOS seemed right choice to me because

1. It is freely available from Microsoft. :)

2. It provides treatment to cancer patients even at their last stage; I mean it helps diagnosing a bad shaped application with ease.

3. It is a perfect blend, which allows Native and Managed debugging both at the same time.

I have had heard of WinDBG quite a few times but never used it until my application would not launch via different profiling tools, I am aware of; while trying to diagnose OOM exception. I found much stuff scattered all over the internet about using WinDBG but it was difficult to bring all useful pieces together to make good understanding on how to actually debug an application using this tool, before I could start using it.

So, I thought to write it down at one place to make things simple for others who just decided to start with WinDBG and unaware of its usage & potential.

2. Things to Understand Before Using WinDBG and SOS

What is WinDBG?

WinDBG is a multipurpose debugger for Microsoft Windows, distributed on the web by Microsoft. It can be used to debug user mode applications and kernel mode such as drivers and the operating system as well. We will be dealing with debugging user mode applications in this article.

See this figure to have a quick look at WinDBG.

What WinDBG Can Do?

1. WinDBG is used to debug crash dumps or mini dumps. Dumps are generated when application crashes/hangs or memory shoots up and used for postmortem debugging. Debugging Essentials section explains dumps in more detail.

2. WinDBG can also be used for debugging live targets, usually on developers’ machine.

3. WinDBG debug engine is part of the windows operating system; it supports debugging native code but its extensions allow managed debugging as well.

4. WinDBG is very extensible, as it supports loading external libraries (We will be working with SOS.dll) for debugging common language runtime.

5. WinDBG supports sharing the debug session on remote machine, which means two people can analyze an issue at the same time by sharing a debug session.

What is SOS.dll (Son of Strike)?

An NTSD, short for NT system debugger (a low-level debugger); an extension DLL, which enables managed code debugging while used with WinDBG ( used for native debugging ). SOS extension dll can also be used directly from within the visual studio IDE and allows seeing what is happening inside GC and heap.

Notes:

SOSEx is an upgrade of SOS. Read about it in Points of Interest section.

Debugging Essentials

Following are the terms we need to understand before delving into the original debugging steps using WinDBG:

1. Debug Symbols

2. Program Database

3. Managed vs. Native Debugging

4. First and Second Chance Exceptions

5. Crash dumps vs. Mini dumps

6. Crash vs. Hang dump (mini-dump)

7. Kernel Mode vs. User Mode

Debug Symbols

Debug symbols help the debugger to map raw addresses in the executable to source-code lines. These symbols are nothing but pdb files each stored in a separate directory and can be downloaded from Microsoft site http://msdl.microsoft.com/download/symbols

See this figure to check how to set debug symbols in visual studio.

From MSDN: You must have symbol information when you debug applications with various Microsoft tools. Symbol files provide a footprint of the functions that are contained in executable files and dynamic-link libraries. Additionally, symbol files can present a roadmap of the function calls that lead to the point of failure. For example, you must have the symbols when you dump call stacks inside a debugger.

Program Database

Program database files with extension .pdb, stored in a file separately from executable. A program database (PDB) file holds debugging and project state information that allows incremental linking of a Debug configuration of your program. They are really required when you do not have source code on the debugging machine.

How debugger finds pdbs?

Let's take Visual Studio debugger first, as most of us use it frequently, it uses the project PDB file created by the linker directly and embeds the absolute path to the PDB in the EXE or DLL file. If the debugger cannot find the PDB file at that location or if the path is invalid (for example, if the project was moved to another computer), the debugger searches the path containing the EXE, the Symbol Path specified in the solution's Property Pages (Common Properties folder, Debug Symbol Files page).

See this figure to check how to set debug symbols in visual studio.

WinDBG searches for the required pdbs at the path defined for symbol server which is 'Symbol File Path'. Also see this figure.

Managed vs. Native Debugging

Let’s suppose that we have a C# or VB project that uses unmanaged COM objects.

Considering Visual Studio debugger again,

If we want to debug only the c# or vb code written by us, that's managed debugging.

But at some point we might want to debug into the COM object, that's native debugging.

And, if we enable native debugging following the steps below

I) Right click on the project in the Solution Explorer

II) Go to properties -> “Configuration Properties/Debugging/Enable Unmanaged Code”, and check it to enable.

Figure 1: Enabling Unmanged Debugging in Visual Studio

It means now you can debug both managed and native code which is actually mixed debugging.

Notes:

WinDBG is an advanced level native code debugger and when used with SOS, also supports debugging managed code.

First and Second Chance Exceptions

A debugger gets notified of each exception twice – it is notified the first time before the application gets a chance to handle the exception (‘first chance exception’); if the application does not handle the exception, the debugger is given a chance to handle the exception ( ‘second-chance exception’). If the debugger does not handle a second-chance exception, the application quits.

Crash dumps vs. Mini dumps

There are two ways to use the debugger; either attach it to a running process or use it to analyze a crash dump. The first option allows you to look into what is going on in your application while it is running but this technique is feasible while debugging in development environment not production environment (here we need memory dump). There are crucial differences between the two environments. I would not discuss the differences here as it is out of scope of this article. So, Dumps provide an option to separate the actual debugging phase from the data collection process.

Crash dumps

The first generation of crash dumps, often called “full user dumps” and included every byte of the entire process space, so a crash in a simple application like Text Editor would be several megabytes in size. While undoubtedly useful for post-mortem debugging, such dumps often became so huge that it was impossible, or at least inconvenient, to transfer them to the software developers electronically. These are no longer used.

Minidumps

Minidumps are highly customizable. While generating a mini dump instead of saving the entire process space, only certain sections are saved. But if needed, a mini dump can contain even more information than an old style crash dump (for example, mini dumps can contain information about kernel objects used by the process).

The customizable nature of mini dumps allow to choose what information about the application’s state do we need to debug effectively while keeping the mini dumps as small as possible? Minidumps are to be taken using external tools; we’ll be using

adplus.vbs script; part of WinDBG installation.

We’ll be seeing to take a customized dump in Capturing Dumps section.

Writing a mini dump without using an external tool:

Redistributable DbgHelp.dll exposes a public API for creating mini dumps programmatically and we do not have to rely on external tools anymore. This can be really helpful in a sensitive production environment where some customer might not want to install tools to take dumps. Writing a mini dump with MiniDumpWriteDump() is explained nicely by Andy Pennell

Crash vs. Hang dump (mini-dump)

A crash dump is needed when we can't determine when the problem (typically a crash like the name implies, but that's not the only case) will happen, so we can configure the debugger in advance to monitor our target process and capture a dump when the process will be terminated, or when we need to capture a dump on a specific exception (We’ll be seeing capturing dumps in later section Capturing Dumps).

A hang dump can be captured after the problem has occurred but the process is still in memory, for example in a memory leak scenario but also when a process is burning the CPU.

Notes:

From here onwards, when I say dump, I mean to say mini dumps as we’ll be working with mini dumps throughout the discussion.

How to Configure WinDBG?

Launch WinDBG

Please install Debugging Tools for Windows from WinDbg Installation

After installing you can see WinDBG.exe in the installation folder, launch the executable.

Figure 2: A Quick Look at WinDBG Options

Set Symbol Path

To the Microsoft Symbol Server with WinDBG, follow these steps:

1. Start the Windows Debugger.

2. On the File menu, click Symbol File Path (Ctrl+S).

3. In the Symbol Path box, type the following

SRV*your local folder for symbols*http://msdl.microsoft.com/download/symbols

Where your local folder for symbols is the folder in which you copy your local symbol cache. The debug symbols are downloaded to this location.

Figure 3: Set Symbol Path

Set Source Path

1. Start the Windows Debugger.

2. On the File menu, click Source File Path (Ctrl+P).

3. In the Source Search Path box, type the following

C:\SourceDir\

Where, SourceDir contains source files of target application to be debugged.

Figure 4: Set Source Path

WinDBG and SOS Commands

Following is the cheat sheet, which explains most of the commands to start with:

You can download it from Kent Boogaart blog, download is available in various formats.

SOS help: Typing “!SOS.help” in the WinDBG command window would list all the SOS commands.

List of SOS Commands By Category

Following is well categorized list of SOS commands, borrowed from Johan Straarup's blog . Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.

Object Inspection ----------------------------- DumpObj (do) DumpArray (da) DumpStackObjects (dso) DumpHeap DumpVC GCRoot ObjSize FinalizeQueue PrintException (pe) TraverseHeap Examining CLR data structures ----------------------------- DumpDomain EEHeap Name2EE SyncBlk DumpMT DumpClass DumpMD Token2EE EEVersion DumpModule ThreadPool DumpAssembly DumpMethodSig DumpRuntimeTypes DumpSig RCWCleanupList DumpIL

Examining code and stacks ----------------------------- Threads CLRStack IP2MD U DumpStack EEStack GCInfo EHInfo COMState BPMD Diagnostic Utilities ----------------------------- VerifyHeap DumpLog FindAppDomain SaveModule GCHandles GCHandleLeaks VMMap VMStat ProcInfo StopOnException (soe) MinidumpMode

Help on Specific Command

If you need help on a specific command you can use !< SOS >.help <commandname>:

0:000> !sos.help dumpobj

-------------------------------------------------------------------------------

!DumpObj [-v] [-short] [-r 2] <object address>

This command allows you to examine the fields of an object, as well as learn important properties of the object such as the EEClass, the MethodTable, and the size.

Notes:

Here is an entire list of WinDBG commands.

Executing Batch Commands

There are two ways to execute the above commands as a batch

1. Enter all the commands in the debugger window as a single string, all separated by a semicolon.

2. Store all the commands in a script file and run using the following command

$$>< (Script file)

CPU Registers

While analyzing stack and heap we would be seeing the following names:

EAX - Generic 32 bit (8 byte) register

EBX - Generic 32 bit (8 byte) register

ECX - Generic 32 bit (8 byte) register

EDX - Generic 32 bit (8 byte) register

EDI - Extended Destination Pointer

EBP - Extended Base Pointer

ESI - Extended Source Index

ESP - Stack Pointer

EIP - Current CPU address (Instruction pointer)

3. Capturing Dumps

Taking Dumps using Adplus Script

Adplus script “adplus.vbs” is a part of WinDBG installation, which allows taking customized dumps with various configurable options.

Please follow the link which explains step by step on how to capture customized dumps.

Notes:

1. Usually, it works to take dump just after the exception has occurred, But if process exits shortly after exception has occurred; then start WinDBG and run the application directly from WinDBG. It will load the .exe and immediately break. Type "g" (go) and press enter.

When the process terminates WinDBG will automatically break again. You can then save a dump using “ .dump” command.

.dump /ma <output file>

4. Debugging with WinDBG and SOS

Debugging Live Targets

This type of debugging is mostly done on development machines, where application source code is also available.

I. using WinDBG

1. Start WinDBG

2. Run the application directly from WinDBG using File -> Open Executable (Ctrl+E)

Or

Attach to a running process using File -> Attach to a Process (F6)

"Non Invasive" option in the dialog showing process list, would allow you to detach from the debugged process without killing it (if you’re on Windows XP or 2003) but it limits the commands that you can use so leave it unchecked.

Figure 5: Attach to Running Executable

3. WinDBG will load the executable and immediately break the debugee execution.

4. Type ".load SOS" in the command window, to load SOS extension and enable managed code debugging.

Figure 6: Load SOS Extension

5. Setup breakpoint on a function issuing the following command

"!bpmd TestApp.exe TestClass.MyMethod "

If required to break program execution at some specific point.

6. Type "g" (go command) and press enter to switch to debugee (target application), so that you can work on it.

7. WinDBG will automatically break again, when either the process has terminated exceptionally or breakpoint hits. You can then investigate the process (At this point you can also save a dump using the .dump command for postmortem debugging).

Notes:

1. When the debugee is running you can always get back to the debugger by issuing a CTRL+BREAK to the WinDBG window when it will try and halt the debugee for you and drop you back to the command prompt; some people call it immediate window.

2. More Examples: These examples are enough to achieve good comfort level to use useful commands to analyze the issues.

II. without Using WinDBG [Using SOS from visual studio]

SOS can also be used directly from within VS2003 and updated version of visual studio without WinDBG.

Follow these steps to work with SOS:

1. Open the target visual studio project in debug mode, let’s call it debugee.

2. Enable native debugging

In order to load any extension like SOS.dll you have to be debugging in native mode, so before starting the debugger.

Go into Project/Properties/Debug on the context menu for the project and check the box for Enable unmanaged code debugging.

3. Set Symbol Server Path

a) Setup Visual Studio 2003 to use the Microsoft Symbol Server

In the Project Properties dialog box, set the Symbol Path on the Debugging page to:

SRV*c:/symbols*http://msdl.microsoft.com/download/symbols

This string tells the debugger to use the symbol server to get symbols and create a local symbol server, where they will be copied.

Figure 7: Setting up Debug Symbols in Visual Studio

b) Setup Visual Studio 2005 to use the Microsoft Symbol Server

4. Press F5 to start debugging

5. Load SOS Extension

See the output window, it will show all the loading symbols. After visual studio stop loading the symbols, load SOS typing the following command in the immediate window ( Alt+Clt+I )

.load C:\WINDOWS\Microsoft.NET\Framework\vx.x.xxxx\SOS

Where vx.x.xxxx the version of .Net framework used by debugee

Or

.loadby SOS mscorwks

6. Analyze heap using SOS commands

An Example: It is a small tutorial to use SOS from within visual studio 2005

Notes:

More Examples: These examples are enough to achieve good comfort level to use useful commands to analyze the issues.

III. Using Both WinDBG and Visual Studio Debugger at once

This explains how one can use ease of use of Visual Studio and the power of WinDBG together.

Debugging Dump files (Postmortem Debugging)

I. Using WinDBG

1. Open dump file using WinDBG File -> Open Crash Dump (Ctrl+D) menu.

2. Load SOS by typing in the command window

.load SOS

OR

.load C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\SOS

3. Analyze the dump

An Example

Notes:

More Examples: These examples are enough to achieve good comfort level to use useful commands to analyze the issues.

II. Without Using WinDBG [Using SOS from visual studio]

The interesting thing is that VS 2003 actually does support postmortem debugging of managed applications as well using SOS. However, it supports only managed mini dumps unlike VS upgraded versions, which support both managed and native dumps.

1. Open the .dmp file in Visual Studio using File->Open->Project

2. Press F5 to start debugging

3. Load SOS

Once you've clicked through the prompts and symbols have been loaded (you can see all the symbols loaded in output window), open the immediate window and type.

".load SOS”

Once the extension loads successfully you can type “!help” that would list all the SOS command

4. Analyze the state of your managed application as had captured in the dump

Notes:

More Examples: These examples are enough to achieve good comfort level to use useful commands to analyze the issues.

5. Examples

1. Troubleshooting OutOfMemoryExceptions

2. Debugging .NET Hang

3. Debugging .NET Memory Leak Part I: Define the “Where?”

4. Debugging .NET Memory Leak Part II: A Case Study

6. Points of Interest

1. This blog contains the following useful readings:

I. How to automate WinDBG startup; just by selecting a menu option available on .dmp file right click

II. How to connect to a remote debug session?

2. SOSEx

It is an upgrade of SOS; more powerful and easy to use, also solves some of the common limitations while using SOS. You can see SOSEx commands in action here .

3. Dumping on a terminal session

If you need to dump a process on a Windows NT or Windows 2000 machine, you cannot do so through a Terminal Server session, since the debugger cannot attach to a process which runs in a different Window Station, and if you try you'll likely get a message like the following:

Win32 error 5
"Access is denied."

This is by design but there are a couple of solutions if you need, check You cannot debug through a Terminal Server session or "Running in crash mode remotely" topic in Windbg help.

DebugDiag makes it easier to overcome this limitation, simply configuring it to run in service mode through its property dialog. Read more here.

7. History

February 12, 2008 - Initial Version

You must Sign In to use this message board.

FAQ  Noise Tolerance Very HighHighMediumLowVery Low  Search Messages    Layout Expand All MessagesThread ViewNo Javascript (slow)No Javascript Preview    Per page 102550

Msgs 1 to 8 of 8 (Total in Forum: 8) (Refresh) FirstPrevNext

Pictures! Initri 11:37 12 Feb '08   An article like this needs pictures and captions to capture attention. Without it, it's just a long article that may have substance but it difficult to read without seeing what you're doing, visually. I can't just scan it, and think to myself, "Oh! This would really help.. look at that.." because there is nothing to look at, and / or capture my attention. Thanks! Matthew MacSuga Sr. Software Developer Interlink Advantage 509-789-2443 Sign In·View Thread·PermaLink Re: Pictures! sonal.india 18:02 13 Feb '08   Thanks Matthew, I have uploaded some of the images and reorganized the contents a bit for better understanding. I Hope, this will help. sonal Sign In·View Thread·PermaLink Re: Pictures! Initri 18:19 13 Feb '08   Sonal, I think that's a huge addition to the article. Some more pictures, with more detail of what's happening during a debug session would make it even better. You're doing a terrific job explaining something very complex. Walk-us-through it! Thanks, - Matthew Matthew MacSuga Sr. Software Developer Interlink Advantage 509-789-2443 Sign In·View Thread·PermaLink Re: Pictures! ThatsAlok 22:20 10 Mar '08   good article i must say! "Opinions are neither right nor wrong. I cannot change your opinion. I can, however, change what influences your opinion." - David Crow Never mind - my own stupidity is the source of every "problem" - Mixture cheers, Alok Gupta VC Forum Q&A :- I/IV Support CRY- Child Relief and You/codeProject$$> Sign In·View Thread·PermaLink Re: Pictures! sonal.india 3:39 11 Mar '08   Thanks Alok sonal Sign In·View Thread·PermaLink Re: Pictures! ThatsAlok 3:53 11 Mar '08   tell me one thing .. do you only work in vc++ or some other language also?? Sign In·View Thread·PermaLink Re: Pictures! sonal.india 3:57 11 Mar '08   I work in vc# only not even vc++. sonal Sign In·View Thread·PermaLink Re: Pictures! ThatsAlok 4:03 11 Mar '08   great than! noida is god it hub i must say! keep up the good work! Sign In·View Thread·PermaLink 5.00/5 (1 vote)

Last Visit: Friday, November 21, 2008 11:07 AM     Last Update:Friday, November 21, 2008 11:07 AM