NetExt – SOS on Steroids

I have been playing recently with quite a new windbg extension (released by Rodney Viana from Microsoft) called NetExt. Rodney Viana published an introductory post about it, which you may find on his blog. In this post, I would like to show you my usage samples as well as encourage you to start using it by yourself. Netext documentation is thorough and nicely organized which is good because at the beginning you probably will spend a lot of time on this page. :) In paragraphs that follow, I will focus mainly on dump debugging, but most of the techniques presented here should work as well in live debugging sessions.

Finding Yourself In A Dump

Starting steps depend on the type of diagnosis you need to perform. Though it is almost always worth knowing the CLR Runtime version (!wver):

!wver
Runtime(s) Found: 1
0: Filename: mscordacwks_amd64_Amd64_4.0.30319.34209.dll Location: 
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll
.NET Version: v4.0.30319.34209
NetExt (this extension) Version: 2.0.1.5550

and the Windows version (vertarget and version):

0:000> vertarget
Windows 7 Version 7601 (Service Pack 1) MP (4 procs) Free x64
Product: WinNt, suite: SingleUserTS
kernel32.dll version: 6.1.7601.18869 (win7sp1_gdr.150525-0603)
Machine Name:
Debug session time: Wed Jul  8 12:40:05.000 2015 (UTC + 2:00)
System Uptime: 0 days 4:40:15.908
Process Uptime: 0 days 0:16:10.000
  Kernel time: 0 days 0:00:00.000
  User time: 0 days 0:00:02.000

Information about the debuggee process (such as the command line, environment variables, security context, etc.) can be revealed with the help of the !peb command and finally you may list appdomains hosted by this process with !wdomain.

Scanning Through Threads (Exceptions, Locks)

Knowing what threads are doing is helpful when you are diagnosing application failures or hangs. To list threads currently running in the application use the !wthreads command. This command will show you also a number of locks acquired by a given thread as well as information about the thread type and the last exception that occurred in it. Example output:

0:000> !wthreads
   Id OSId Address          Domain           Allocation Start:End              COM  GC Type  Locks Type / Status             Last Exception
    1 23e4 0000000001f76e50 0000000001f69720 0000000000000000:0000000000000000 MTA  Preemptive   0 Background
    2 1fc0 0000000001fa9200 0000000001f69720 0000000000000000:0000000000000000 MTA  Preemptive   0 Background|Finalizer
    3 0a3c 0000000001ff8830 0000000001f69720 0000000000000000:0000000000000000 MTA  Preemptive   0 Background|Timer|Worker
    4 2124 0000000001ff9000 0000000001f69720 0000000000000000:0000000000000000 NONE Preemptive   0 Background
    6 0e3c 0000000001ff9fa0 0000000001f69720 0000000000000000:0000000000000000 NONE Preemptive   0 Background|Wait|Worker
    7 ---- 0000000001ffa770 0000000001f69720 0000000000000000:0000000000000000 MTA  Preemptive   0 IOCPort|Worker|Terminated
    8 ---- 0000000001ffaf40 0000000001f69720 0000000000000000:0000000000000000 MTA  Preemptive   0 Worker|Terminated

Next step would be to dump managed stacks for all the running threads ~*e!wclrstack. To dump the exception data, use the !wpe command (it requires an exception address, but I’ve already created a pull request to the netext repo so it could also print the current exception in the thread) or !wdae to dump all the exceptions in the heap. Maybe you don’t find those commands very innovative, but in my opinion they produce more readable output than their SOS alternatives. Unfortunately, to diagnose problems with locks, you still need to stick to SOS or SOSEX commands (an example of such a diagnosis session can be found in one of my recent posts).

Analyzing the GC Heap

The NetExt extension unveils its real power when it comes to the GC heap analysis. Apart from standard commands such as !wdo (dump object), !wclass (dump class definition), !wheap (with interesting switches: -detailsonly, -type {partial-type-name} or -mt), we have at our disposal a heap-query language! To make it work, we need to first create a heap index using the !windex command. If our dump file is huge and generating the index takes a lot of time, it might be worth saving it (with the -save {file-name} switch) and later load it (-load {file-name}) each time we analyze the dump. !windex command has also an interesting -implement {partial-type-name} switch which dumps all the objects implementing a given type. Finally, it has a switch to filter objects by type (-type {partial-type-name}) or method table (-mt {method-table}). With the generated index, we are now ready to query our heap. We can start with the basic !wselect, but the most powerful of the NetExt commands is !wfrom. Its syntax is as follows:

!wfrom [/nofield] [/withpointer] [/type <string>]
       [/mt <string>] [/fieldname <string>] [/fieldtype <string>]
       [/implement <string>] [/obj <expr>]
       [where (<condition>)] select <expr1>, ..., <exprN>

There is a big number of functions you may use in the where and select parts and they are all listed in the documentation. Below, I present you some example queries to show you how flexible this command is.

Show execution contexts bound to running threads

You may check the SOS way of finding this information in my debug recipe and compare.

> !wfrom -nospace -nofield -type System.Threading.Thread select "System TID: ", 
     $thread(DONT_USE_InternalThread), ", Managed TID: ", m_ManagedThreadId, ", address: ", $addr(), ",
     execution context: ", m_ExecutionContext

System TID: #INVALID#, Managed TID: 0n22, address: 0000000100066D48, execution context: 0000000000000000
System TID: 28, Managed TID: 0n23, address: 0000000100068D48, execution context: 0000000000000000
System TID: 29, Managed TID: 0n26, address: 0000000100098E18, execution context: 0000000000000000
System TID: 7, Managed TID: 0n1, address: 00000001FFE674E8, execution context: 00000002FFEB2CD8

Show all recently run SQL queries

List all SQL queries with the addresses of their parameters arrays:

> !wfrom -nospace -nofield -type System.Data.SqlClient.SqlCommand select $addr(), 
   ", params: ", _parameters._items._items, ", sql: ", _commandText

00000002000198E8, params: 0000000200019AE0, sql: dbo.TempResetTimeout
00000003000682A8, params: 0000000300068528, sql: SELECT * FROM ttt WHERE category=@category 
AND parent_id=@parent_id AND (enabled = @enabled Or enabled = 1) ORDER BY sortorder
...

For a given query, we may then list its parameter values:

> !wselect _parameterName, _value from 0000000300068528

[System.Data.SqlClient.SqlParameter[]]
***************
[0]: 00000003000683c0
[System.Data.SqlClient.SqlParameter[]]
(string)System.String _parameterName = @category
System.Object _value = 00000000fff0c620 testcat
***************
[1]: 0000000300068568
[System.Data.SqlClient.SqlParameter[]]
(string)System.String _parameterName = @parent_id
System.Object _value = 0000000300068210 5
***************
[2]: 0000000300068678
[System.Data.SqlClient.SqlParameter[]]
(string)System.String _parameterName = @enabled
System.Object _value = 00000000ffeb4950 False

Show all current HTTP requests data

This example is taken from the Rodney’s post:

> !wfrom -nospace -nofield -type *.HttpContext select $rpad($addr(),10)," ",$if(!_thread, "  --",
    $lpad($thread(_thread.DONT_USE_InternalThread),4))," ",$if((_timeoutSet==1),$tickstotimespan(_timeout._ticks), 
    "Not set "), " ", $if(_response._completed || _finishPipelineRequestCalled,"Finished", 
    $tickstotimespan($now()-_utcTimestamp.dateData)), " ", $replace($lpad(_response._statusCode,8),
    "0n","")," ", $rpad($isnull(_request._httpMethod,"NA"),8), " ", $isnull(_request._url.m_String, 
   _request._filePath._virtualPath)

00000000FFF22F80   -- 00:00:00 Finished    200 GET      http://localhost:80/test/WebResource.axd?d=xxxx
0000000100061CE0   -- 00:00:00 Finished    200 GET      http://localhost:80/test/home.aspx

Show all open memory streams

> !wfrom -implement System.IO.MemoryStream where (_isOpen == 1) select $addr()

calculated: 0000000200027E58
calculated: 0000000200209B90
calculated: 000000020021C4A0
calculated: 00000002004FD540
calculated: 00000002004FDE20
calculated: 000000040176E498

6 Object(s) listed
17 Object(s) skipped by filter

Working with GUID/datetime/IP fields

It was always troublesome to display GUID or datetime fields in a readable format. I used to run John Robbin’s scripts to accomplish that. With NetExt, you just use one of the available functions, for example to convert datetime ticks to a date, use the $tickstodatetime function:

0:007> !wfrom -obj 00000002009df6d8 select $tickstodatetime(dateData)
calculated: 2015-06-02 08:18:25

1 Object(s) listed

For GUIDs, use the $toguid function and for IP addresses, the $ipaddress function. Other field functions can be found in the documentation.

Misc Commands

To make the debugging experience even better, some shortcut commands are available:

• !wruntime to list active HTTP runtimes
• !whttp – show current HTTP context objects
• !wcookie – shows current HTTP cookies
• !wconfig – shows config lines in the memory (useful when the only thing you have is a file dump)
• !wdict – displays dictionary objects
• !wkeyvalue – displays objects stored in a NamedValueCollection

Extending

What makes this extension even greater is the fact that its source code is available under the GNU GPLv2 licence. So if you happen to miss some functionality, you may implement it by yourself. The setup might require few changes in the project configuration (there are some paths hardcoded and you need to download boost and tinyxml libraries), but it’s nothing too hard – in case you run into problems, contact me and I will help you deal with them.

The solution is composed of two projects: a native NetExt and a managed NetExtShim, where NetExt references NetExtShim and at the same time implements interfaces required to talk to the debugger engine (engextcpp.cpp). NetExtShim under the hood uses CLRMD to query the CLR runtime information and exports its managed functions to the native world with the help of the Unmanaged Exports package and COM interoping.