Debugging .NET Framework and MS Visual Studio Managed Classes at Run time and Design time

• Download source code - 25.9 Kb

Introduction

While the .NET platform has much to offer developers, one of the frustrations is the "black box" nature of the framework itself.

While Delphi has always shipped with the source code to the entire VCL, Microsoft has chosen not to ship the source code to the framework - leaving developers attempting to do advanced "stuff" in dark, when the often patchy documentation is inadequate.

This article attempts to show how it is possible to create debug versions of the framework assemblies (as also of any managed assemblies such as those used by Visual Studio itself) and how one can set breakpoints in IL code and step from one's own code into that of the framework.

Background

My frustration and quest started when I was attempting to create a user control with 3 panels. I wanted the end user to be able to drop the user control on a form and then add controls to individual panels using the Visual Studio designer. However, the Windows Forms designer does not offer this facility and is only able to place controls on the user control itself, not on its sub controls.

I felt that, knowing how Visual Studio and the framework work behind the scenes would help me understand how a custom designer that offers this functionality could be written.

Reflector .NET, an excellent free tool written by Lutz Roeder helped me view the decompiled source of the concerned assemblies (Microsoft.VisualStudio.dll and System.Design.dll) and figure out how designers are created and individual controls selected, etc.

However, the hierarchy is quite complex and the exact sequence in which the methods of the many classes are called and the contents of the variables was still obscure. Only stepping through the Visual Studio code would really help me understand its operations well.

Browsing the web did not yield any articles on this topic, with many submissions asserting that it was not possible to step into the framework classes.

However, perusal of the .NET Framework Developer's guide and a lot of luck helped me find one way to achieve the goal.

Since debugging design time behavior is more difficult and less well documented, the walkthrough demonstrates how to debug a custom designer for a user control. Of course, debugging run time behavior would not require the second instance of Visual Studio and one would just have to set a breakpoint in the source in the main instance of Visual Studio 2003 itself.

Step by step walkthrough

1. Ask the JIT-compiler not to strip debug info

   This requires the creation of an INI file with the same name as the process that is to be debugged - since we are going to be debugging Visual Studio in its design time mode, the name of the file will be devenv.ini and it will be located in the same directory as devenv.exe, usually \Program Files\Microsoft Visual Studio .NET 2003\Common7\IDE.

   The INI file is simple and looks like this:

   [.NET Framework Debugging Control]
   GenerateTrackingInfo=1
   AllowOptimize=0

   More information can be obtained from the .NET Framework Developer's Guide article Making an Image Easier to Debug.

2. Disable system file protection for the .NET framework directory

   If this is not done, any modified version of the assembly is immediately replaced with the original version by Windows' system file protection mechanism.

   While there are some manual ways to disable system file protection (see Disable Windows File Protection on the excellent WinGuides site), the shareware wfpAdmin utility from Collake Software is very convenient and allows specific folders to be removed from Windows File Protection. The minimum required is to remove file protection for the <WINDIR>Microsoft.NET\Framework\v1.1.4322 directory and its sub directories.

   The Microsoft Visual Studio assemblies are not under system file protection and do not require this measure.

3. Create debug versions of the assembly

   This involves a round trip from retail DLL -> ILDASM to extracted resources and an IL file containing MSIL instructions -> ILASM to recompiled DLL with debug information embedded and accompanying PDB file created.

   A pair of batch files is convenient to automate this process where multiple assemblies are to be converted.

   The first batch file simply loops through the specified directory, and for each file that matches the passed file mask, calls the batch file that does the actual processing.

   REM DEBUGMAKEALL.BAT
   
   rem process each matching file
   for %%a in (%1\%2) do DebugMake.bat "%%a"
   :end

   The second batch file first calls ILDASM to decompile the assembly and then calls ILASM to create a debug version of the assembly. The IL file is saved with the name of the assembly with .il suffixed.

   REM DEBUGMAKE.BAT
   
   rem delete any il file left over from a previous invocation, 
     else output will be appended to it and compilation will fail
   del %1.il
   
   rem call ILDASM to create the il file
   ILDASM /OUT=%1.il /NOBAR /LINENUM /SOURCE %1
   
   rem call ILASM to compile a debug version 
                 of the dll as well as a pdb file
   ILASM /DEBUG /DLL /QUIET /OUTPUT=%1 %1.IL

   Steps are:

   1. Open a "Visual Studio .NET 2003 Command Prompt"
   2. Change to the directory containing the assembly to be decompiled
   3. To create debug version of System.Design.dll, type:

      <Pathtothebatchfile>DebugMakeAll.bat . System.Design.dll

      To create debug version of all System.*.dll, type:

      <Pathtothebatchfile>DebugMakeAll.bat . System.*.dll

   The end result

   The end result of the round trip is an <assemblyname>.il file that contains MSIL, a recompiled assembly with the DebuggableAttribute set and a PDB file that contains debugging information. A number of ico and BMP files are also created and these can be deleted if so desired.

   A small portion of the IL file for System.Design.dll looks like this:

   .namespace System.Design
   {
   .class private auto ansi sealed beforefieldinit SRDescriptionAttribute
   extends [System]System.ComponentModel.DescriptionAttribute
   {
   .custom instance void 
       [mscorlib]System.AttributeUsageAttribute::.ctor(valuetype 
       [mscorlib]System.AttributeTargets) = 
       ( 01 00 FF 3F 00 00 00 00 ) // ...?....
   
   .field private bool replaced
   .method public hidebysig specialname rtspecialname 
   instance void .ctor(string description) cil managed
   {
   // Code size 15 (0xf)
   
   .maxstack 8
   IL_0000: ldarg.0
   IL_0001: ldc.i4.0
   IL_0002: stfld bool System.Design.SRDescriptionAttribute::replaced
   IL_0007: ldarg.0
   IL_0008: ldarg.1
   IL_0009: call instance void 
     [System]System.ComponentModel.DescriptionAttribute::.ctor(string)
   IL_000e: ret
   } // end of method SRDescriptionAttribute::.ctor

   To ensure none of these assemblies are loaded by any process, it is best to reboot the machine and open only a command prompt and possible Windows Explorer.

4. Install the debug version of the Framework assemblies into the Global Assembly Cache

   This is required only for Framework assemblies.

   The GacUtil supplied with the framework does this well and has the advantage of being able to process multiple files from the command line, unlike the GUI extension to Windows Explorer.

   Here again a pair of batch files do the trick.

   REM GACInstallAll.bat
   
   rem process each matching file
   for %%a in (%1\%2) do GACInstall.bat "%%a"
   :end

   REM GACInstall.bat
   
   rem call gacUtil, asking it to install the passed file
   gacutil /i %1

   Steps are:

   1. From the same command prompt, type, for example:

      <Pathtothebatchfile>GACInstallAll.bat . System.Design.dll

      or

      <Pathtothebatchfile>GACInstallAll.bat . System.*.dll

5. Fire up the instance of Visual Studio 2003 that will be debugged (the debuggee)

   Start Visual Studio 2003 and open up the solution that is to be debugged. A sample solution TestApp.sln is included in the source files download.

6. Initialize a new instance of Visual Studio 2003 that will be the debugger

   This second instance of Visual Studio 2003 will be the debugger.

   A. Create new blank solution

   Name it as devenv.sln and save it in the same directory that devenv.exe is located.

   B. Specify paths to source and symbol files

   This option is accessed via Tools -> Options.

   

   C. Attach to the debuggee

   

   D. Open the source file containing code to be debugged and set a breakpoint at a procedure that will be called by Visual Studio.

   

   The sample project devenv.sln in the sample files can be used for this. The paths to source and symbol files would of course have to be changed. In the walkthrough, a breakpoint has been set in the designer's overridden method Initialize - this will be called by Visual Studio when an instance of the designed control is created - either by dropping on to a form or by opening a form that contains the control in design view.

7. Switch to the debuggee and initiate the action to be debugged

   Switch over the Visual Studio 2003 instance and initiate an action that should cause the breakpoint to be hit in the other instance.

   If you are using the TestApp.sln, just open Form1.vb from the Solution Explorer.

8. Debug Framework classes at last

   As soon as Visual Studio creates the custom control, it will instantiate the custom designer and call its Initialize method. The debugger instance will stop execution at the placed breakpoint and come to the foreground automatically.

   As the image shows, Visual Studio has loaded the symbols for a number of framework as well as Visual Studio assemblies.

   The call stack also shows calls originating in framework and Visual Studio assemblies. The black (instead of grey) font indicates that symbols are available for the procedure higher up the stack. In case the stack says "<Non user Code>", right click and select "Show Non User Code" to at least view the names and parameters of methods for which debug versions of assemblies have not been created.

   

   Double click on the calling method from microsoft.visual.studio.dll!DesignerHost.Add and the debugger will automatically load the decompiled IL file and show the return line. Though the language is unfamiliar, all the features of the debugger are available for this source file also, including setting breakpoints, stepping into and over, etc. Also, as the image shows, the locals displays all locals in the calling procedure and any variable can also be inspected using the watch window.

   

   Of course the language is IL and not as easy to understand as VB or C#. With the help of a few articles (ILDASM is Your New Best Friend in Bugslayer, for example), it is however possible to get the gist of what the code is doing and the locals window is really useful in understanding what is happening.

Debugging System.dll and MSCorlib.dll

This section is in response to queries by some users as to whether it is possible to debug core assemblies such as System.dll and Mscorlib.dll also.

Preparing System.dll

This is relatively straight forward provided system file checking has been turned off.

The next screen shot shows the commands given and the results (@echo off has been added to all batch files to keep the display clear).

Preparing Mscorlib.dll

This is more complicated as Windows seems to load this DLL on normal startup. Preparing it requires:

1. Copy to another directory, decompile and recompile

The next screen shot shows the commands given and the results.

2. Reboot in Safe mode, replace mscorlib.dll with the debug version and install to GAC

Debugging these assemblies

We can use the same sample application.

The Sub main of Form1.vb has been modified to instantiate a CodeSnippetStatement (resident in System.dll) and also a StringBuilder (resident in mscorlib.dll).

The following screens show how it is possible to step through from the user code into the IL code for the constructors (called .ctor in MSIL) in each assembly.

Points of interest

I wonder why Microsoft chose not to ship the .NET Framework with code - they did choose not to obfuscate it (thanks, Microsoft) but considering many decompilers (Salamander, a commercial program from RemoteSoft, appears to do the best job and can decompile entire assemblies) do a good job of decompiling it anyway, why not just release the code itself?

Also, stepping through the MSIL code is very slow (about 20 times slower than stepping through custom code) and there seems to be a memory leak in Visual Studio 2003 as the virtual memory used went up to 800 MB (on my 1 GB RAM machine) after 15 rounds on breaking and resuming. I wonder if there are any ways of getting around these problems?

Note:

The article mentioned the need to create an INI file in the same directory as the process being debugged and with the same root name. In response to a query by Scott, further testing showed that this step is not required if working with assemblies recompiled to include debug information.