Customized Visual Studio .NET package: your fully integrated document window inside the IDE

• Download project - 212 Kb
• Download VS package spy - 27.2 Kb

Introduction

The most annoying and blatantly absent feature of Visual Studio .NET 'extensibility' is the incapacity to insert a custom window into the IDE. Obviously, the Extensibility Object model was deliberately castrated in the interest of Microsoft salesmen, since the VSIP (Visual Studio Integrated Products) package is distributed and sold separately. This article and supplied code will provide you with essential knowledge and the means to implement a customized Visual Studio .NET package - the software module allowing you to create your own integrated document windows inside the IDE. Integrated here means that the Visual Studio IDE will be completely aware of your component's presence:

• Standard IDE commands like 'Copy', 'Undo', 'Save' etc. will be routed to your module.
• 'Changed' state will be reflected by asterisk symbol right beside the document title.
• If the document is changed and not saved yet, the IDE will ask you to save it on 'Close' command as it does for other documents like source files and solutions.
• Your document will be able to alter particular command availability status (for example, enabling 'Undo' button on an IDE toolbar when the state of a document was changed).
• Your window will be inserted into the standard Visual Studio tab control along with other windows and can be dragged into different tab groups as well,
• Corresponding Document and Window automation objects will be created inside the IDE as happens to other 'native' windows opened in the IDE.
• You will be able to open your documents (i.e. show your windows) from Visual Basic macros using DTE.Documents.Open( szFileName, 'Auto'…) command.
• You can debug your package using standard Visual Studio debugger and devenv.exe as a debugger target.

To demonstrate, I implemented a package that opens files with the extension .bine and shows a histogram of the file binary data (i.e. a bar graph, so that each bar represents a number of bytes having a particular value). The snapshot above demonstrates a file temp1.bine opened side by side with C++ source files and a binary file opened in the binary editor.

I hasten to assure anyone reading this article that I have never had access to Microsoft documentation describing the development of custom packages or any other information regarding VSIP. All implementation details depicted here, with one exception, are the result of my own exploration, as is the source code. The exception is this: early in this project, I found the web document VSIPPackageCreationHandout.doc that contributed significantly to my understanding of Visual Studio inner mechanics. That document and other related pieces of information can be found in the Docs subdirectory of the project.

For reasons described above, my custom package cannot be a stand-alone module; in fact, it is actually a 'smart proxy' for the Visual Studio Binary editor package (bined.dll). It monitors the dialogue between the IDE and the package, and simply modifies the bined.dll responses if required. I checked the package on Windows 2000 with Visual Studio .NET and on Windows XP (Home ed.) with Visual Studio 2003. I expect no problems with these two versions of Visual Studio installed on other Windows operating systems, or on the next version of Visual Studio.

Installation procedure

To load a package into the IDE, you will need a PKL - Package Load Key. Since we don't have one, I worked around this by naming my package bined.dll and substituted it for the original bined.dll. Also, to cause a package to be loaded by the IDE, we need to associate a particular file extension with the package. So to install a package onto the system, follow the next three steps:

1. Find original bined.dll (it is usually found in Visual Studio root directory under \Vc7\vcpackages) and rename it to orig-bined.dll. Also, it is a good idea to back up a copy of original bined.dll in some other safe place before you start doing anything.
2. Copy a custom package DLL into the aforementioned directory and name it bined.dll. According to the settings in my workspace, the resultant file will be copied into C:\Program Files\Microsoft VisualStudio .Net\Vc7\Vcpackages on the 'post build event', so probably you will need to change it to reflect your machine's settings.
3. To associate a file extension with bined.dll, you will need to change the system registry. Under the following key: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\ 7.0\Editors\{25834150-CD7E-11D0-92DF-00A0C9138C45}\Extensions] (for Visual Studio 2002) and/or [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ VisualStudio\7.1\Editors\{25834150-CD7E-11D0-92DF-00A0C9138C45}\Extensions] (for Visual Studio 2003), add a DWORD value set to 0x32. The name of the value is the same as a file extension that you are going to use. For example: value name = xyz, type = DWORD, data = 0x32. Now the custom package will be loaded into the IDE if a file having extension 'xyz' is dragged into the Visual Studio IDE or opened through the standard "Open file" dialog.

Note 1: This does not associate your files with Visual Studio when using Windows Explorer!

Note 2: My custom package is using only a .bine extension and the use is hard-coded. To change this, you will need to modify a little, the implementation of CVsEditorFactory::CreateEditorInstance method.

The next two sections will not necessarily help you to build your components - by now, you have all the means; rather, they will explain how the code was developed and how code components execute.

Package life-time dynamics

A Visual Studio Package is a COM object that implements the IVsPackage interface. The layout of the interface is known (and can be extracted by the OleView utility from a type library). The other interfaces that the package must expose are: IVsEditorFactory, IVsPersistDocData, IPersistFileFormat, IVsWindowPane and IOleCommandTarget. Do not rush to search for the interfaces in the system registry or to explore them using the OleView utility. Some of them, like IVsWindowPane and IPersistFileFormat, are not even registered as interfaces under the [HKEY_CLASSES_ROOT\Interface] key. In addition to these interfaces, according to the Microsoft document that I mentioned in the introduction, the package should implement IDispatch. However, this interface had never been queried by the IDE while I was developing the package, so I left it unimplemented.

The loading of the package starts when a user opens in the Visual Studio, a file with an extension associated with the package. The IDE calls CoCreateInstance, obtains the IClassFactory interface from the package and creates an IVsPackage object. After the IDE has obtained the package object, it calls IVsPackage::SetSite method, supplying as a parameter, a pointer to the IServiceProvider object. The interface is described in MSDN and is intended to provide a client (in our case - the package) with an access to other interfaces (i.e. services) that a provider (the IDE) may implement. The package implementing SetSite method queries for IVsRegisterEditors interface (service), and calls IVsRegisterEditors::RegisterEditor method when the interface (service) has been obtained. The package supplies as parameters to this call, the editor's GUID and a pointer to an object implementing the IVsEditorFactory interface. After the internal registration is finished, the IDE calls IVsEditorFactory::CreateEditorInstance method and thus obtains pointers to Document and View objects created by the package (CDocumentData and CDocumentView classes respectively in the source code). These two objects must implement IVsWindowPane, IVsOleCommandTarget, IVsPersistDocData and IPersistFileFormat interfaces. The distinction between them is logical only and it can be a single object implementing all of the interfaces. The CreateEditorInstance method is the place where we first can see a name of the file (passed as a parameter) that is about to be opened. During the next stage, the IDE queries Document and View objects for all the above interfaces and starts the actual open file sequence: it calls IVsWindowPane::CreatePaneWindow to create a window, IVsPersistDocData::Load to load file data and IOleCommandTarget::QueryCommand to enable particular UI commands.

When a user closes the document window, the IDE calls IVsWindowPane::Close to close the window, IVsPersistDocData::Close to close the document and IVsRegisterEditors::UnregisterEditor to de-initialize editor factory. (The last method is declared as NoName() in my source code). Note, that at this stage, the package object is not being released. It only happens when the whole Visual Studio Environment is closed. At this last stage of the package's life, the IDE calls IVsPackage::QueryCanClose and then IVsPackage::Close methods.

Studying VS package

In this section, I will describe the process that led me to the understanding and implementation of the customized package. The process can be best described using the following algorithm:

1. For each known interface:
2. --For each method that might be queried for a new interface (QueryInterface):
3. ----Override method.
4. ----Find all interfaces that were successfully queried.
5. ------If no new interfaces – stop.
6. ------Investigate new interfaces, find number of methods and parameters.
7. ------Now new interfaces have become known – go to step 1.

I started from coding a DLL proxy for bined.dll with the same exported functions and implementation of IClassFactory. The next version of the proxy included implementations of IVsPackage interface and IServiceProvider wrapper. At this point, it had become impossible to accomplish the sixth step of the algorithm with conventional means, since I could not find any appropriate description of obtained interfaces.

To tackle this problem, I developed a tool named 'Spy'. The idea behind the tool is redirection of all virtual method calls to stub routines by modifying a virtual table (v-table) pointer of an original object. Obviously, each stub routine corresponds to a particular interface method. A stub routine enables to perform additional tasks before the actual method is being invoked. The tasks may include printing of diagnostic messages, setting breakpoints and etc.

Investigation of an interface begins from implementation of a spy for it. The only essential parameter to supply for the implementation is a number of methods that the interface has. It can be obtained from the system registry under the key [HKCR\Interface\GUID\NumMethods] where GUID designates the interface ID. Otherwise, if the interface (its IID) cannot be found in the system registry, we can make an intelligent guess based on analyzing the object’s v-table in memory in debug mode (usually a v-table is a sequence of adjacent addresses in the process's .text section followed by zeros or unrelated values, also remember that three first addresses are the pointers to IUnknown methods).

The implementation of a stub routine depends on whether or not the number of parameters of the corresponding interface method is known. If it is not, default implementation that only prints a diagnostic message is only available. Note that in this case, __declspec(naked) attribute is used in stub declaration to prevent C++ compiler from generating prolog and epilog code that may violate stack integrity. Also, if any additional actions in a stub routine caused a change in a stack state (like function calls), the initial stack state must be restored. And finally, the original method must be invoked using jmp assembly instruction (vs. call instruction), because the stack cannot be set properly to do so (the number of parameters is unknown).

With all these restrictions, a stub routine may seem useless, but in fact it is a good starting point for the step 6. First, you can see (based on diagnostic message) that the method is invoked and when. Second, you can set a break point in a stub and step into disassembled original method code. Step by step execution will lead you eventually to a return assembly instruction. Because COM methods are defined using __stdcall convention, they are responsible to clear the stack when the method terminates. The corresponding assembly instruction is ret X where X designates the number of bytes to be added to the stack pointer register on exit. For example, if X is 8, the method has only one parameter (usually each parameter is 4-byte long and any method has this pointer (4 bytes) as a hidden parameter). The return value can be obtained based on the EAX register value just before the ret instruction executes. Using this technique, I found for example, that many methods were actually empty just returning S_OK or E_NOTIMPLEMENTED codes.

The second possible implementation of a stub is relevant, when a number of parameters of the corresponding method is known. In this case, we can supply more comprehensive stub code that is particularly applicable regarding the IUnknown::QueryInterface method. Since every interface inherits from IUnknown and an address of the QueryInterface method occupies the first entry of the interface’s v-table, we can provide a full featured function that intercepts all interface queries of a spy target object, and thus be able to perform steps 2 through 4 of the investigation algorithm. Just remember to invoke the original object’s method. Also, in our implementation of QueryInterface, we can filter returned interfaces to find out which of them are essential and which are supplementary.

I realize that this section does not answer all possible questions, but I think it gives you a pretty good idea about why the final code looks like it does. Also, in case you wish to elaborate on code details, I supplied the final debug version of custom package source code that uses spies. The installation procedure is exactly the same as it was described above.

Limitations and known problems

First, the most obvious problem of my custom package is its tight dependency on bined.dll. But since it is a standard IDE package, I hope it will appear in future Visual Studio versions, otherwise we can choose another package to parasitize on. Also, I did not encounter and do not foresee in future any co-existence issues. The only thing to remember is to save the original package DLL and to make it accessible to the customized package.

Secondly, it seems to be a difficulty to add/create custom commands available to the package. Probably, the solution is creating an add-in with a command set known to the custom package as well.

Finally, I noticed while debugging the custom package that the IDE throws an exception when a mouse pointer hovers over a window caption tab. The IDE prints a message in the output debug window and continues to perform as though nothing had happened. It does occur even when only C++ source files are opened and my package still has not been loaded yet. I am assuming that the origin and only cause of the exception is the IDE itself.

Conclusion: further development

A possible direction for further investigation is extending DTE automation model with custom package supplied objects. A good starting point is the last paragraph of the discussed Microsoft document and implementation of the IDispatch interface in both Document and View objects. Another promising direction is to investigate/implement the integration of custom commands that a package can register on start-up and execute in steady state. Of course, any augmentation of current package functionality with respect to unimplemented methods and/or unused parameters is most welcome.

I hope my work will provide you with an essential tool to write more comprehensive add-ins for the Visual Studio IDE. The ability to combine tool windows with document windows in a single add-in, definitely leverages the complexity and diversity of tasks that can be accomplished. The only limit as usual is your imagination.