C-like Exports between VB6 and VB.NET

• Download source - 43.08 KB

Contents

• Introduction
• GetHashCode in VB6
  • How to make a C-like export in .NET
• Let's do something a little more complex: Hooks
  • Here comes .NET
  • Let's go back to our VB6 project
• Conclusions
• References
• History

Introduction

The purpose of this article is to explain the interoperation between .NET and VB6 without involving COM. To achieve such a result, we'll make C-like exports of some functions from a VB.NET DLL (wait, you'll see how). The first part of the tutorial will introduce a technique used with a very simple sample and then it will be explained how to set a global hook just using VB (6 and .NET), normally impossible without involving C or C++, but exporting from .NET in a C-way we'll do it.

GetHashCode in VB6

Every good .NET programmer knows the GetHashCode method of the String class: it "returns the hash code for this string." Well, now suppose that, for some reason, you need to use the algorithm [String].GetHashCode in VB6. Let's consider a clever way to do this.
The VB6 code (VB6Hasher project) is extremely simple. You just have a form with two textboxes and a button: when you click on the button, the content of the first textbox is hashed and the result is displayed in the second one.

Private Declare Function JustHash Lib "HashExporter.Net.dll" _
    (ByVal str As String) As Long

Private Sub btnHash_Click()
    txtHash.Text = JustHash(txtInput.Text)
End Sub

If you're a little familiar with VB6 syntax, you already have recognized the Declare statement: it refers to the following simple function in an external DLL made in VB.NET (HashExporter.Net project):

Public Function JustHash(ByVal str As String) As Integer
    Return str.GetHashCode()
End Function

Obviously this syntax isn't enough to make C-like export (that is to say, accessible from VB6 through the Declare statement), indeed you can't make an export like this in VB.NET, neither in C# nor any .NET language I know, except... except ILAsm (IL assembly language). Do not ask me why C# or VB.NET don't implement this feature...

How to Make a C-like Export in .NET

Once you have built the HashExporter.Net project, open the Visual Studio Command Prompt (Start menu, Programs, Microsoft .NET Framework SDK, SDK Command Prompt), reach the folder that contains the HashExporter.Net.dll and type:

ildasm /out:HashExporter.Net.il HashExporter.Net.dll

This command extracts the ILAsm code from the compiled DLL: with a dir you'll be able to see that HashExporter.Net.il and HashExporter.Net.res have been generated. Now, with a simple text editor, let's make some changes to HashExporter.Net.il.
The first thing to do is find the line starting with ".corflags" and replace it (that usually specifies the value 0x00000001) with ".corflags 0x00000002": this means that the executable will work on Win32 only. Then, just after this line, add the following:

.vtfixup [1] int32 fromunmanaged at VT_01
.data VT_01 = int32(0)

This would work in our case because we only need to export a single function, but obviously you can export as many functions as you want:

.vtfixup [3] int32 fromunmanaged at VT_01
.data VT_01 = int32[3] 

Then you have to find where the declaration of the class that contains the method to be exported is (something like .class private auto ansi sealed HashExporter.Net.mdlHashExporter), inside the class block look for .method public static int32 JustHash(string str) cil managed and right after the curly bracket, add the following code:

.vtentry 1:1
.export [1] as JustHash

or more in general:

.vtentry VTable:VTEntryIndex
.export [ExportOrdinal] as ExportedFunctionName

where VTable indicates the number (VT_XX) of the VTable (an array that stores exports, there can be more than one VTable, e.g. VT_01, VT_02, but usually one is enough), VTEntryIndex the index of the element of the VTable, ExportOrdinal the ordinal number of the export and ExportedFunctionName represents the name with which the function will be exported.
At this point, we can rebuild the code writing on the command prompt:

ilasm "HashExporter.Net.il" /DLL /OUT:"HashExporter.Net.dll" 
/RESOURCE:"HashExporter.Net.res"

If you have accomplished all the passages in the correct way, opening with Dependency Walker (a VS Tool) the new HashExporter.Net.dll you will see in the right pane the name of the exported function!
This is the long way to make a C-like export: it's possible to automate this process. I developed a tool that lets you manage these kind of exports like you do in C/C++: using a .def file. In my case, it was a .defnet file that a post-build tool read and used to edit the ILAsm. Anyway, this was for .NET 1.1, and to work with 2.0 needs a little edit. Maybe in future, I'll publish it. In the meanwhile, you could use this [^], but I think that working a little manually with ILAsm is interesting.
For further details, see Unmanaged code can wrap managed methods [^], Inside Microsoft .NET Assembler [^] and Standard ECMA-335 - Common Language Infrastructure (CLI) [^], partition III: CIL Instruction Set.
Now you can try to run VB6Hasher.exe and see what happens.

Let's Do Something a Little More Complex: Hooks

Windows Hooks. Hooking is a sort of subclassing, only it isn't associated with a single window, but with a thread or even the whole system. It's a kind of filter of Windows' messages that allow you to override, or add functionalities when a particular message is received. MSDN [^] hook definition:

A hook is a point in the system message-handling mechanism where an application can install a subroutine to monitor the message traffic in the system and process certain types of messages before they reach the target window procedure.

To set a hook, you need to call SetWindowsHookEx from user32.dll. Here's the VB6 declaration:

Private Declare Function SetWindowsHookEx Lib "user32" _
    Alias "SetWindowsHookExA" (ByVal idHook As Long, ByVal _
    lpfn As Long, ByVal hmod As Long, ByVal dwThreadId As Long) As Long

If you've ever tried to use SetWindowsHookEx in VB6, most probably you know what limitation this language imposes, in fact MSDN documentation says:

If the dwThreadId parameter is zero or specifies the identifier of a thread created by a different process, the lpfn parameter must point to a hook procedure in a dynamic-link library (DLL).

SetWindowsHookEx Reference [^]

That's because the thread (or threads if the hook is global, dwThreadId = 0) that you want to monitor will need to load the specified DLL into the address space of its own process.
When you create a hook, you have to pass to SetWindowsHookEx a pointer to a function (lpfn parameter). You may say: well, what's the problem? VB6 has the AddressOf operator! Yes, that's true and using AddressOf would work, but just if you are trying to hook a thread created by the calling process! As MSDN says, the parameter receiving the pointer must point to a function an external DLL, but VB6 (usually) can't expose function in a C-like way! So it's impossible to create a global hook or a hook associated with another process.

Here Comes .NET

The HookCExport.Net solution contains a function called HookWndProc that is defined exactly like CallWndProc [^] . Let's repeat the process done in the first part of the article to expose this hook procedure in a C-way.

1. Build HookCExport.Net solution
2. Use ildasm on HookCExport.Net.DLL
3. Edit HookCExport.Net.il as seen before
4. Rebuild with ilasm
5. Check with Dependency Walker if in HookCExport.Net.DLL is visible HookWndProc

Important: There's another fundamental thing to do to avoid a system crash. A global hook will call a lot of times per second the callback function and because of this, that function must be very fast or you will slow down the whole system until it crashes. As you know, .NET programs are compiled by the JIT compiler at first execution, so our DLL gets at the first call of HookWndProc. That's very dangerous because while the library is getting compiled, the same function is called a lot of times, this always lead the system to block. The solution manually compiles the DLL from the SDK Command Prompt:

ngen install "HashExporter.Net.dll"

Let's Go Back to our VB6 Project

That's the core of the VB6HookListener project, you can find it in mdlHooking:

'Creates the Hook
Public Sub Hook(hThread As Long)
    Dim hProc As Long
    'Load the external DLL
    hModule = Chk(LoadLibrary("HookCExport.Net.DLL"))
    'Gets the address of the callback function in the external DLL
    hProc = Chk(GetProcAddress(hModule, "HookWndProc"))
    'Sets the Hook for the message sent using SendMessage
    hHook = Chk(SetWindowsHookEx(WH_CALLWNDPROC, hProc, hModule, hThread))
    [...]
End Sub

Note: Chk is a function that just checks if the API call returns 0, if so throws an error, else returns the received value.
We said that the function pointer passed to SetWindowsHookEx must be in an external DLL, so we used the LoadLibrary and GetProcAddress which respectively loads the specified DLL in memory and gets a function pointer (the address) of the specified function (HookWndProc). You may be asking why we didn't use a declare statement as in VB6Hasher example; that's for two reasons: first you can't use AddressOf over a function declared through a Declare statement, and second you wouldn't be able to get the module handle that is needed from SetWindowsHookEx.
Finally the hook is set. Explaining the remaining code about hooking is beyond the scope of this article.

Conclusions

In this article, we saw how to make use of .NET capabilities from VB6 to realize things normally impossible without using heavy COM wrappers but just exposing from VB.NET some methods in C-like way. This technique is very useful whenever you need to use low-level features of .NET (just like the possibility of setting a global hook). You won't need to ask help to father-C anymore!

References

• MSDN, for the various Win32 API functions [^]
• Unmanaged code can wrap managed methods [^] - Emilio Reale
• How to set a hook in Visual Basic .NET [^] - Microsoft Knowledge Base (read the "Global hooks are not supported in the .NET Framework" section, it's false! I've just demonstrated it with this article!)
• Inside Microsoft .NET Assembler [^] - Serge Lidin, Microsoft Press
• Standard ECMA-335 - Common Language Infrastructure (CLI) [^], partition III: CIL Instruction Set - ECMA-international.org
• Spy: Monitoring Messages with SPY [^] - VC++ 6.0 SDK Examples

History

• 26^th June, 2007: Initial post