The first thing I'd do is go to the manufacturer and say "help!" While I've done it in the past reverse engineering the interface to a DLL is a real pain in the bum.
You can divide C++ Dlls up by what sort of interface they have. As a broad sweeping generalisation they are:
- The full monte - the DLL exports classes (or rather enough of a class to enable it's use by a client program). Classes in the DLL act just like they were in the client program.
- Walled garden - the DLL exports a set of pairs of factory/demolition functions and the rest is done through interfaces described in a header file.
- The great pretender - it says it's C++ but it's just got a C interface.
Each of them have their own particular quirks and difficulties. Note that any DLL can have elements of all three in it to add fun and frivolity to the mix.
In some respects the full Monte is the easiest to handle. If you've got a library that exports whole classes you'll probably get enough information from Dependency Walker (as Andrew suggested) to be able to reconstruct something that looks like the public interface to that class. The problem here is that you've not quite got enough information to create instances of the class as you don't know it's size. About the only way I found of working this out was trying different sizes (making the data of a class an array of characters) until I found one that didn't crash. One good thing is that if you overestimate the size then you can generally get away with passing the object by reference. If you have to pass by value though the Dll will interpret the call stack differently from your client code and much merriment follows. Oh, and you won't get any inline functions - which may or may not be a problem depending on what they do and how they do it.
The great pretender can also be fairly easy to call without crashing your program on the call but only for certain types of functions. Functions with a
type have the size of their parameters appended to their name. So if you see something like
as a name you know it wants 4 bytes worth of parameters. Declaring it as something as a single 4 byte parameter would sort that out. if it's declared as a C function then you can start experimenting with huge parameter sizes as the client is the thing that does the cleanup.
The walled garden is a complete bitch. You have no type information. You don't know what you're being supplied, except that it's a pointer. Of some kind. And you have to give it back afterwards. If the factory functions are
you won't even have a type name for a return type.
Anyway, if you're lucky you might end up with a header file that describes the interface to the DLL after a fashion. You won't know what the parameters to the functions are - you might know their type names for a full monte class but you won't know what they're used for. The next thing you have to do is start tickling the Dll through the interface you've discovered and see if you can get it to giggle appropriately. You can either start hacking your way through with a debugger or try disassembling it. The idea is to find what the parameters are used for and what internal state, if any, they modify. Some functions are easy to spot - brain damaged get/set pairs for example. However sometimes the compiler has a bit of a laugh and you'll find that the optimiser has been busy and the code you're looking at doesn't appear very descriptive. If your library has virtual functions you can find the address of the v-table (it's usually the first 4 bytes of an object, watch out for multiple inheritance) for an instance of the class and then trace through how a particular call would execute. You're going to have to do a lot of that for walled gardens.
The upshot of this lot is that you really have to need that library to fiddle around like this. By the time you can get something reliable out of it you might be better off writing your own code to do it. Please don't think it's an easy job! Although knowing my track record of making such pronouncements someone's probably got an open source program that'll do all this for you
PS: Just thought of something else I've tried - if you've got an example client that's compiled you can see what happens around the calls to the Dll. If something drops the stack pointer by 24 bytes, pushes the address on the stack then calls the default constructor you've got the size of the class.
So if you've got one pull any clients to pieces for extra information.