Debugging Custom DLLs with InstallShield for MSVC 6.0 Professional

• Download source files - 263 Kb

Introduction

Using the version of InstallShield that comes with the professional version of Visual C++ 6.0 allows you to easily generate installation programs. However, there may be some situations where you need additional functionality not included with InstallShield. This article explains how you can create and debug custom DLLs that allow you to add your own functionality to InstallShield projects. The example source demonstrates how you can execute applications, register ActiveX controls, and upload registration information to an FTP server.

Note: Debugging the custom DLL may only be performed with Windows NT 4.0, 2000, and XP.

Background

Soon after I had added all the files of my application to my new InstallShield project, I discovered that the method, LaunchApp, was disabled in the version of InstallShield that I have. Needing this functionality to launch my documentation at the end of the setup, I discovered a method, CallDLLFx, that allows you to call functions from a custom built DLL.

After a little research, I wrote a custom DLL that duplicated the LaunchApp functionality. I also discovered that debugging the custom DLL would prove to be challenging as well.

My first attempt to debug the custom DLL involved trying to run the setup.exe file from the debugger. Much to my surprise, Visual C++ cannot use the setup program used with InstallShield installations. I realized that I would have to debug the setup application a different way. The remainder of this article discusses the route I took and explains how to create your own custom DLL.

NOTE: It is assumed that you know how to generate an installation with InstallShield. However, if you don't, there is a Simple InstallShield tutorial by Bryce that can be found on the CodeProject web site.

Prerequisites

In order to follow the remainder of the article more easily, it is recommended that you make sure that the following items have been completed before continuing.

• Install both Visual C++ 6.0 Professional and the version of InstallShield that comes with it.
• Download and unzip the supplied source code (using folder names) to your C:\ directory. The InstallShield project relies on this directory structure.
• Compile the supplied source code found in the GreatProduct.dsw project workspace located in the folder, C:\My Installations\Projects\GreatProduct. You'll need to compile the release builds for the ActiveX_Control and GreatProduct projects. For debugging purposes build the DEBUG build for the IS_CustomDll project.

  The Great Product will serve as the product that InstallShield's setup program will be installing to your customer's computer.

The Visual C++ projects

The source code for the Great Product consists of an ActiveX control (an ATL full control) and a dialog-based MFC application. Both applications essentially contain the default boiler plate code that is provided by Microsoft. The only differences are that the text for the ActiveX control has been changed and the ActiveX control has been added to the dialog-based application.

The custom DLL

The source for the custom DLL can be found in the following file: C:\My Installations\GreatProduct\Projects\IS_CustomDll\IS_CustomDll.cpp

The DLL's project was generated by Microsoft Visual C++ as a Win32 Dynamic-Link Library. It exports five functions for use with the InstallShield setup program. Their prototypes are as below:

• int __stdcall StartProgram(HWND hWnd, LPLONG val, LPSTR str);
• int __stdcall RegisterServer(HWND hWnd, LPLONG val, LPSTR str);
• int __stdcall UnregisterServer(HWND hWnd, LPLONG val, LPSTR str);
• int __stdcall UploadRegistrationInfo(HWND hWnd, LPLONG val, LPSTR str);
• int __stdcall InsideDll(HWND hWnd, LPLONG val, LPSTR str);

StartProgram

The StartProgram function attempts to open the file specified in the str parameter using ShellExecute. The following code demonstrates how this is accomplished.

int __stdcall StartProgram(HWND hWnd, LPLONG val, LPSTR str)
{
  if (ShellExecute(hWnd, "open", str, NULL, NULL, SW_SHOW) <= (HINSTANCE)32)
  {
    return -1;
  }
  return 0;
}

RegisterServer / UnregisterServer

RegisterServer will be used by the setup script to register the ActiveX control used in the Great Product. I do realize that the ActiveX control could have been registered by InstallShield. This example demonstrates how to use LoadLibrary and GetProcAddress to obtain the address of the ActiveX control's exported DllRegisterServer function.

Looking at the prototype for RegisterServer, the str parameter contains the name of the control to register. If the control is loaded with LoadLibrary and the RegisterServer's address is found, then an attempt to register the control is made. Otherwise an error is returned.

Note that a prior call to CoInitialize is made in the DllMain function. The call to CoInitialize is critical to the registration of the control.

Also, because of the similarity between RegisterServer and UnregisterServer, UnregisterServer will not be discussed.

typedef HRESULT (__stdcall *__DllRegisterServer)();
typedef HRESULT (__stdcall *__DllUnregisterServer)();

BOOL APIENTRY DllMain( HANDLE hModule, 
                       DWORD  ul_reason_for_call, 
                       LPVOID lpReserved
           )
{
  switch (ul_reason_for_call)
  {
  case DLL_PROCESS_ATTACH:
    CoInitialize(NULL);
    break;

  case DLL_THREAD_ATTACH:
    DisableThreadLibraryCalls((HINSTANCE)hModule);
    break;

  case DLL_THREAD_DETACH:
  case DLL_PROCESS_DETACH:
    break;
  }
  return TRUE;
}


int __stdcall RegisterServer(HWND hWnd, LPLONG val, LPSTR str)
{
  // Attempt to load the ActiveX control using LoadLibrary.
  HMODULE hModule = LoadLibrary(str);
  HRESULT hResult = S_OK;

  if (NULL == hModule)
  {
    // the ActiveX couldn't be loaded.
    return 1;
  }

  // Obtain a pointer to the DllRegisterServer exported function
  
  __DllRegisterServer pDllRegisterServer = 
    (__DllRegisterServer)GetProcAddress(hModule, "DllRegisterServer");

  if (NULL == pDllRegisterServer)
  {
    // The function address couldn't be found
    return 2;
  }

  // Call the returned function address as you would a regular function
  hResult = pDllRegisterServer();

  if (S_OK != hResult)
  {
    MessageBox(hWnd,
               "DllRegisterServer failed",
               str, 
               MB_ICONEXCLAMATION);
  }

  // Free the resources.
  FreeLibrary(hModule);

  return hResult;
}

UploadRegistrationInfo

The UploadRegistrationInfo function is provided to give your users the opportunity to upload their registration information to your FTP server. This code uses the WININET.DLL to login and transfer the user's name, company and serial number to an FTP server running on the local host (obviously, this would need to be changed.) The following listing shows the implementation of this function.

#include "stdafx.h"
#include "wininet.h"

int __stdcall UploadRegistrationInfo(HWND hWnd, 
                  int value, char *registrationInfo)
{
  DWORD timeout = 60000;
  
  HINTERNET hInternet = NULL;
  HINTERNET hFtpSession = NULL;
  HINTERNET hFtpFile = NULL;

  // you'd want to change the IP address to something other
  // than the local host.  No sense in uploading the registration
  // to the customer's own PC.  :-)
  
  char FtpServerName[100] = "127.0.0.1";
  int FtpServerPort = 21;
  HANDLE hFile = INVALID_HANDLE_VALUE;
  char fileName[MAX_PATH];

  DWORD bytesWritten = 0;
  DWORD transferSize = 0;

  // Set the name for the transferred file to "reg.txt"
  //
  // NOTE: This file does not exist on your hard drive.  The string data in
  //       the registrationInfo will be stored on the FTP server as this
  //       filename.
  //

  strcpy(fileName, "reg.txt");
  
  // Set the wait cursor to let the user know something's happening.
  
  SetCursor(::LoadCursor(NULL, IDC_WAIT));

  
  // Attempt to open an internet connection
  
  hInternet = InternetOpen("HomeServer", INTERNET_OPEN_TYPE_PRECONFIG,
    NULL, NULL, 0);

  
  // Let the user know if it wasn't successful.
  
  if (hInternet == NULL)
  {
    MessageBox(hWnd, 
      "There was an error opening a connection to the internet.",
      "FTP Error", MB_ICONINFORMATION);

    SetCursor(::LoadCursor(NULL, IDC_ARROW));
    return false;
  }

  
  // set timeout durations for connects, sends, and receives.
  
  if (FALSE == InternetSetOption(hInternet, 
           INTERNET_OPTION_RECEIVE_TIMEOUT, &timeout, 
           sizeof(DWORD)))
  {
    // not necessarily a serious error, but report it anyway.
    MessageBox(hWnd, "Couldn't set receive timeout.",
      "FTP - warning", MB_ICONINFORMATION);
  }

  if (FALSE == InternetSetOption(hInternet, 
           INTERNET_OPTION_SEND_TIMEOUT, &timeout, 
           sizeof(DWORD)))
  {
    // not necessarily a serious error, but report it anyway.
    MessageBox(hWnd, "Couldn't set send timeout.",
      "FTP - warning", MB_ICONINFORMATION);
  }

  if (FALSE == InternetSetOption(hInternet, 
           INTERNET_OPTION_CONNECT_TIMEOUT, &timeout, 
           sizeof(DWORD)))
  {
    // not necessarily a serious error, but report it anyway.
    MessageBox(hWnd, "Couldn't set connect timeout.",
      "FTP - warning", MB_ICONINFORMATION);
  }


  // Attempt to connect to the FTP server.  This is hard-coded to the
  // local host (127.0.0.1) IP address.
  //
  // For your convenience, a sample FTP server has been provided and is
  // located in the c:\My Installations\GreatProduct\FTP folder.

  hFtpSession = InternetConnect(hInternet,
                                FtpServerName,
                                FtpServerPort,
                                "registered",
                                "registered",
                                INTERNET_SERVICE_FTP,
                                0,
                                NULL);


  if (NULL == hFtpSession)
  {
    MessageBox(hWnd, "There was an error connecting to the FTP server.",
      "FTP Connect Failed", MB_ICONINFORMATION);

    InternetCloseHandle(hInternet);

    SetCursor(::LoadCursor(NULL, IDC_ARROW));
    return 1;
  }

  hFtpFile = FtpOpenFile(hFtpSession,
                         fileName,
                         GENERIC_WRITE,
                         FTP_TRANSFER_TYPE_BINARY,
                         0);

  if (NULL == hFtpFile)
  {
    MessageBox(hWnd, 
      "There was an error uploading the registration information.",
      "FTP Upload Failed", MB_ICONINFORMATION);

    InternetCloseHandle(hFtpSession);
    InternetCloseHandle(hInternet);

    SetCursor(::LoadCursor(NULL, IDC_ARROW));
    return 2;
  }

  transferSize = strlen(registrationInfo);

  if (FALSE == InternetWriteFile(hFtpFile,
                                 registrationInfo,
                                 transferSize,
                                 &bytesWritten))
  {
    MessageBox(hWnd, 
     "There was an error uploading the registration information.",
     "FTP Upload Failed", MB_ICONINFORMATION);

    InternetCloseHandle(hFtpFile);
    InternetCloseHandle(hFtpSession);
    InternetCloseHandle(hInternet);

    SetCursor(::LoadCursor(NULL, IDC_ARROW));
    return 3;
  }

  if (bytesWritten != transferSize)
  {
    MessageBox(hWnd, 
     "There was an error uploading all of the registration information.",
     "FTP Upload Incomplete", MB_ICONINFORMATION);

    InternetCloseHandle(hFtpFile);
    InternetCloseHandle(hFtpSession);
    InternetCloseHandle(hInternet);

    SetCursor(::LoadCursor(NULL, IDC_ARROW));
    return 3;
  }

  InternetCloseHandle(hFtpFile);
  InternetCloseHandle(hFtpSession);
  InternetCloseHandle(hInternet);

  SetCursor(::LoadCursor(NULL, IDC_ARROW));


  MessageBox(hWnd, 
    "Your registration information was uploaded successfully.\n"
    "Thanks for registering the software.",
    "Registration Complete", MB_ICONINFORMATION);


  return 0;
}

The UploadRegistrationInfo function will attempt to login to the FTP server with registered as the username and registered as the password. If successful, an attempt will be made to create a file called reg.txt. If the file creation succeeds, the registration information stored in the pointer, registrationInfo will be sent to the FTP server.

NOTE: For your convenience, a demo FTP server has been provided in the source download.

Look in the C:\My Installations\GreatProduct\FTP folder for the FTP server demo's ZIP file. To use the demo, unzip both files into a folder of your choice. Double-click the file, FtpServerDemo, and press the Start button to start the server once the main screen is displayed. The demo will execute for approximately one day.

InsideDll

The last function, InsideDll, is a dummy function that does nothing code-wise. Its primary purpose is to allow the symbol table to be loaded into the Visual C++ debugger. It will be called by the setup program immediately after the user presses the "Next" button on the initial Welcome screen.

The InstallShield setup project

The InstallShield project for your Great Product is located in the file, C:\My Installations\GreatProduct\GreatProduct.ipr

To build the setup installation, first compile the script by choosing the menu, "Build->Compile". Then generate the actual setup files by choosing the "Build->Media Build Wizard...." Choose the Default listing in the Existing Media section.

When you get to the Build Type dialog, choose "Full Build". Otherwise, take all the defaults by pressing "Next."

A closer look...

The setup project supplied in this tutorial consists of three screens: the welcome screen, a destination screen, and the finished installation screen. Two of these screens are displayed in the ShowDialogs function through calls to DialogShowSdWelcome and DialogShowSdAskDestPath.

// this code can be found in the following path
// C:\My Installations\GreatProduct\Script Files\setup.rul

////////////////////////////////////////////////////////////////////////
//                                                                    //
// Function:  ShowDialogs                                             //
//                                                                    //
//  Purpose:  This function manages the display and navigation        //
//            the standard dialogs that exist in a setup.             //
//                                                                    //
////////////////////////////////////////////////////////////////////////
function ShowDialogs()
    NUMBER  nResult;

begin

  Dlg_Start:
    // beginning of dialogs label

  Dlg_SdWelcome:
    nResult = DialogShowSdWelcome();

    if (nResult = BACK) goto Dlg_Start;

    Dlg_SdAskDestPath:

    // ...

    nResult = DialogShowSdAskDestPath();
    if (nResult = BACK) goto Dlg_SdWelcome;

    // This indicates the type of installation to be used.
    // There is only one possibility, so a dialog is not needed.

    svSetupType = "Typical";

    return 0;

end;

The last function, DialogShowSdFinishReboot, contains the calls to CallDLLFx. CallDLLFx takes four parameters:

1. The name of the custom DLL, IS_CustomDll.dll, which must reside in the same directory as the setup program.
2. The function to be called (either StartProg, RegisterServer, or UploadRegistrationInfo)
3. A LONG value (not used)
4. A STRING which will contain either registration information for the FTP upload, the name of the ActiveX control to register, or the executable to start.

////////////////////////////////////////////////////////////////////////////
//                                                                        //
// Function: DialogShowSdFinishReboot                                     //
//                                                                        //
//  Purpose: This function will show the last dialog of the product.      //
//           It will allow the user to reboot                             //
//           and/or show some readme text.                                //
//                                                                        //
////////////////////////////////////////////////////////////////////////////
function DialogShowSdFinishReboot()
    NUMBER nResult, nDefOptions;
    STRING szTitle, szMsg1, szMsg2, szOption1, szOption2;
    STRING str, dllName, dllFunction;
    LONG value, retVal;
    NUMBER bOpt1, bOpt2;
begin
  if (!BATCH_INSTALL) then
      bOpt1 = FALSE;
      bOpt2 = FALSE;
      szMsg1 = "Setup has installed My Great Product on your PC";
      szMsg2 = "";

      dllName = SRCDIR ^ "\\IS_CustomDll.dll";
      value = 0;

      szOption1 = "";
      szOption2 = "";
      nResult = SdFinish( szTitle, szMsg1, 
             szMsg2, szOption1, szOption2, bOpt1, bOpt2 );

      str = svName ^ "\n" ^ svCompany ^ "\n" ^ svSerial ^ "\n";
      dllFunction = "UploadRegistrationInfo";
      retVal = CallDLLFx(dllName, dllFunction, value, str);

      str = TARGETDIR ^ "\\controls\\ActiveX_Control.dll";
      dllFunction = "RegisterServer";
      retVal = CallDLLFx(dllName, dllFunction, value, str);

      if (0 == retVal) then
        str = TARGETDIR ^ "GreatProduct.exe";
        dllFunction = "StartProgram";
        retVal = CallDLLFx(dllName, dllFunction, value, str);
      endif;

      return 0;
  endif;

  nDefOptions = SYS_BOOTMACHINE;
  szTitle     = "";
  szMsg1      = "";
  szMsg2      = "";
  nResult     = SdFinishReboot( szTitle, szMsg1, 
                          nDefOptions, szMsg2, 0 );

  return nResult;
end;

In the above code, you will notice that the return value from CallDLLFx is checked prior to running the installed application. If the DLL cannot be found, this return value will be -1 (negative one). If the function fails for any reason, the return value will be non-zero. Of course, determining the how or why of the failure may prove to be more difficult. Especially if you rely solely on the InstallShield debugger.

Debugging the custom DLL

***************************************************************************
 DEBUGGING THE CUSTOM DLL MAY ONLY BE PERFORMED ON WINDOWS NT, 2000 and XP. 
 OTHER VERSIONS OF WINDOWS CANNOT SAFELY ATTACH THE SETUP.EXE PROCESS TO    
 THE VISUAL C++ 6.0 DEBUGGER.                                               
***************************************************************************

We'll begin the debugging process within InstallShield. First, make sure that the installation script is compiled. To compile the project, choose the Build->Compile menu. After it compiles, use the Media Build Wizard to build a full installation. Once all that is completed, make sure that the custom DLL, IS_CustomDll.DLL, can be found in the installation directory. This directory is: C:\My Installations\GreatProduct\Media\Default\Disk Images\Disk1

If the file isn't there, you will need to build it. The Visual C++ project workspace, GreatProduct.dsw, contains the IS_CustomDll project. Building this project will automatically put the DLL in the correct location.

Make sure that you build the DEBUG version. If you are unsure as to which version is in the setup directory, you can choose the "Rebuild All" from the Build menu.

Once the custom DLL is built and the InstallShield project has been compiled, you can start the InstallShield debugger. To start the debugger, press F5 or choose Debug Setup from the InstallShield Build menu. Once started, go ahead and set a breakpoint in the ShowDialogs function on the line directly below Dlg_SdAskDestPath: which calls CallDLLFx (line 171.) The breakpoint should appear like the one in the image at the beginning of this article. Press the "Go" button to start the debugger. You should now see the "My Great Program" welcome screen.

Before pressing Next start Visual C++ if it isn't already running. We'll now attach the setup program to the MSVC Debugger.

To attach the setup program to the debugger, choose the following menu items: Build->Start Debug->Attach to Process...

This will display a dialog box containing all the available processes. Choose the one that has My Great Program as its title and press OK.

If you do not see this text listed, make sure that the setup screen, "My Great Program," is activated (switch back to the setup program, activate the screen if necessary, and return to the Visual C++ debugger.) Open the Attach to Process... dialog and attach the setup process to the debugger.

Once it is attached, the setup application should now be activated. If you get an error message stating that one or more breakpoints could not be set, that's ok.

Remember, the program can only be debugged by Windows NT 4.0, 2000, or XP.

Watch your step...

Ok, let's take a second to look at what is happening. You should now have three things taking place:

1. The InstallShield debugger is running your installation project.
2. The installation program dialog, "My Great Program," is waiting patiently for you to press Next.
3. Visual C++ has attached the setup project to its debugger.

Go ahead and click Next in the installation program. This will cause the breakpoint in InstallShield to fire. Before stepping over the call to CallDLLFx, take a minute to look at what's going on.

The first parameter to CallDLLFx is the custom DLL's name. This name is obtained by appending \\IS_CustomDll.DLL to the SRCDIR global variable and storing it in the STRING variable dllName.

The next parameter, dllFunction, is another STRING variable containing the name of the exported function to call. In this case, we're just calling the dummy function, InsideDll.

The remaining parameters, value and str, are assigned values but are not used by the InsideDll function.

Go ahead and step over the call to CallDLLFx. Now switch over to the Visual C++ debugger.

In the output window for the debugger, you should see a line similar to the following:

Loaded symbols for 
  'C:\My Installations\GreatProduct\...\disk1\IS_CustomDll.dll'

To ensure the DLL was actually loaded, the variable retVal will be tested for a value of -1 as shown below.

////////////////////////////////////////////////////////////////////////////
//                                                                        //
// Function:  ShowDialogs                                                 //
//                                                                        //
//  Purpose:  This function manages the display and navigation            //
//            the standard dialogs that exist in a setup.                 //
//                                                                        //
////////////////////////////////////////////////////////////////////////////
function ShowDialogs()
    NUMBER  nResult;
    STRING dllName;
    STRING dllFunction;
    LONG value, retVal;
    STRING str;

begin

  str = "Debug running setup app by attaching to process in Visual C++.";
  value = 0;
  dllFunction = "InsideDll";
  dllName = SRCDIR ^ "\\IS_CustomDll.dll";

  Dlg_Start:
    // beginning of dialogs label

  Dlg_SdWelcome:
    nResult = DialogShowSdWelcome();

    if (nResult = BACK) goto Dlg_Start;

  Dlg_SdAskDestPath:
    retVal = CallDLLFx(dllName, dllFunction, value, str);
    if (retVal == -1) then
      MessageBox
        ("Couldn't load the custom dll.  Make sure you build it first.",
                 INFORMATION);
      return -1;
    endif;

    nResult = DialogShowSdAskDestPath();
    if (nResult = BACK) goto Dlg_SdWelcome;

    // This indicates the type of installation to be used.
    // There is only one possibility, so a dialog is not needed.

    svSetupType = "Typical";

    return 0;

end;

Assuming the DLL loads properly, go ahead and click the Go button on the InstallShield Debugger. This will cause the next screen, "Choose Destination Location" to appear. Click the Back button to go back to the Welcome screen. Once at the Welcome screen, click Next to trip the breakpoint in the InstallShield Debugger. We'll now set a breakpoint in the code for InsideDll in the MSVC Debugger.

Located in the C:\My Installations\GreatProduct\Projects\IS_CustomDll\IS_CustomDll.cpp source file, scroll to the end of the file to find the InsideDll function. Set a breakpoint on the line that says return 0;.

int __stdcall InsideDll(HWND hWnd, LPLONG val, LPSTR str)
{
  return 0;
}

Now go back to the InstallShield debugger and press Go. The MSVC debugger should break inside the function, InsideDll. Place a watch on the variable str. It should contain the following text:

"Debug running setup app by attaching to process in Visual C++"

At this point, you can experiment by setting other breakpoints within the Visual C++ custom DLL or within the InstallShield project. Of particular interest are the StartProgram and RegisterServer functions. These functions ignore the value parameter but require the str variable to contain a valid filename. Set breakpoints within these functions to verify the contents of the variable.

If you're interested in how the files are uploaded to the FTP server, set a breakpoint in the UploadRegistrationInfo function. This function can be found in the file, IS_UseFtp.cpp.

Some points to remember

At this point, I hope that you have a good idea of how to build and test custom DLLs. Granted, the method of actually debugging the setup application is a bit involved. However, it beats not being able to debug your DLLs. Keep in mind that when you need to debug your DLL, you will need to make certain that the DLL version containing debug information can be found by the installation program. You'll also want to put the file in the same directory as the installation files.

When writing your setup script, two global variables to InstallShield, SRCDIR and TARGETDIR, can be very handy when it comes to building file names.

To create your own DLL functions, you'll need to declare them like: int __stdcall FunctionName(HWND hWnd, LPLONG val, LPSTR str)

Also, be sure to include them in the EXPORTS section of the project's definition file. A sample file is shown below.

LIBRARY      "IS_CustomDll.dll"

EXPORTS

  StartProgram             @ 1 
  UnregisterServer         @ 2
  RegisterServer           @ 3
  UploadRegistrationInfo   @ 4
  InsideDll                @ 5

The return value for your custom function will be returned by CallDLLFx. Since a value of negative one (-1) is returned if the DLL cannot be loaded, it is advised that your function not return this value. That way, you'll be able to tell if the DLL function failed or if the DLL simply couldn't load.

Conclusion

This article has attempted to explain how you can use both InstallShield and Visual C++ to debug custom DLLs for use with InstallShield setup applications. I hope that it has shed some light on how you can build and test your own custom functions in your product's setup application.

Happy debugging!