Restricting DLL loading

Background

During the development of a shell extension, you have little control over when your extension can and cannot be loaded. This becomes rather annoying as you try to rebuild and are prevented from doing so because the file is in use. There does exist a set of registry keys to help with this problem, but they don't exactly make the process developer friendly.

The solution I'll present is simple and effective. I've been using it for some time and thought I'd share it with the community.

The Solution

The shell loads all extensions by calling the KERNEL32.DLL function, LoadLibrary, with the path of the DLL as registered in the registry. LoadLibrary goes through the process of loading the DLL into the address space of the calling process, typically explorer.exe with shell extensions, resolving other library imports, and finally calling the DLL's entry point function, DllMain. DllMain performs any initialization and returns a BOOL value of TRUE or FALSE to indicate success or failure. If FALSE is returned, the system immediately detaches the DLL from the process, unloads and closes it, and returns NULL to the caller of LoadLibrary to indicate that the DLL was not loaded successfully. So with this knowledge, we have a sure fire way of preventing a DLL from being loaded by a process. The remaining trick is to specify exactly which processes, or more specifically, which modules can or cannot load the DLL. Enter IsDebuggerPresent and GetModuleHandle.

Being that you are the one developing the DLL, you have a good idea of exactly which modules you do or do not want loading your DLL. For one, it is a fairly safe assumption that if the calling process is being debugged, then you want your DLL to be loaded. But there are also some processes that need to load the DLL, such as REGSVR32.EXE, which typically do not run in the context of a debugger. To support both cases, we have two functions available from KERNEL32.DLL. IsDebuggerPresent, which returns TRUE when the calling process is being debugged and FALSE otherwise. And GetModuleHandle which returns the HMODULE of the specified module if it is loaded in the process, and NULL otherwise. So with these two functions, we create a rule that states:

If the process is being debugged, or if the process contains a known module, allow DllMain to succeed and return TRUE, otherwise fail DllMain and return FALSE.

The Code

• Part 1 - Known Modules

  We first need to create a list of modules that are permitted to load the DLL in all contexts. This will include applications such as REGSVR32.EXE so that the DLL can use its registration features, other applications which might need to load the DLL for things such as resource acquisition, or other modules which utilize the DLL as part of a package or feature, such as DLLHOST.EXE or SVCHOST.EXE.

  /*********************************************************
   *  _szUnrestrictedModules
   *      Modules which can load this library.
   */
  LPCTSTR _szUnrestrictedModules[] =
  {
      _T("REGSVR32.EXE"),
      _T("SVCHOST.EXE")
  };

• Part 2 - Compiler control

  Next, we want a compiler flag to disable this feature for release builds or for when we do not want to restrict the DLL from loading.

  /*********************************************************
   * When the _LOADRESTRICTIONS_ON flag is defined, DllMain 
   * succeeds for the following conditions:
   *  1) The process is being debugged.
   *  2) One of the modules listed in _szUnrestrictedModules 
   *     is loaded in the process.
   * 
   * This prevents apps like the shell from loading and 
   * locking the module, preventing new builds. The 
   * _szUnrestrictedModules is needed so that non-debug apps 
   * like regsvr32 can load the module.
   * 
   * Defining _LOADRESTRICTIONS_OFF will disable this 
   * functionality.
   */
  #ifndef _LOADRESTRICTIONS_OFF
      #define _LOADRESTRICTIONS_ON
  #endif

• Part 3 - Validation

  Finally, we want a function to perform the restriction checks.

  /*********************************************************
   *  _IsUnrestrictedProcess
   *      Checks if the current process is allowed to load 
   *      this module.
   */
  BOOL __stdcall _IsUnrestrictedProcess( ) throw()
  {
    if ( ::IsDebuggerPresent() )
    {
      return ( TRUE );
    }
  
    for ( size_t idx = 0; idx < 
          (sizeof(_szUnrestrictedModules) / sizeof(LPCTSTR));
          idx++ )
    {        
      /* Check for the module by attempting to retrieve 
       * its HMODULE */
      if ( NULL != 
           ::GetModuleHandle(_szUnrestrictedModules[idx]) )
      {
          return ( TRUE );
      }
    }
   
    return ( FALSE );
  }

Now, all that remains is to call the _IsUnrestrictedProcess function from DllMain and succeed or fail accordingly.

extern "C" BOOL WINAPI DllMain( HINSTANCE hInstance,
                                DWORD dwReason, 
                                LPVOID lpReserved )
{
    /* Do load restrictions, if enabled */
#ifdef _LOADRESTRICTIONS_ON
  if ( !_IsUnrestrictedProcess() )
  {
    return ( FALSE );
  }
#endif

  /* Perform other initialization here if necessary */

  return ( TRUE );
}

And there we have it. Adding this code to a shell extension DLL prevents it from being loaded except for those conditions that we specify. This allows rebuilds without waiting for the system to unload the DLL.