Click here to Skip to main content
Click here to Skip to main content

Visualizing the Windows Mobile Virtual Memory Monster

, 6 Jan 2009 CPOL
Rate this:
Please Sign up or sign in to vote.
This article presents VirtualMemory.exe, a memory monitor application that visualizes Windows Mobile's virtual memory model graphically, and thus allows common memory issues, such as the infamous DLL Crunch, ordinary leaks, and a full device.exe, to be identified quickly and easily.

Introduction

Windows Mobile's memory model is very different from that of most 32 bit Operating Systems. A typical 32 bit Operating System allows large numbers of applications to access gigabytes of virtual memory--often much more than the physical amount of memory on the system. Windows XP, for example, allows scores of processes two gigabytes of virtual memory each. On Windows Mobile, however, only 32 applications can exist, and each can normally only access 32 megabytes of virtual memory--much less than the physical amount of memory on an average modern device.

Of course, there is an understandable reason for doing things this way--performance. Ten years ago, Windows Mobile devices had wimpy processors and very little memory. It made sense to limit the amount of virtual memory an application could access if the processor didn't have to do as much work. Today, however, processors are powerful, and memory is cheap and plentiful.

As a result, this decade-old performance decision has now been turned on its head. Instead of helping the system, today Windows Mobile's memory model can cause headaches that are often difficult to debug. Applications may encounter out of memory errors when there is plenty of free physical memory available, or features may fail to load. Adding to the confusion, other programs running on the system or features added by the OEM device maker or cell carrier can directly cause these issues to happen.

Microsoft, of course, provides many diagnostic tools such as DumpMem to analyze virtual memory usage and Application Verifier to track leaks. The log files they produce are extremely detailed. But, when the problem involves the interaction of multiple programs, it can be difficult to pinpoint the issue from the pages of hexadecimal output. This article presents VirtualMemory.exe, a memory monitor application that visualizes Windows Mobile's virtual memory model graphically, thus allows common memory issues, such as the infamous DLL crunch, ordinary leaks, and a full device.exe, to be identified quickly and easily.

Background

A full treatment of the Windows Mobile memory model is well beyond the scope of this discussion. For details, please consult the following excellent blog entries (the inspiration for this article's title):

Graphing the Output

VirtualMemory.exe's Display

As the image above shows, VirtualMemory.exe displays a column for every process slot on the device. Slot 0 is assigned to the currently running process, and thus is not graphed. This leaves 32 remaining slots. Slot 1 contains DLLs that are stored in ROM, and is usually full. The next few slots are normally system processes such as filesys.exe, device.exe, gwes.exe, and others that load early at boot time. After that are user application programs. At the bottom of each column are the lowest virtual memory addresses starting at 0. This is where program code is located and upward growing heap space is allocated. At the top of the column are the highest virtual memory addresses ending at 32 megabytes, where DLLs reside and load downward.

Obviously, there are not enough pixels on a Windows Mobile device display to readably show the status of every individual virtual memory block. In order to cope with this, the color of each tic in a column is mixed according to the memory blocks it represents. Red is completely empty. Blue means that the memory space has been reserved. Green means the memory space has been committed. A mixture of colors means some combination of all three. As expected, most of the display is red--free virtual memory. There is also plenty of blue (reserved) and some green (committed). Remember that Windows CE does not commit address spaces containing code until they are accessed. To speed screen refreshes, drawing is done to an off screen bitmap and then blitted to the display.

Using the Application

Starting the memory monitor will show the current virtual memory status of the device. The left and right navigation buttons highlight the current slot, and display the corresponding process name. Clicking the Snapshot soft button will refresh the view of virtual memory. Clicking the Exit soft button quits the application.

Under the Hood

Profiling Virtual Memory

In order to see how virtual memory is being used across the device, the VirtualQuery system call is used. As shown below, it returns a data structure for every contiguous section of memory of the same allocation type. To make things easier for the graphing routine, these data structures are flattened to make an array detailing the status (free, reserved, or committed) for each block of memory in every slot.

void GetVirtualMemoryStatus(VIRTUALDATA *pvd)
{
  MEMORY_BASIC_INFORMATION mbi;
  int idx;
  DWORD addr;
  BYTE state;

  memset(pvd->pageAllocated,0x00,sizeof(pvd->pageAllocated));

  for(idx=STARTBAR;idx<STARTBAR+NUMBARS;idx++)
  {
    DWORD offset;

    addr = idx * 0x02000000;

    for( offset = 0; offset < 0x02000000; offset += mbi.RegionSize )
    {
      unsigned int i;

      memset(&mbi,0x00,sizeof(MEMORY_BASIC_INFORMATION));

      if(VirtualQuery( (void*)( addr + offset ), &mbi, 
          sizeof( MEMORY_BASIC_INFORMATION ) )==0)
        break;

      state=(BYTE)((mbi.State>>12)&(VMEMCOMMIT|VMEMRESERVE));

      if(state)
      {
        for(i=(offset)/4096;i<(offset+mbi.RegionSize)/4096;i++)
          pvd->pageAllocated[idx-STARTBAR][i]=state;
      }
    }
  }
}

Getting Process Names

Identifying which slot has a problem isn't enough--to make things easier for the developer, the name of the process occupying that slot should be determined as well. The routine below gets that information by using the CreateToolhelp32Snapshot and OpenProcess system calls. The slot for the process is found by dividing the base address by 32 megabytes.

void GetProcessNames(VIRTUALDATA *pvd)
{
  HANDLE hProcessSnap;
  HANDLE hProcess;
  PROCESSENTRY32 pe32;
  int slot;

  for(slot=STARTBAR;slot<STARTBAR+NUMBARS;slot++)
    swprintf(pvd->szExeName[slot-STARTBAR], 
             TEXT("Slot %d: empty"),slot);

  if((1-STARTBAR)>=0)
    wcscpy(pvd->szExeName[1-STARTBAR], 
              TEXT("Slot 1: ROM DLLs"));

  // Take a snapshot of all processes in the system.
  hProcessSnap = 
    CreateToolhelp32Snapshot(TH32CS_SNAPPROCESS|TH32CS_SNAPNOHEAPS, 0 );
  if( hProcessSnap != INVALID_HANDLE_VALUE )
  {
    memset(&pe32,0x00,sizeof(PROCESSENTRY32));
    pe32.dwSize = sizeof( PROCESSENTRY32 );

    if( Process32First( hProcessSnap, &pe32 ) )
    {
      do
      {
        hProcess = OpenProcess(PROCESS_QUERY_INFORMATION, 
                               FALSE,pe32.th32ProcessID );
        if( hProcess != NULL )
        {
          slot=pe32.th32MemoryBase/0x02000000;

          if(slot-STARTBAR<NUMBARS)
            swprintf(pvd->szExeName[slot-STARTBAR],TEXT("Slot %d: %s"),
              slot,pe32.szExeFile);

          CloseHandle( hProcess );
        }
      } while( Process32Next( hProcessSnap, &pe32 ) );
    }

    CloseToolhelp32Snapshot( hProcessSnap );
  }
}

If the device being used has a two tier security model (as is the case on most Windows Mobile standard devices), CreateToolhelp32Snapshot may not return the attributes for all processes on the system unless the requesting application is privileged signed. VirtualMemory.exe is signed by the privileged SDK test certificate which allows it to retrieve all process names on all emulator images. However, this certificate isn't installed on actual devices. To put it there, SdkCerts.cab from the platform SDK must first be installed on the device to put the SDK certificates in the root store. Otherwise, VirtualMemory.exe will not be able to obtain all the process names for every slot.

Common Problems

The DLL Crunch

Most applications use dynamically linked libraries (DLLs) in order to share code between applications. To make it easier for the CPU to share DLLs among processes, Windows Mobile tries to allow virtual address space for a DLL in every process on the system, whether the process uses the DLL or not. This means that one process's DLLs can reduce the virtual memory in every other process. Thus, most processes have much less than 32 megabytes of virtual memory available even before they start running. A lot of it has already been eaten by the DLLs of other processes.

This shrinking of virtual memory can cause a problem known as the DLL Crunch. As more and more applications have DLLs run, the ceiling of free virtual address space keeps lowering. Eventually, an application's heap, which grows upward, may collide with this ceiling. When that happens, there isn't any more virtual memory available for that process. Allocations fail, and DLLs won't be able to load, even though there is still plenty of physical memory on the system.

The image at the beginning of this article shows the problem. The last 6 slots have been loaded with programs that require three different, large DLLs. Starting with the ceiling established by the programs that were loaded earlier, LoadLibrary puts these three DLLs (the large blue blocks) progressively lower in the address space (in case any other program wants to use it). Eventually, there isn't any more address space left for any new DLLs to load. Note that it is still possible to create more processes (at least until the limit of 32 is reached) using the same DLLs. As can be seen in the image, second instances of the same first three programs loaded successfully. This is because Windows CE reserves space across all processes for every DLL (the reason for the DLL Crunch in the first place!). A common way to avoid the DLL Crunch is to statically link your application and to avoid using DLLs entirely.

Leaks

At the bottom of the virtual address space is executable code, and above that is the heap. As a program allocates more memory, the heap gradually grows upwards. A normally operating program should use a relatively constant amount of memory once it is running. However, if the heap keeps growing over successive snapshots in VirtualMemory.exe, that is a good sign there is a leak. Eventually, the heap will hit the DLL ceiling, and the application will run out of virtual address space. Track down these leaks using Application Verifier and fix them!

Device.exe is Full

One of the biggest casualties of the 32 megabyte virtual memory limit is device.exe. Device.exe is the Windows Mobile process that is in charge of loading all the device drivers on the system. This means that drivers that come from Microsoft, the OEM device maker, the cell carrier, and ISVs, all have to fit in the same 32 megabytes. Of course, drivers vary from device to device according to what the OEM maker and the cell carrier decide to install, and so the amount of free virtual memory available for third party drivers varies as well. As a result, an ISV application that requires a driver may work fine on one device, but may not even load on another!

The image at the beginning of this article shows the slot for device.exe on a Windows Mobile 5 emulator image. As you can see, it is relatively empty. However, real world devices include cameras and other memory hogs. A multi-megapixel camera with its large video buffers can easily consume the remaining free virtual memory. As a result, an ISV that loads an after market driver using the following may encounter a FILE_NOT_FOUND error:

hISVdriver = ActivateDevice(TEXT("Drivers\\ISVdriver"), 0);

This error is a bit misleading. The driver file is there, but the memory space to accommodate it in device.exe is not. The only way to avoid this is to carefully profile every device from every carrier that you wish to support. Some may not have much spare memory left.

The Future

As Windows Mobile devices have become more powerful, virtual memory problems have become much more common. As a result, it doesn't make sense for OEMs to put more physical memory on a device or a bigger camera, because these only make virtual memory issues worse. Luckily, though, help from Microsoft is on the way.

Windows CE 6 has a completely different memory model from Windows CE 5 (which is the basis for Windows Mobile 5 and 6). Instead of 32 processes with 32 megabytes of virtual memory each, the number of applications is 32,000, and each will be able to access a full two gigabytes of virtual memory. For the developer, this means that virtual memory issues should completely disappear with the introduction of Windows Mobile 7. For the user, it also means much more powerful devices with much, much more memory should be on their way!

Points of Interest

VirtualMemory.exe was created to diagnose an issue loading a device driver on a certain cell phone. All other devices were working correctly except for this particular model, which returned a FILE_NOT_FOUND error. The problem turned out to be a completely full device.exe, as a result of the additional code loaded by the cell carrier.

History

  • January 6, 2009 - Initial version (WJB).

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Dr. William J. Blanke
Architect
United States United States
I specialize in win32 security software (custom PGP SDK based applications) and mobile development (iOS, Windows Mobile, Windows Phone, and Android). I developed security software for most of those platforms at PGP Corporation (before it was acquired by Symantec in 2010). Previously, I was a lecturer of Computer Science at the University of the South Pacific, in Suva, Fiji Islands.
 
My LinkedIn profile
 
Track us in South America!
 
Twerbil for Android
Twerbil for Windows Phone

Comments and Discussions

 
QuestionDo you have an idea Pinmemberkecoli29-Sep-12 10:43 
GeneralSignTool Error PinmemberKilo72-Aug-12 8:47 
GeneralRe: SignTool Error PinmemberDr. William J. Blanke21-Aug-12 7:40 
GeneralMy vote of 5 Pinmemberhjgode8-Apr-11 7:38 
General+5 PinmemberPaul Heil6-Aug-10 4:28 
GeneralMy vote of 5 Pinmemberroc19628-Jun-10 21:44 
GeneralI don't understand the memory model... Pinmemberpribylf5-Mar-10 1:17 
GeneralRe: I don't understand the memory model... Pinmemberandyturner8-Mar-10 0:28 
GeneralRe: I don't understand the memory model... Pinmemberpribylf8-Mar-10 3:22 
GeneralRe: I don't understand the memory model... PinmemberMike Dimmick24-Mar-10 8:40 
QuestionGreat app, but... [modified] Pinmemberandyturner7-Feb-10 23:46 
AnswerRe: Great app, but... Pinmember__PPS__8-Feb-10 9:43 
GeneralThe DLL Crunch!!!! Pinmembersoupcai28-Sep-09 2:17 
GeneralRe: The DLL Crunch!!!! PinmemberDr. William J. Blanke1-Oct-09 11:14 
GeneralRe: The DLL Crunch!!!! [modified] Pinmembersoupcai13-Oct-09 17:35 
GeneralExcellent, but needs touch selection control PinmemberRobRedbeard772-Apr-09 18:18 
GeneralRe: Excellent, but needs touch selection control Pinmemberinnodb31-Jul-09 16:30 
GeneralGood Job, really educating -- Pinmembermaximus12321-Feb-09 18:29 
GeneralGreat PinmemberHouser24-Feb-09 1:37 
GeneralDevice Applicability PinmemberJeffrey Walton19-Feb-09 11:10 
GeneralRe: Device Applicability PinmemberDr. William J. Blanke19-Feb-09 14:37 
GeneralGreat Article PinmemberMember 367094117-Feb-09 5:34 
GeneralAgain, a great article. PinmemberPoiuy Terry15-Jan-09 12:57 
GeneralRe: Again, a great article. PinmemberDr. William J. Blanke19-Jan-09 4:38 
AnswerRe: Again, a great article. PinmemberPoiuy Terry19-Jan-09 14:48 
GeneralRe: Again, a great article. PinmemberDr. William J. Blanke19-Jan-09 16:20 
Right, I was trying to just keep things simple with the WM_SIZE handler. Your fix sounds good. Probably the best thing to do there would be to multithread the redraw so the app would come up immediately, maybe with just a splash screen on startup. For later WM_SIZE events, you could stretch the previous plotted bitmap to fit the screen in the meantime.
 
BTW Using SetPixel isn't the best solution either, but again it was short and simple. DirectDraw would be faster and you might not even notice a delay then.
 
As far as expanding/improving it, go for it! I wish I had the time Smile | :) . I think a lot of developers would find it useful.
 
Bill
GeneralSource code Pinmemberskanib7-Jan-09 8:06 
GeneralRe: Source code PinmemberDr. William J. Blanke7-Jan-09 8:21 
GeneralGood Article PinmemberMember 41163787-Jan-09 7:33 
GeneralRe: Good Article PinmemberDr. William J. Blanke8-Jan-09 3:08 
GeneralVery Insightful PinmemberJoel Ivory Johnson6-Jan-09 10:30 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.141015.1 | Last Updated 6 Jan 2009
Article Copyright 2009 by Dr. William J. Blanke
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid