Click here to Skip to main content
11,584,443 members (65,930 online)
Click here to Skip to main content

Yet Another Thread Monitor

, 8 Jan 2004 CPOL 94.1K 4K 71
Rate this:
Please Sign up or sign in to vote.
This article discusses the use of Asynchronous Procedure Calls for Kernel/User mode communication.

Thread Monitor in action.

<!------------------------------- STEP 3 ---------------------------><!-- Add the article text. Please use simple formatting (

,

etc) -->

Introduction

Thread Monitor presents a view of process and thread execution with a bit more granularity than Task Manager, and uses the sparsely documented Asynchronous Procedure Call (APC) mechanism to communicate the kernels creation and deletion of threads and processes to the user mode app.

The idea of communicating process and thread activity has been covered here at CP in the article by Ivo Ivanov, "Detecting Windows NT/2K process execution". The difference here is in the use of APCs for communication. While there are other references on the web dealing with APCs, I didn't find any with a working sample, so this may fill a gap.

The Thread Monitor app provides an interesting view of process activity and may be useful for debugging or logging purposes, but the main point of the exercise is to demonstrate an efficient and reliable method of updating a user mode app with information originating from kernel mode - without requiring polling the device or waiting on a shared (kernel/user) synchronization object.

Running the Sample

The driver (_TestDrv3.sys) in its present form is intended for an Intel x86 box running Windows 2000 and XP. With some small tweaks and cleanup it should run on NT 4, which was the original platform for my _testDrv1 skeleton. (The debug version traces out some system structures that have changed in NT 5).

The demo contains the driver (_TestDrv3.sys), a loader (_TestDrvClient1.exe), and the Thread Monitor program (Console.exe). The naming conventions belie the fact that these modules evolved from my explorations of kernel mode stuff in general. You'll find a few bits of code in the project files that don't directly relate to the operations described here, but the release version of the driver should be fairly clean, aside from maybe a few speaker beeps.

The Thread Monitor app (Console.exe) doesn't attempt to load the driver. I found during testing and development that it was cleaner to separate this out into a utility app, which is basically where I left _TestDrvClient1.exe. _TestDrvClient1 can load a driver (given a knowledge of the DOS device names chosen) and perform simple Open, Read, and Close operations. (This is about all _TestDrv1.sys was set up to do).

Loading the driver.

Here's the _TestDrvClient1 app ready to Install the device as TestDriver3 (this name is used to open the device in the the Console sample). Usually, during testing, I hit Install, Start, and Remove, with Demand start up, then exit the app. The device cannot be stopped before a shutdown occurs because the callbacks registered with PsSetCreateProcessNotifyRoutine and PsSetCreateThreadNotifyRoutine can't be unsubscribed to. By routinely selecting Remove, I know the device is marked for deletion and the registry will be clean at next start up. The device can also be set to load at system start up, in which case it should not be marked for deletion. You might want to do this if you will be running the sample (or using the CMonitor class yourself) from an account that does not have admin rights to load drivers.

To test Open and Read, use a Name of \\.\TDRV3. Read is implemented to return the name of the device, which is displayed in a message box. You won't want to thrill to this experience more than once.

Once the driver is in place, you should be able to invoke the Thread Monitor app through Console.exe.

Thread Monitor maintains a list of running apps and tracks thread creation and deletion for each. Processes that have exited are not automatically removed from the list, which helps expose scheduled background processes (virus checkers and the like). There is also a console window that traces process and thread creation details. You can use the Options menu to restrict trace activity to a selected process.

Structure

The Thread Monitor application (Console.exe) uses a small class (CMonitor) which in turn relies on the device driver (_TestDrv3.sys) for information on process and thread creation and deletion.

The _TestDrv3 driver subscribes to the process and thread callbacks through calls to PsSetCreateProcessNotifyRoutine and PsSetCreateThreadNotifyRoutine.

The CMonitor class creates a thread and subscribes to the notifications through a call to DeviceIoControl, passing a pointer to a static member function (CMonitor::ApcCallback) to be invoked through the APC mechanism. It then waits on an event (which the driver needs no knowledge of). When apc callback(s) are invoked, the wait will return with WAIT_IO_COMPLETION, which we can just ignore and re-invoke the wait. The callback(s) queued will execute without any intervention. If the wait completes with WAIT_OBJECT_0, the thread exits. This allows us to terminate the thread by setting the event. When the thread exits, code in the drivers thread notify routine un-subscribes it.

Users of the CMonitor class can call CMonitor::Subscribe with a valid HWND to receive notifications in the form of windows messages, or with NULL to trace raw info to stdout.

Implementation

Central to the drivers operation are calls to KeInitializeApc and KeInsertQueueApc. These calls are not prototyped in the DDK, so they're declared in TDExtern.h as follows:

NTKERNELAPI
VOID
KeInitializeApc (
    IN PRKAPC Apc,
    IN PKTHREAD Thread,
    IN KAPC_ENVIRONMENT Environment,
    IN PKKERNEL_ROUTINE KernelRoutine,
    IN PKRUNDOWN_ROUTINE RundownRoutine OPTIONAL,
    IN PKNORMAL_ROUTINE NormalRoutine OPTIONAL,
    IN KPROCESSOR_MODE ApcMode,
    IN PVOID NormalContext
    );
 
// note some texts give this as void?? (and will link as such)
BOOLEAN 
KeInsertQueueApc(
    PKAPC Apc,
    PVOID SystemArgument1,
    PVOID SystemArgument2,
    UCHAR mode);    // guessing this is Kernel, User, 
                    // or Maximum (0,1,2) 
                    // 11/24/03 - Albert Almeida [AA]
                    // gives this last param as 
                    // IN KPRIORITY Increment

I've seen more than one prototype of these functions in browsing the references listed below. As the comments show, the last parameter to KeInsertQueueApc could well be a priority boost increment for the thread executing the APC. I haven't been able to verify this, and just pass in 0 in the actual call.

Once a thread has subscribed to the driver, code inside the drivers callbacks for the process and thread notification routines calls TDrv3_QueueApcToClientThread to set up the APCs that will fire the CMonitor::ApcCallback:

/*******************************************************************
  TDrv3_QueueApcToClientThread

  Allocates, initializes, and queues an APC for the client thread
  with the arguments passed in. 

  Returns FALSE on failure.
 *******************************************************************/
BOOLEAN  TDrv3_QueueApcToClientThread(PVOID SystemArgument1,
                                      PVOID SystemArgument2) {

    BOOLEAN retval = TRUE;
    struct _KAPC    *pApc;

    // DBGPRINT(("%s: entered TDrv3_QueueApcToClientThread\n", SYSNAME));

    pApc = ExAllocatePool( NonPagedPool, sizeof(struct _KAPC));

    if(NULL == pApc) {
        ASSERT(pApc);
        retval = FALSE;
    }
    else {
        KeInitializeApc(pApc, g_pKThreadClient, 
            OriginalApcEnvironment, // KAPC_ENVIRONMENT: one of Original, 
                                    // Attached, or Current - see [<A href="http://www.windevnet.com/documents/s=7653/win0211b/0211b.htm">AA</A>]
            &TDrv3_KernelApcRoutine,// kernel APC routine
            NULL,                   // rundown APC routine
            g_KMessageInfo.pFn,     // user APC routine 
            UserMode,               // ApcMode 
            (PVOID) NULL);

        retval = KeInsertQueueApc(pApc, SystemArgument1, 
                    SystemArgument2, 0);
    }

    if(FALSE == retval) {
        DBGPRINT(("%s: KeInsertQueueApc failed...\n", SYSNAME));
    }

    return retval;
}

TDrv3_QueueApcToClientThread allocates a KAPC structure from the non-paged pool and initializes it with pointers to a kernel mode function (whose only task is to delete the apc object) and a user mode function (which was passed in by the subscribing thread). We could add a rundown routine as well, but the apc will be deleted by the OS if the thread terminates with apcs queued.

Here's the minimal kernel apc routine:

/*******************************************************************
  TDrv3_KernelApcRoutine

  Just delete the apc.

 *******************************************************************/
VOID TDrv3_KernelApcRoutine( IN struct _KAPC *Apc, 
         IN OUT PKNORMAL_ROUTINE *NormalRoutine, 
         IN OUT PVOID *NormalContext, IN OUT PVOID *SystemArgument1, 
         IN OUT PVOID *SystemArgument2 ) 
{
    // DBGPRINT(("%s: entered TDrv3_KernelApcRoutine\n", SYSNAME));
    ExFreePool(Apc);
}

After initialization, the apc is inserted into the queue with the data to be passed to the functions (SystemArgument1 and SystemArgument2). The KTHREAD object (pointed to by g_pKThreadClient) was set up during the subscription process in TDrv3_DeviceControl().

That about covers the kernel mode code for queuing APCs. Note that no attempt is made to synchronize data access or serialize the queuing of APCs. We basically work with opaque and newly allocated objects, and let the OS take care of the details of re-entry etc. Synchronization objects used inside this kind of callback processing could create a bottleneck in high activity situations, and I think its best to avoid them. That said, I haven't tested this on a multi-processor machine.

I can't provide a full explanation of exactly how all the parameters affect the dispatch of the APC, or their intended use in the APC routine. One of the most interesting (and probably most complex to use) is the NormalContext. I think some drivers create completion routines that need to run in a non-arbitrary context (most probably one of a thread created in kernel mode) and that this parameter allows for a context switch, after which(?) the NormalRoutine can be invoked. _TextDrv3 (like most drivers) just runs in whatever thread context happens to invoke the thread/process callbacks, and does not use these parameters.

Here's my (very) simplified take on what the OS does with this type of APC (keep in mind that this is a User mode APC, as opposed to a Kernel or Special mode APC - you'll find more on these in the refs below, notably AA.)

User mode APCs are only delivered to a thread in an alertable state. (Special mode APCs don't have this restriction, but are kernel mode APCs). Our subscribing thread (whose KTHREAD object is used to initialize the APC) is waiting on an event object in user mode, making it alertable. The event object however doesn't have to be signalled for an APC to be delivered. When the OS gets around to allowing the thread it's quantum of runtime, it checks the threads APC queue (actually, there are two APC queues associated with a thread, see AA), and if there are user mode APCs to be delivered, and the thread is alertable, the OS will, for each PKAPC in the queue, call the user mode function and then the kernel mode function (if any) for each of them on a first in first out basis. (Note that the kernel mode function is probably optional, but it gives us a way to free the KAPC allocation. Another strategy would be to work from a pool of pre-initialized APCs, queuing those whose Inserted member is false, but this comes up against the headache of managing the pool size and other unknowns. Life is simpler this way.)

Once the APCs have been delivered, our thread picks up where it left off - in our case, returning from the WaitForSingleObjectEx call with a return code of WAIT_IO_COMPLETION, whereupon it mumbles something about a disturbance in The Force and goes back into its wait. The thread procedure of the CMonitor class shows the call to subscribe and the event wait:

/*********************************************************
    ApcThreadProc()

    The _TestDrv3.sys driver communicates by queueing APCs
    to a subscibing thread. User mode APCs are delivered
    to alertable threads, so will dedicate one as such.
    Also, it's nice to use a thread that doesn't have a 
    window, to avoid the problems hinted at in the docs 
    for SleepEx().

 *********************************************************/
unsigned CMonitor::ApcThreadProc( void* pThis )
{
    // std::cout << "!!" << std::endl;

    unsigned int threadexitcode = 0;

    // cast our pointer
    CMonitor* that = ((CMonitor*)pThis);

    // invoke the driver - note that life is a lot easier in driver
    // land if the thread which is to receive the apc callbacks 
    // is the one that invokes the subscribe IOCTL, because 
    // KeGetCurrentThread() seems to be the only way to grab 
    // the KTHREAD needed for the apc initialization...
    if(false == that->KernelSubscribe()) {
        threadexitcode = 1;
    }
    else {
        // The rest of this thread proc is devoted to
        // making thread alertable so that user mode 
        // apcs queued in kernel mode can be delivered...

        // we'll use a bare bones event...
        that->m_hEvent = CreateEvent(NULL,  // security attributes
            FALSE,  // manual reset
            FALSE,  // initial state non-signaled
            NULL);  // unnamed event

        // check handle...
        if( NULL == that->m_hEvent) {
            // poop...
            that->m_nLastError = GetLastError();
            TRACE(_T("CreateEvent failed"));
            threadexitcode = 2;     // will need to examine 
                    // exit code thread in subscribe method...
        }
        else {
            DWORD dwWaitResult;
            do {
                dwWaitResult = WaitForSingleObjectEx(that->m_hEvent, 
                  INFINITE, TRUE);
            } while (dwWaitResult != WAIT_OBJECT_0);
            // This call will return with a code of 
            // WAIT_IO_COMPLETION when an apc is delivered.
            // The call returns with WAIT_OBJECT_0 when the 
            // event is signalled, so we can use 
            // SetEvent to kill the thread.
        }
    }
    _endthreadex( threadexitcode );

    return 0;
}

The APCs cause the code in CMonitor::ApcCallback (declared as a static __stdcall member function) to fire, which will parse out the type of the notification from the parameters and notify the subscribing HWND (or just print to stdout):

void CMonitor::ApcCallback(PVOID NormalContext, 
  PVOID  SystemArgument1, PVOID SystemArgument2) {
    ...
}

Et voila. Disregarding the fact that I'm using a couple of less-than-fully-documented functions, and generally dealing with forces I cannot possibly fully comprehend, I am trying to play by the rules as I understand them. I think using APCs in this way is a good approach to kernel/user mode communication, and it seems to work very reliably.

Using the code

The VS.NET solution contains two driver projects and three user mode applications. You shouldn't need the DDK to compile the user mode apps _TestDrvClient1, Console (the Thread Monitor app), and Console3.

To compile _TestDrv1.sys and _TestDrv3.sys, you'll need the DDK for the NT 4/5 target of your choice. I've found that getting other folks driver code to compile can be a bit of a hassle, but hope that all you'll need to change in the project settings are the include and library paths for the DDK. If you're setting up for NT4 you'll find that these paths further diverge into chk and free.

_TestDrv1 is a fairly minimal driver 'skeleton' that was my first attempt at a driver, and which was used as a template for experiments that followed. I include it here for those who would like to experiment with something simple.

If you'd like to use the CMonitor class or a variation thereof in your own code, you'll find the Monitor.cpp and .h files in the TDCommon directory. You can include them from there, though you may find you need to open the properties dialog for Monitor.cpp and give it an include path for your stdafx.h. You'll also need to reference the TD3defs.h file to use and/or change the windows message defines that your subscribing window will receive, and and set up the handlers accordingly.

Basically, all you need to do is create an instance of CMonitor and call Subscribe(HWND). The small Console3 app shows an example of using this call in _tmain with NULL for an HWND, letting the class trace the raw parameter data to stdout:

        CMonitor    monitor;

        if(!monitor.Subscribe(NULL)) {
            int result = monitor.GetStatus();
            cout << "Failed to subscribe - error code " << result << endl;
        }

The Console.exe (Thread Monitor) app has message handlers in place and subscribes with the dialog hwnd:

//*************************************************************/
// SubscribeToKernelNotifications()
//
// Open and subscribe to driver messages re thread and process 
// creation and deletion via the CMonitor class. 
//
// Return 0 for success, else error code
//*************************************************************/
int CConsDialog::SubscribeToKernelNotifications(void)
{
    int retval = 0;
    
    if(! m_PTMonitor.Subscribe(this->m_hWnd) ) {
        MessageBox(
          _T("Thread Monitor failed to subscribe yada yada yada..."));
        retval = m_PTMonitor.GetStatus();
    }
    
    return retval;
}

The CMonitor class itself could be cleaned up a bit, especially in the error reporting area, but serves the purpose for now - again, the main focus here is in using APCs.

A breakdown of the source directories:

  • _TDCommon contains code files used by more than one project, and the CMonitor files.
  • _TestDrv1 has the main Visual Studio solution files (open this solution to load all projects) and the _TestDrv1 code and project files.
  • _TestDrv3 contains _TestDrv3.c, _TestDrv3.h, and _TestDrv3.vcproj
  • _TestDrvClient1 contains the _TestDrvClient1 project files.
  • Console contains the Thread Monitor files.
  • Console3 contains the Console3 files.

Testing

The drivers included have in various forms been compiled under VC6 and VC7 on NT4, 2000, and XP boxes. Most of the latest development and testing has been done on an XP Professional SP2 box, running a P3 with a whopping 128 Meg of RAM.

A big concern to me has been making sure that all activity is captured as threads and processes are created and deleted, and testing has focused on how things behave under low resource conditions when many copies of an app (usually iexplore, since it routinely creates multiple threads on startup) are set in motion.

I've used Verifier sparingly. Under conditions that would cause, say, the KAPC allocation to fail, the code should just fail to queue the APC. The debug build (running under a checked ntoskrnl) will assert.

If you're interested in playing with the drivers, you may want to get a copy of Windbg, an extra box and a null modem and set up for remote debugging. More on this here, in the refs below, and elsewhere.

If anyone has a 2000 or XP Server box their not afraid of blue-screening, I'd appreciate a little testing help. A multiprocessor test would be most welcome as well.

What next?

Maybe an investigation of data caching on the kernel side would be useful. _TestDrv3 gets off easy, since all the info it needs to communicate can be folded into the two APC data parameters, and we don't need to read anything further from the driver.

Maybe I could even think about programming an actual device...

In the known bugs and features department, the process list in the dialog will occasionally fail to get the name of a process - usually one that dies before the Thread Monitor can get around to identifying it (keep in mind there's nothing 'real time' about the notifications, and then there's the overhead of posting a message to the subscribing window). The code substitutes 'NA' for these process names.

A future version could try to make use of the PsSetLoadImageNotifyRoutine rather than relying on a call to CreateToolhelp32Snapshot after the first thread of a process appears. In general this problem is limited to a few fleeting processes associated with virus checkers and the like.

References

This is not in itself a discussion of DDK development, but I’ll end by providing some references to literature and sites I’ve looked at (or want to). Much of the code in the sample draws directly from these sources. These references are also contained in the major header files for the driver(s), and I try to reference them when I engage in efficient code reuse.

This list is aging fast, but may save some Google time.

Programming the Microsoft Windows Driver Model
Walter Oney
1999 Microsoft Press
IBSN: 0-7356-0588-2

This was the first book I picked up in my quest for kernel more mastery, but it may not have been the best first choice. While it is a good reference, and it fills in some tough to find material, I found it frustrating to get a good overview of the task at hand. Good to have after you’ve gotten started.

The Windows 2000 Device Driver Book / device driver handbook
Art Baker, Gerry Lozano
2001 Prentice Hall PTR
IBSN: 0-13-020431-5

This book contains the most readable presentation of the driver environment I have come across. After a week of struggling through the Walter Oney book, this was a welcome relief, serving to organize and put in perspective the many questions and unresolved references I had accumulated. One of the appendices contains the settings for Visual C++ driver projects that guided me in setting up the sample.

Inside Microsoft Windows 2000
David A. Solomon, Mark Russinovich
Microsoft Press
ISBN: 0-7356-1021-5

This is about as good as it gets without cracking the Redmond repository – maybe better. Not directly targeted at device driver development ,but full of the kind of information that you just don’t get in the MSDN. The CD includes some interesting tools, including a kernel debugger that can be run on the target machine, and the contents of the sysinternals site which is full of useful info and samples – such as those I mention below.

Both of the above books come with Visual C++ custom Appwizards on CD that can be used to configure driver projects. The simple project I include here was not based on these – I felt the need to start from scratch in order to get a little more ‘hands on’ feedback. Besides, I think bugchecks are fun – don’t you?

Asynchronous Procedure Calls in NT
http://www.osr.com/ntinsider/1998/apc.htm

Ctrl2Cap sample from SysInternals
Mark Russinovich
http://www.sysinternals.com/ntw2k/source/ctrl2cap.shtml
Nice example, with source, of a basic filter driver on an input interrupt-driven device.

Inside the Native API
Mark Russinovich
http://www.sysinternals.com/ntw2k/info/ntdll.shtml

Pop Open a Priveleged Set of APIs with Windows NT Kernel Mode Drivers
Microsoft Systems Journal March, 1998
James Finnegan

Nerditorium
Microsoft Systems Journal July, 1999
James Finnegan
http://www.microsoft.com/msj/0799/nerd/nerd0799.aspx

Installing a Partially Checked Build
Open Systems Resources
http://www.osr.com/ntinsider/2001/checking/checked.htm

Build Tricks: Checked and Free Revisited
Open Systems Resources
http://www.osr.com/ntinsider/1998/build1.htm

Bugslayer
Microsoft Systems Journal January 1999
John Robbins

How to Verify Windows Debug Symbols
Microsoft Corp.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;q148660(obsolete link)

Inside NT's Asynchronous Procedure Call
Albert Almeida [AA]
http://www.windevnet.com/documents/s=7653/win0211b/0211b.htm (requires registration)

Very interesting treatment of the subject, which I stumbled on late in the project while googling for KiDeliverApc.

Inside Windows NT System Data
Sven B. Schreiber
Dr. Dobbs Journal
http://www.ddj.com/documents/ddj9911c/ (requires registration)

Note: I haven’t actually read this, but looks like a classic – if anyone knows the name of the law that states that if you collect magazine issues for 3 years straight and miss one, that will become the one you want, please let me know.

The DDJ article just mentioned deals with investigating the NtQuerySystemInformation() kernel API function – while you don’t need to run in kernel mode to use this API, I found the DDK learning path naturally leads to this type of exploration. Mr Schreiber has also authored the book Undocumented Windows Secrets: A Programmers Cookbook, and in his preface praises Gary Nebbetts subsequent Windows NT/2000 Native API Reference (an excerpt appears here on CodeProject) . He also has good words for Edward N. Dekker and Joseph M. Newcomer’s Developing Windows NT Device Drivers (CPians should be familiar with Newcomers many fine article contributions). Schreiber’s list also points to an article from NT Insider that has short reviews on a few more books, and Microsoft themselves maintain a list of titles so I’ll stop here for now.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Tim Deveaux
Software Developer
Canada Canada
No Biography provided

You may also be interested in...

Comments and Discussions

 
Question64 bit os Pin
sr333017-Aug-13 19:32
membersr333017-Aug-13 19:32 
AnswerRe: 64 bit os Pin
Tim Deveaux18-Aug-13 6:16
memberTim Deveaux18-Aug-13 6:16 
GeneralStreaming data question Pin
rosenbsn27-Sep-05 6:40
memberrosenbsn27-Sep-05 6:40 
GeneralRe: Streaming data question Pin
Tim Deveaux28-Sep-05 3:16
memberTim Deveaux28-Sep-05 3:16 
GeneralKeInsertQueueApc Pin
Alexander M.12-Jun-05 3:45
memberAlexander M.12-Jun-05 3:45 
GeneralCritism Pin
x3source24-Apr-05 20:48
memberx3source24-Apr-05 20:48 
GeneralRe: Critism Pin
Tim Deveaux25-Apr-05 4:21
memberTim Deveaux25-Apr-05 4:21 
GeneralNice article Pin
Ivo Ivanov14-Jan-05 11:38
memberIvo Ivanov14-Jan-05 11:38 
Generalcompanion CD Pin
mika12310-Sep-04 21:05
membermika12310-Sep-04 21:05 
GeneralRe: companion CD Pin
Tim Deveaux11-Sep-04 3:08
memberTim Deveaux11-Sep-04 3:08 
GeneralRe: companion CD Pin
andrewgs7311-Sep-04 20:51
memberandrewgs7311-Sep-04 20:51 
GeneralRe: companion CD Pin
andrewgs7311-Sep-04 20:52
memberandrewgs7311-Sep-04 20:52 
GeneralRe: companion CD Pin
Tim Deveaux12-Sep-04 3:23
memberTim Deveaux12-Sep-04 3:23 
GeneralExcellent Article Pin
Anonymous26-Aug-04 7:44
sussAnonymous26-Aug-04 7:44 
GeneralFeature report Pin
Tim Deveaux7-Apr-04 3:34
memberTim Deveaux7-Apr-04 3:34 
QuestionHow To use CMonitor class Pin
mailMonty13-Feb-04 21:25
membermailMonty13-Feb-04 21:25 
AnswerRe: How To use CMonitor class Pin
Tim Deveaux14-Feb-04 3:48
memberTim Deveaux14-Feb-04 3:48 
GeneralRe: How To use CMonitor class Pin
mailMonty14-Feb-04 19:58
membermailMonty14-Feb-04 19:58 
GeneralRe: How To use CMonitor class Pin
Tim Deveaux15-Feb-04 4:29
memberTim Deveaux15-Feb-04 4:29 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.150603.1 | Last Updated 9 Jan 2004
Article Copyright 2004 by Tim Deveaux
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid