QueueUserAPCEx Version 2: Truly Asynchronous User-Mode Notification on Windows Platforms

• Download demo project - 12.7 Kb
• Download source - 22.3 Kb

Introduction

In this work, we introduce a new implementation of our QueueUserAPCEx function that provides, at user-level, truly asynchronous procedure calls (APCs) without requiring the target thread itself to call any special function in order to enter an alertable state.

What's new

The new implementation of QueueUserAPCEx extends rather than invalidates the previous one. For this reason, everything mentioned in our previous article is still valid. The main features of this new implementation of QueueUserAPCEx are the following:

1. Truly asynchronous notification:

   According to the initial version of QueueUserAPCEx, the APC routine will be called next time the thread runs in user-mode [3]. However, if the thread is blocked (e.g. it has called Sleep, WaitForSingleObject, recv) then the routine is not going to be executed before the blocking call returns. The new implementation resolves the above issue by utilizing Kernel-Mode APCs, ensuring this way that the APC will be executed even if the target thread is blocked. The validity of the initial implementation stems from the fact that a user may prefer the previous behavior.

2. Full Portability:

   The initial version of QueueUserAPCEx sets the target thread in alertable state by setting equal to 1 the memory address that corresponds to 0x4a bytes offset from the base address of the ETHREAD data structure [4]. In the current implementation, the target thread sets itself in alertable state without utilizing any undocumented code. Thus, we have a portable implementation of the kernel mode device driver.

Background

For asynchronous notification, Windows supports Asynchronous Procedure Calls (APCs). An APC is a kernel-defined control object that represents a procedure that is called asynchronously. APCs have the following features [1]:

• An APC always runs in a specific thread context.
• An APC runs at OS predetermined times.
• They can cause preemption of the currently running thread.
• APC routines can be preempted.

There are three different types of APCs in the kernel [2]: User mode, Normal Kernel mode and Special Kernel mode APCs. Kernel mode APCS are executable by default. On the other hand, User mode APCs are, by default, disabled; that is, they are queued to the user-mode thread, but are not executed, except at a well-defined point in the program. Specifically, they can be executed only when an application has either called a wait service and has enabled alerts to occur, or called the test-alert service.

The Win32 API provides the QueueUserAPC function, which allows an application to queue an APC object to a thread. The queuing of an APC is a request for the thread to call the APC function. When a user-mode APC is queued, the thread is not directed to call the APC function unless it is in an alertable state. Unfortunately, a thread enters an alertable state only by using one of the following Win32 API functions: SleepEx, SignalObjectAndWait, WaitForSingleObjectEx, WaitForMultipleObjectEx, or MsgWaitForMultipleObjectsEx.

Application Programming Interface

The user-level functionality of QueueUserAPCEx is included in the two files (QueueUserApcEx.c and QueueUserApcEx.h) of the source code distribution. QueueUserAPCEx has the same prototype with the ordinary Win32 QueueUserAPC call. Two additional calls are also provided for the initialization and the shutdown of the package. Analytically:

• DWORD QueueUserAPCEx_Init(VOID): Initializes QueueUserAPCEx by opening a handle to the kernel-mode device driver.
• DWORD QueueUserAPCEx(PAPCFUNC pfnApc, HANDLE hThread, DWORD dwData): Adds a user-mode asynchronous procedure call (APC) object to the APC queue of the specified thread AND sets this thread in alertable state.
• DWORD QueueUserAPCEx_Fini(VOID): Shutsdown QueueUserAPCEx by closing the handle to the kernel-mode device driver.

Of course, it is necessary to install and load (start) the <alertdrv.sys> driver first. This can be performed very easily by following the directions of the Readme1st.txt file that is included in the demo project. Another file (Readme.txt) in the source code distribution provides directions for building and installing the kernel-mode driver and running successfully the application.

/* testapp.c : demo of QueueUserAPCEx */
#define _WIN32_WINNT 0x0501
#include <windows.h>
#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#include "QueueUserApcEx.h"

#include "QueueUserApcEx.h"

DWORD WINAPI TestSleep(LPVOID lpParam)
{
    printf("[Thread %4ld] Calling Sleep...\n", GetCurrentThreadId());
    Sleep(INFINITE);
    printf("[Thread %4ld] Exiting!\n", GetCurrentThreadId());

    return 0;
}

DWORD WINAPI TestWait(LPVOID lpParam)
{
    HANDLE hEvent;
    DWORD dwEvent;

    hEvent = CreateEvent(NULL, FALSE, FALSE, NULL);

    printf("[Thread %4ld] Calling WaitForSingleObject...\n", 
                                       GetCurrentThreadId());
    dwEvent = WaitForSingleObject(hEvent, INFINITE);
    printf("[Thread %4ld] WaitForSingleObject returned %d\n", 
          GetCurrentThreadId(), dwEvent); /* WAIT_IO_COMPLETION */
    printf("[Thread %4ld] Exiting!\n", GetCurrentThreadId());

    return 0;
}

DWORD WINAPI APCRoutine(DWORD APCParam)
{
    printf("[Thread %4ld] Inside APC routine with argument (%ld)\n",
        GetCurrentThreadId(), APCParam);
    return 0;
}

int main(void)
{
    DWORD   APCData;
    HANDLE  hThread;
    ULONG   id;
    INT     res;

    QueueUserAPCEx_Init();
    printf("[Thread %4ld] Starting\n", GetCurrentThreadId());

    /* Test: send an APC to myself */
    printf("[Thread %4ld] Sending an APC to myself\n", GetCurrentThreadId());
    APCData = 33;
    res= QueueUserAPCEx((PAPCFUNC) APCRoutine, GetCurrentThread(), APCData);


    hThread= CreateThread(NULL, 0, TestSleep, NULL, 0, &id);
    /* Sleep for a while; then send an APC to hThread */
    Sleep(5000);
    printf("[Thread %4ld] Sending an APC 
        to the thread that called Sleep\n", GetCurrentThreadId());
    APCData = 44;
    res= QueueUserAPCEx((PAPCFUNC) APCRoutine, hThread, APCData);
    WaitForSingleObject(hThread, INFINITE);


    hThread= CreateThread(NULL, 0, TestWait, NULL, 0, &id);
    /* Sleep for a while; then send an APC to hThread */
    Sleep(5000);
    printf("[Thread %4ld] Sending an APC to the thread 
       that called WaitForSingleObject\n", GetCurrentThreadId());
    APCData = 55;
    res= QueueUserAPCEx((PAPCFUNC) APCRoutine, hThread, APCData);
    WaitForSingleObject(hThread, INFINITE);

    printf("[Thread %4ld] Exiting\n", GetCurrentThreadId());

    QueueUserAPCEx_Fini();

    return 0;
}

If you compile and run the above program (testapp.dsp), provided you have started the driver, your output will be like this:

[Thread 3572] Starting
[Thread 3572] Sending an APC to myself
[Thread 3572] Inside APC routine with argument (33)
[Thread 3696] Calling Sleep...
[Thread 3572] Sending an APC to the thread that called Sleep
[Thread 3696] Inside APC routine with argument (44)
[Thread 3696] Exiting!
[Thread 3216] Calling WaitForSingleObject...
[Thread 3572] Sending an APC to the thread that called WaitForSingleObject
[Thread 3216] Inside APC routine with argument (55)
[Thread 3216] WaitForSingleObject returned 192
[Thread 3216] Exiting!
[Thread 3572] Exiting
Press any key to continue

Implementation Details

QueueUserAPCEx differs from QueueUserAPC in that it sets the target thread in alertable state. More specifically, the QueueUserAPCEx function calls the ordinary Win32 API QueueUserAPC function and next, through a kernel mode device driver (alertdrv.sys) manages to set the target thread in alertable state. When a thread sends an APC to another, it also sends transparently (through DeviceIoControl) the handle of the target thread to the underlying device driver. The driver's interrupt service routine receives the information, finds the corresponding ETHREAD structure for that handle (ObReferenceObjectByHandle), and then sends a Kernel-Mode APC to that thread. The APC routine, which is executed in the context of the target thread, sets that thread in alertable state. In contrast to our previous approach, this is performed by calling the documented routine KeDelayExecutionThread [3]. Another possible solution, described in [5], is to call KeWaitForSingleObject. The final result is the execution of the additional APC object that was inserted in that thread with QueueUserAPC.

Further Issues

• Instead of using QueueUserAPC, we could pass the APC routine to the driver and execute it as the user-mode callback of the APC object that the driver inserts to the target thread. However, we preferred to keep the driver completely decoupled from any user-level code.
• The execution of the APC routine causes the interrupt of the waitable Windows functions (e.g., WaitForSingleObject). In addition, these routines return the WAIT_IO_COMPLETION value, as if the user had called their corresponding *Ex parts (e.g., WaitForSingleObjectEx). Despite the execution of the APC routine, some other system services are restarted, and thus threads return into their previous blocked state. This interaction of QueueUserAPCEx with the system services is still under study.
• According to the above, it becomes obvious that the new implementation of QueueUserAPCEx provides a useful tool for asynchronous thread cancellation, by executing ExitThread within the APC routine. Therefore, it provides to users a method to terminate their threads in a safe way, instead of using TerminateThread. This functionality of QueueUserAPCEx is being used by the POSIX Threads library for Windows [6].
• If the target thread is already in alertable state then the user-mode APC will be executed immediately. If the thread exits before the execution of the kernel-mode APC but after the latter's insertion in the thread, the kernel APC object is automatically released. In our case, QueueUserAPCEx first suspends the target thread, next inserts the user-mode (QueueUserAPC) and the kernel-mode APCs (driver), and finally resumes the thread. This ensures that the kernel-mode APC will be executed first even if the target thread has been already in alertable mode.
• The asynchronous notification that both versions of QueueUserAPCEx support can be exploited for providing user-defined Unix signals on Windows platforms [7].
• An installation program for the device driver of QueueUserAPCEx is not provided along with the source code. Not experienced users are advised to use OSR's Driver Loader [8], a freely available program that allows users an easy way to register, unregister, start, and stop a device driver.

License

QueueUserAPCEx is distributed under the terms of the GNU Lesser General Public License (LGPL).

History

• May 23, 2003 - Initial release.
• May 25, 2004 - Second version.

References

1. Open Systems Resources, Inc.
2. D. A. Solomon. Inside Windows NT, Second Edition, Microsoft Press, 1998.
3. Microsoft Corporation, Microsoft Developer Network Library.
4. User mode APCs.
5. Nerditorium, MSJ, July 1999.
6. Open Source POSIX Threads for Win32.
7. Panagiotis E. Hadjidoukas, A Device Driver for W2K Signals, Windows Developer Magazine, Volume 12, Number 8, August 2001.
8. OSR Online, OSR Driver Loader.