July 1999

Code for this article: July99Nerd.exe (56KB)

James Finnegan is a developer at Communica, Inc., a Massachusetts-based company specializing in system software design. He can be reached via or jimf@nema.com.

Microsoft Corporation does not make any representation or warranty, express or implied, with respect to some of the techniques used in this column.

In my January 1999 column, I presented a way to notify user-mode apps about interesting events that occur in kernel-mode Windows NT® and Windows® 2000 drivers. This was meant to demonstrate how the all-powerful Windows NT kernel mode can see interesting things on a global scale (or at least global to a particular computer). The kernel-to-user-mode signaling was done to get information displayed without unreasonable requirements (such as the use of a debugger).     This month, I'll explore how to notify user-mode applications asynchronously from kernel mode in more depth. As you'll see, the various techniques range from the obvious to the obscure. Some of these techniques are not endorsed by Microsoft and therefore are always subject to change and breakage. Observe the Microsoft Surgeon General's Warning (sometimes referred to as the seal of approval) on the lower part of this page. That aside, let's dig in. Why User-mode Notification?       At one point or another, any application is going to interact with hardware. In environments such as MS-DOS®, this was fairly simple. An MS-DOS-based application might install an interrupt service routine that would get invoked whenever the associated hardware signaled its respective interrupt request line. The hardware would temporarily redirect application processing of an application through low-level interfaces provided by the PC architecture.     Windows NT provides a level of abstraction between an application and the hardware. The Windows NT kernel-mode driver architecture restricts hardware access to a subset of controlled drivers, keeping user-mode applications at arm's length from the hardware. Programming in kernel mode to provide privileged access to user-mode applications is fairly straightforward, but how does a user-mode application find out when some piece of hardware passes along some interesting event? How does a user-mode app get things like mouse movement messages without constantly checking to see if the mouse has moved?     Even though user-mode Win32®-based apps cannot directly interact with hardware or register routines that get called when hardware events occur as in MS-DOS, you can exploit some Windows NT interfaces to permit asynchronous notification from kernel mode. To demonstrate some of the mechanisms available to you, I'll present a sample app and associated kernel-mode driver that exercises some of these asynchronous capabilities (see Figure 1). The kernel-mode driver is a dynamically loadable driver that requires administrative privileges to load and run. The code to do this is implemented in Dynamic.c. For a complete description of this code please refer to my article, "Pop Open a Privileged Set of APIs with Windows NT Kernel Mode Drivers" (MSJ, March 1998). Coercing Asynchronous Behavior       Before exploring the signaling mechanisms presented here, I'd like to discuss how my sample driver is emulating asynchronous behavior without the aid of specific hardware. I did this strictly for demonstration purposes—the idea here is that your (real-world) application wants notification of some physical hardware event. Within the confines of the sample app, "random" behavior is cajoled by the user-mode app requesting a specific kernel-mode notification mechanism to be exercised. From there, the Windows NT kernel driver will request that a piece of its code run in the context of a system worker thread.     System worker threads are a limited resource created by Windows NT at boot time for use by installable kernel drivers. These system worker threads eliminate the overhead of each driver, creating threads for presumably lightweight and short-duration tasks. It's important to note that these system worker threads are running outside of the context of the calling application's thread and process space. This proves that the mechanisms presented here can occur outside on the context of the calling app, thereby being truly asynchronous.     How do you use system worker threads? Sprinkled within the sample kernel driver, snippets of kernel-mode code are scheduled for execution by the use of the ExInitializeWorkItem macro and the ExQueueWorkItem function.

// Queue execution to a system thread to permit this // IRP to complete... ExInitializeWorkItem(pItem, KSignalEvent, UserEvent); ExQueueWorkItem(pItem, DelayedWorkQueue);

The function referenced in the second parameter of ExInitializeWorkItem will be called at some later point within the context of a system worker thread. The function can accept a single parameter, and must be prototyped as such:

VOID WorkerRoutine(PVOID parameter);

Be aware that since system worker threads are shared among all Windows NT drivers, you should not execute code that takes an inordinate amount of time to process, since this ties up the thread for use by other drivers. As you'll see in the sample driver, calls to KeDelayExecutionThread are made within the context of the worker thread (to simulate delayed processing). This may not be the brightest idea, but hey, this worker thread scheme is contrived throwaway code anyway!     While I'm on the topic of system worker threads (and outside the scope of this column), you should note that ExInitializeWorkItem and ExQueueWorkItem cannot be safely used within plug and play drivers in future Microsoft operating systems (such as Windows 2000), since the arbitrary unloading of a plug and play driver cannot be synchronized with potential outstanding queued work items. For Windows Driver Model (WDM) 1.10 (which is planned for implementation in Windows 2000), you should use IoAllocateWorkItem, IoFreeWorkItem, and IoQueueWorkItem. These calls will properly associate work items with your device object by using internal reference counts for pending or executing work items. Signaling Using Event Handles       The simplest way to notify a user-mode thread about something significant in kernel mode is to use a kernel mode-created event handle. This is done by creating a named event using IoCreateNotificationEvent (as shown in Knotify.c):

// Create event for user-mode processes to monitor RtlInitUnicodeString(&uszNotifyEventString, L"\\BaseNamedObjects\\KnotifyEvent"); extension->NotifyEvent = IoCreateNotificationEvent(&uszNotifyEventString, &extension->NotifyHandle);

A user-mode app can then reference the created named event in a call to the OpenEvent Win32 API call. The returned object handle can then be used in a call to WaitForSingleObject or WaitForMultipleObjects to wait for the event handle to be signaled by kernel mode.

// Get a handle to the named kernel event and wait on it! KnotifyEvent = OpenEvent(SYNCHRONIZE, FALSE, "KnotifyEvent"); WaitForSingleObject(KnotifyEvent, INFINITE);

The kernel-mode driver can signal the event with a call to KeSetEvent, as demonstrated in the KSignalEvent function in Knotify.c:

KeSetEvent(NotifyEvent, 0, FALSE);

There are a few downsides to using kernel-created events for signaling user mode. Although this mechanism works fine, a user-mode app may be restricted (by Windows NT security) to opening this type of object when it calls OpenEvent. Second, the kernel-created event handle is global to the system, which means all apps are sharing this single event. Although this may be favorable in some circumstances, it probably isn't ideal for general use. Finally, and perhaps most important, the state of the kernel-mode event handle cannot be changed by the user-mode app under any circumstances. This means that the kernel-mode driver must set and clear the event handle's state itself. This can cause a number of problems. First, a user-mode thread may outrun the kernel-mode driver's ability to set and clear the event, resulting in looped calls to WaitForSingleObject to succeed in rapid succession. Conversely, if a user-mode app goes off to outer space for a while, it may miss the "pulsing" of the event by kernel mode.     Alternatively, you can create a user-mode event handle in your user-mode code and reference it in kernel mode. This eliminates many of the limitations of kernel-mode named events. The code for doing this is presented in UserModeEvent (in Async.c). First, a user-mode event handle is created:

// Create an event handle that the KM driver can signal UserModeEvent = CreateEvent(NULL, // Default security TRUE, // Manual reset FALSE, // non-signaled state NULL); // No name

This handle is then passed to the kernel-mode driver. The kernel-mode driver must then obtain a kernel-mode event pointer to the user-mode object. This is done with a call to ObReferenceObjectByHandle:

PKEVENT UserEvent; • • • // Obtain a kernel pointer to the user-mode-created // event handle... ObReferenceObjectByHandle( *((PHANDLE)Irp->AssociatedIrp.SystemBuffer), 0, (POBJECT_TYPE) NULL, UserMode, (PVOID)&UserEvent, (POBJECT_HANDLE_INFORMATION) NULL);

The pointer returned in the fifth parameter of ObReferenceObjectByHandle can subsequently be used like any other kernel-created object. However, a user-mode app has the ability to set and clear its state, unlike the previously presented method.     You should note that the event handle in the sample code is set in the context of the system worker thread. This shows that the kernel-obtained event pointer can be manipulated outside the context of the calling user-mode thread. This is important since asynchronous processing can occur in any thread's context (most likely as the result of some external hardware event). Therefore, resources that are only visible in the context of the calling thread may not be available when an external event occurs. As you can see, however, once the kernel event pointer is obtained, it can be referenced outside of the thread's context.     The call to ObReferenceObjectByHandle increments the event object's reference count, which means that subsequent validation of the event handle is not necessary, since the object manager will keep the event object around, even if the calling Win32-based app crashes or closes its handle to the event prior to you setting it. Utilizing Asynchronous IRP Processing       An inherent ability of the Windows NT I/O model is asynchronous request processing. This is achieved by opening the I/O target in overlapped mode. This requires you to create a user-mode event handle that you wait on. This event handle is signaled when I/O is complete. In the sample code, when the kernel-mode driver is opened (in Dynamic.c), a call to CreateFile is used, passing the FILE_ FLAG_OVERLAPPED flag as a parameter:

// Open the driver... hDevice = CreateFile("\\\\.\\Knotify", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, 0, // Default security OPEN_EXISTING, FILE_FLAG_OVERLAPPED, // Perform asynchronous I/O 0); // No template

All subsequent I/O operations (through ReadFile, WriteFile, or DeviceIoControl) will require the use of an OVERLAPPED structure to complete I/O to this driver. This is shown in the calls to the driver in Async.c (see Figure 2).     The event handle embedded in the OVERLAPPED structure is initialized prior to the I/O call to the driver. This event handle is signaled when I/O is completed by the driver. (GetOverlappedResult waits on this event handle.) Any processing can occur between the I/O call and the event handle being set. This means that you can send an I/O request to a kernel-mode driver, and the driver can leave the IRP in an incomplete state until it is ready to return something to user mode.     Marking an IRP as incomplete is straightforward. Simply call the IoMarkIrpPending macro and return STATUS_ PENDING to the I/O Manager.

// Mark the IRP pending. The driver will complete it // later... IoMarkIrpPending(Irp); ntStatus = STATUS_PENDING;

The I/O Manager returns from the I/O call, but the event handle within the OVERLAPPED structure is not set. An IRP can remain in a pending state indefinitely. When IoCompleteRequest is eventually called (shown in KDelayIRPCompletion in Knotify.c), I/O is completed, and the results of the IRP (with its associated output buffer, if any) is passed back to the user-mode caller. Using Asynchronous Procedure Calls       Although the previous techniques work, they all require you to check on the status of an event handle at one point or another. Because of this, most code that waits on the event handle is placed in its own thread to avoid blocking any other user activity within the application. You could alter your main thread to use the MsgWaitForMultipleObjects Win32 API call, which will permit Windows message processing while waiting on any number of events. However, a setup in which a callback function is called asynchronously whenever the user-mode thread is scheduled to run would be closest to emulating ISR processing in user mode. This would eliminate the need for dedicated threads for asynchronous processing, since scheduled routines could be integrated with the thread switching and scheduling code without the knowledge of the user-mode code.     Asynchronous procedure calls (APCs), which are not documented in the Windows NT DDK, but are partially present in the ever-tremendous NTDDK.H file (see Figure 3), almost fit the bill. APCs are powerful yet misunderstood. They are used extensively throughout Windows NT, but are not well represented outside the confines of Microsoft® code. Although no DDK documentation exists, APCs make cameo appearances in the ReadFileEx and WriteFileEx Win32 API calls. In addition, mention of them is made within the DDK documentation in functions such as KeWaitForSingleObject, since the use of functions that suspend the thread can cause queued asynchronous processing to occur, as you'll see.     In short, APCs are functions that execute asynchronously within the context of a specified thread. All threads in Windows NT contain a queue to schedule the execution of APCs. This differs from deferred procedure calls (DPCs), which are usually used within the processing of an interrupt. DPCs are queued in a single, systemwide queue. Usually, functions are inserted into the APC queue by user-mode apps by using the QueueUserAPC Win32 API call. Functions can also be queued by kernel-mode code using the APIs presented here.     The APC queue is serviced each time a user-mode thread is rescheduled to run, if the thread is in an alertable state (more on this later). The implementation is straightforward. An application passes a pointer to its user-mode APC callback function. This function receives three PVOID arguments and cannot return a value to the caller:

void ApcCallback(PVOID NormalContext, PVOID SystemArgument1, PVOID SystemArgument2)

The three arguments are passed from kernel mode without any translation or sanity checking. It's up to you to validate and utilize their contents. These values, however, are completely arbitrary and definable by you. The address of this callback function must be relayed to kernel mode. The sample app does this by using a private IOCTL (IOCTL_ NOTIFY_REGISTER_APC). The address is then used by the kernel driver to create and queue an APC.     As shown in Figure 3, the definitions for the APC callbacks and structures are supplied in NTDDK.H. The kernel-mode initialization and queuing function prototypes are not provided, even though the exports are present in the standard DDK import libraries. The definitions for these functions are as follows:

// Definitions for Windows NT-supplied APC routines. // These are exported in the import libraries, // but are not in NTDDK.H void KeInitializeApc(PKAPC Apc, PKTHREAD Thread, CCHAR ApcStateIndex, PKKERNEL_ROUTINE KernelRoutine, PKRUNDOWN_ROUTINE RundownRoutine, PKNORMAL_ROUTINE NormalRoutine, KPROCESSOR_MODE ApcMode, PVOID NormalContext); void KeInsertQueueApc(PKAPC Apc, PVOID SystemArgument1, PVOID SystemArgument2, UCHAR unknown);

The APC is first initialized with a call to KeInitializeApc, passing the user-mode callback address in the NormalRoutine parameter. The first parameter passed to the kernel and user-mode callbacks is passed in NormalContext:

KeInitializeApc(Apc, KeGetCurrentThread(), 0, (PKKERNEL_ROUTINE)&KMApcCallback, // kernel mode routine 0, // rundown routine (PKNORMAL_ROUTINE)*UserRoutine, // user-mode routine UserMode, (PVOID)(ULONG)1);

The APC is then queued in the thread's APC queue with a call to KeInsertQueueApc. The second two parameters are passed in here:

KeInsertQueueApc(Apc, (PVOID)(ULONG)2, (PVOID)(ULONG)3, 0);

The user-mode callback will be called when the thread is placed into an alertable state. A thread is placed into an alertable state by the following Win32 API calls: SleepEx, SignalObjectAndWait, WaitForSingleObjectEx, WaitForMultipleObjectsEx, or MsgWaitForMultipleObjectsEx. Although calling one of these functions to coerce the thread scheduler to run APCs isn't a big deal, it's not much better than waiting on an event, as I did earlier. Maximum coolness would be achieved if the APC callback just executed, without any influence by user-mode code. Thread Hack-O-Rama       Kernel synchronization calls, such as KeWaitForSingleObject, can also optionally set a thread into an alertable state to permit the execution of APCs. (See the documentation for KeWaitForSingleObject, which will become suddenly clear now that you know how APCs work). This means that you can simply call a kernel-supplied function like this after queuing your APC (as is done in the processing of the IOCTL_NOTIFY_REGISTER_APC_WITH_KM_ ALERTING IOCTL). This will force it to execute any queued APCs next time the thread is scheduled to run:

// Cheezy way to force the thread into being alertable KeInitializeEvent(&event, SynchronizationEvent, FALSE); KeWaitForSingleObject(&event, Executive, UserMode, TRUE, &Timeout);

The third and fourth parameters in KeWaitForSingleObject allow the thread scheduler to execute any pending APCs, which is the behavior you want. Note that even though it appears that KeWaitForSingleObject is effectively doing what was previously being done in user mode (waiting on an event), a timeout value of zero is passed in, causing the wait to immediately return (but not before the APC is executed). Of course, this is a contrived and nonstandard use of the kernel mode synchronization functions, but it does not utilize any undocumented code, nor should its behavior change in future versions of Windows NT. However, this is as close as you can get to maximum coolness without delving into the netherworld of paltform and OS version-specific hacks, which I will not approach in this month's column.     Although not all-inclusive, the techniques I presented will give you a leg up on maximizing system throughput by exploiting the asynchronous processing nature of the Windows NT I/O model. Although it is not as bare metal as some of you may be accustomed to, these techniques give you near equivalent functionality for your app. Have a suggestion for Nerditorium? Send it to Jim Finnegan at jimf@nema.com.

From the July 1999 issue of Microsoft Systems Journal.