7.3.5 Managing Supplemental Device Queues
The I/O Manager usually (except for FSDs) creates an associated device queue object when an NT driver calls IoCreateDevice. It also provides IoStartPacket and IoStartNextPacket, which drivers can call to have the I/O Manager insert IRPs into the associated device queue or call their StartIo routines with an IRP, as desribed in Section 7.2.
Consequently, very few NT driver designers find it necessary (or particularly useful) to set up their own device queue objects for IRPs. Likely candidates are drivers, such as the NT SCSI port driver, that must coordinate incoming IRPs from some number of closely coupled class drivers for heterogeneous devices that are serviced through a single controller or bus adapter.
In other words, the designer of an NT driver for a disk array controller is more likely to use a driver-created controller object than to set up supplemental device queue object(s), while the designer of an NT driver for an add-on bus adapter and of a set of class drivers is slightly more likely to use supplemental device queues.
Support for Supplemental Device Queues
The KeInsert..DeviceQueue routines return a Boolean value indicating whether the given IRP was inserted into the device queue. As mentioned in in Chapter 3, KeInsertDeviceQueue either inserts an entry at the tail of the device queue and sets the state of the device queue object to Busy, or it simply sets the state to Busy and returns FALSE to indicate that the IRP was not queued. A driver that sets up a supplemental device queue can impose a driver-determined order on the processing of IRPs by calling KeInsertByKeyDeviceQueue, which also returns a Boolean value indicating whether the given IRP was inserted into the device queue.
Consider the following an implementation guideline for NT drivers that call KeInsertDeviceQueue or KeInsertByKeyDeviceQueue directly:
If the return value is FALSE, the caller must pass the IRP on for further processing by another driver routine immediately.
A call to KeInsertDeviceQueue or KeInsertByKeyDeviceQueue sets the state of the device queue object to Busy, even if the queue is currently empty and the input IRP is not inserted into the queue. A reciprocal call to KeRemove..DeviceQueue resets the state of an empty device queue object to Not-Busy.
Such a driver can remove an IRP from its supplemental device queue (or reset the device queue state to Not-Busy) by calling KeRemoveDeviceQueue. The driver also can remove a particular entry or determine whether it is currently queued by calling KeRemoveEntryDeviceQueue.
Using Supplemental Device Queues with a StartIo Routine
By calling IoStartPacket and IoStartNextPacket, an NT device driver’s Dispatch and DpcForIsr (or CustomDpc) routines synchronize calls to its StartIo routine using the device queue that the I/O Manager created when the driver created its own device object. For a port driver with a StartIo routine, IoStartPacket and IoStartNextPacket insert and remove IRPs in the device queue for the port driver’s shared device controller/adapter. If such a port driver also sets up supplemental device queues to hold requests coming in from closely coupled higher-level class drivers, it must “sort” incoming IRPs into its supplemental device queues, usually in its StartIo routine.
However, the closely coupled class drivers of such a port driver must identify their devices in the IRPs they send down so that the port driver can attempt to insert each IRP into the appropriate driver-managed device queue before attempting to program its shared controller/adapter to carry out the request. Thus, such a port driver can process incoming requests for all devices on a first-come, first-served basis until a call to KeInsertDeviceQueue puts an IRP into a particular class driver’s device queue.
However, such a port driver’s StartIo routine must determine which class driver sent any incoming request. When each class driver called IoGetDeviceObjectPointer to layer itself over the port driver, each class driver was given a unique file object pointer for its “open” of the port driver’s device object. Each class driver can set this file object pointer in the port driver’s I/O stack location to identify itself when it sets up IRPs for the port driver.
By using its own device queue for all IRPs to be processed through its StartIo routine, such an underlying port driver serializes operations through the shared device (or bus) controller/adapter to all attached devices. By sometimes holding IRPs for each supported device in a separate device queue, this port driver inhibits the processing of IRPs for an already busy device while increasing I/O throughput for every other device doing I/O through its shared hardware.
At the call to IoStartPacket from such a port driver’s Dispatch routine, the I/O Manager either calls that driver’s StartIo routine immediately or puts the IRP into the device queue associated with the device object for the port driver’s shared controller/adapter.
Managing Supplemental Device Queues in StartIo
Such a port driver with supplemental device queues would have a StartIo routine with the following features:
{
PIO_STACK_LOCATION irpSp =
IoGetCurrentIrpStackLocation(Irp);
PDEVICE_EXTENSION deviceExtension =
DeviceObject->DeviceExtension;
PFILE_OBJECT whichClass;
PATTACHED_DEVICE_STATE attachedDevice;
: :
//
// Switch on MajorFunction code in IRP.
//
switch (irpSp->MajorFunction) {
case IRP_MJ_READ:
case IRP_MJ_WRITE:
//
// Get which device this request is for.
//
whichClass = irpSp->FileObject;
attachedDevice = GetAttachedDeviceState(
deviceExtension,
whichClass);
//
// Increment the following to get a unique value
// to give the error log routine, if necessary.
//
attachedDevice->SequenceNumber++;
: :
case IRP_MJ_DEVICE_CONTROL:
//
// Get which device this request is for.
//
whichClass = irpSp->FileObject;
attachedDevice = GetAttachedDeviceState(
deviceExtension,
whichClass);
: :
} // end switch
}
Note that this code fragment checks state information, based on the file object pointer that each class driver sets in the underlying port driver’s I/O stack location of the IRP, in order to determine the current state of the (attached) target device for any given request. The device object pointer input to its StartIo routine gives the port driver access to its own device object, representing the device controller/adapter. It does not give this port driver access to the class driver’s device object for the target device. Such a port driver must maintain its own state information about each of the heterogeneous devices that it services through the shared device controller/adapter.
NT driver writers who design class/port drivers with supplemental device queues in the port driver should keep in mind the following facts:
·An NT driver cannot get a pointer to the device object of any driver layered above itself easily.
By design, the I/O Manager does not provide a support routine for getting such a pointer. Moreover, the order in which NT drivers are loaded makes it impossible for lower drivers to get pointers for higher-level drivers’ device objects, which have not yet been created when any lower-level driver is initializing itself.
·An NT driver cannot use a pointer to the device object of any driver layered above itself.
There is no way to synchronize access to a single device object (and its device extension) between two drivers in a multiprocessor-safe manner. Neither driver can make any assumptions about what I/O processing the other driver is currently doing.
Even for closely coupled class/port drivers, each class driver should use the pointer to the port driver’s device object(s) only to pass on IRPs with IoCallDriver. The underlying port driver must maintain its own state, probably in the port driver’s device extension, about requests that it processes for any closely coupled class driver(s)’ device(s).
Managing Supplemental Device Queues Across Driver Routines
Any NT port driver that queues IRPs in supplemental device queues for a closely coupled set of class drivers also must handle the following situation efficiently:
1.Its Dispatch routines have inserted IRPs for a particular device in the driver-created device queue for that device.
2.IRPs for other devices continue to come in, to be queued to the driver’s StartIo routine with IoStartPacket, and to be processed through the shared device controller.
3.The device controller does not become idle, but each IRP held in the driver-created device queue also must be queued to the driver’s StartIo routine as soon as possible.
Consequently, such a port driver’s DpcForIsr must attempt to transfer an IRP from the driver’s internal device queue for a particular device into the device queue for the shared adapter/controller whenever the port driver completes an IRP, as follows:
1.The DpcForIsr routine calls IoStartNextPacket to have the StartIo routine begin processing the next IRP queued to the shared device controller.
2.The DpcForIsr routine calls KeRemoveDeviceQueue to dequeue the next IRP (if any) that it is holding in its internal device queue for the device on whose behalf it is about to complete an IRP.
3.If KeRemoveDeviceQueue returns a nonNULL pointer, the DpcForIsr routine calls IoStartPacket with the just dequeued IRP to have it queued to the shared device controller/adapter. Otherwise, the call to KeRemoveDeviceQueue simply resets the state of the device queue object to Not-Busy, and the DpcForIsr omits the call to IoStartPacket.
4.Then, the DpcForIsr calls IoCompleteRequest with the input IRP for which the port driver has just completed I/O processing, either by setting the I/O status block with an error or by satisfying the I/O request.
Note that the preceding sequence implies that such a DpcForIsr routine also must call the internal GetAttachedDeviceState routine shown in the preceding StartIo code fragment. That is, the DpcForIsr must determine the device for which it is completing the current (input) IRP in order to manage internal queueing of IRPs efficiently.
If such a port driver attempted to wait until its shared controller/adapter was idle before dequeueing IRPs held in its supplemental device queues, the driver might starve a device for which there was heavy I/O demand while it promptly serviced every other device for which the current I/O demand was actually much lighter.