The most common interrupt-related problem on multiprocessor computers is not the rate of interrupts and DPCs, but their distribution among processors. To determine whether you have a distribution problem:
Observe the proportion of time each processor is spending servicing interrupts and DPCs.
Observe the rate of interrupts and DPCs for each processor.
Note whether interrupts and DPCs are distributed equally among all processors or whether one or more processors are servicing all of the interrupts or DPCs.
A busy processor that is servicing all of the system's interrupts or DPCs is likely to become a bottleneck. The source of the problem is not the number of interrupts or DPCs but the way that interrupts and DPCs are distributed among the processors.
The following sections describe common strategies for distributing interrupts and DPCs on Windows NT servers.
Different processor platforms use different methods to distribute interrupts. The distribution of interrupts from network adapter cards is controlled by the Hardware Abstraction Layer (HAL) for each processor platform. The interrupt scheme implemented by the HAL depends on the capability of the processor. Some processors include interrupt control hardware, such as the Advanced Programmable Interrupt Controller (APIC) on newer Pentium and Pentium Pro processors. The APIC allows processors to route interrupts to other processors on the computer.
You cannot control or modify the distribution of network card interrupts on your computer. The information in this section is included to help you interpret the interrupt data you collect and, in particular, to explain one of the reasons why one processor might be busier than the others. Monitoring the processor that services the majority of interrupts can help you anticipate or prevent a bottleneck.
There are many different strategies for distributing interrupts on multiprocessor computers. Common strategies are:
No distribution. The traditional method for managing interrupts is to send all interrupts from all network cards to a single processor, usually the first (lowest-numbered) processor. If your data shows that all interrupts are being serviced by a single processor, your system might use this method.
Static distribution. Some systems distribute network card interrupts among processors statically, that is, the distribution doesn't change. These systems associate each network adapter card with a processor. The interrupts generated by the network card are always sent to the network card's associated processor, regardless of whether the processor is busy or idle.
Typically, these systems associate the first (lowest-numbered) network card with the first (lowest-numbered) processor and each subsequent network card with each subsequent processor. If there are more network cards than processors, the system begins its assignments again with the first processor.
If your data shows that interrupts are distributed rather evenly among processors, you might have a system that provides static distribution. However, if one of the network cards is idle or not used, static distribution results in unequal distribution of interrupts.
Dynamic distribution. Some systems distribute interrupts to processors dynamically based on one or more elements of system data, such as processor activity. The interrupts generated by a network card can be sent to any one of the processors.
If your data shows that interrupts are almost always distributed evenly, you might have a system that provides dynamic distribution.
Distribution of interrupts to all processors, whether statically or dynamically, is commonly known as symmetric interrupt distribution. Symmetric interrupt distribution is designed to improve scaling and to prevent a single processor from becoming a bottleneck while other processors have excess capacity. It is available on the Microsoft Windows NT 4.0 HAL for Pentium and Pentium Pro processors. For specific information on the distribution method used for your processor platform, consult your system vendor.
Strategies for producing the most efficient interrupt distribution systems are evolving. You cannot change the interrupt distribution scheme on your computer, but you can coordinate the distribution of DPCs with the distribution of interrupts.
The distribution of DPCs generated by network cards is a function of Ndis.sys, the Windows NT implementation of the Network Driver Interface Specification (NDIS). Ndis.sys is a wrapper that shields the details of the network adapter card from the rest of the operating system. Ndis.sys controls DPC distribution on all network cards that use miniport drivers, which is nearly all of them.
Ndis.sys attempts to balance the processor workload generated by interrupts. By default, Ndis.sys associates each network card with a processor and directs all DPCs from a network card to the processor associated with that network card. Ndis.sys attempts to compensate for the burden placed on the lowest-numbered processor by associating the first network card with the highest-numbered processor. Each subsequent network card is associated with the next processor in descending order of processor number.rrich
On many systems, this strategy balances the load. If a server that does not distribute interrupts has two processors and one network card, Processor 0 services the interrupts and Processor 1 services the DPCs.
Unfortunately, this distribution strategy doesn't always work as planned:
DPCs evolve from interrupts. When a DPC is generated from an interrupt, the DPC must be switched to another processor. The switch requires an interprocessor interrupt from the sending processor to the receiving processor. These very high priority interprocessor interrupts consume additional processor time.
Information gathered when the interrupt is processed is stored in the processor cache. When the DPC generated from the interrupt is switched to another processor, the data in the cache of the sending processor is flushed and must be collected anew by the receiving processor.
On many platforms, interrupts are distributed among all processors. Leaving the DPC on the processor that handled the interrupt reduces the number of interprocessor interrupts and allows the DPC to use data stored in the processor cache when the interrupt was serviced.
Fortunately, if DPC distribution is a problem in your system, you can improve system performance by optimizing DPC distribution.
The solutions to poor DPC distribution differ depending on whether the system distributes interrupts symmetrically:
No distribution. If you are administering a multiprocessor server that does not distribute interrupts symmetrically, monitor the highest-numbered processor carefully. If the processor frequently operates at capacity (if Processor: % Processor Time = 100%) and more than half of its time is spent servicing DPCs (if Processor: % DPC Time > 50%), you can improve the performance of your system by one or more of the following methods:
Upgrading to a system that distributes interrupts.
Adding network adapter cards so that you have one network adapter card for each processor. Ndis.sys then distributes DPCs to all processors. Generally, you should only add a network adapter card if you also need the bandwidth because each additional network card has some intrinsic overhead.
Adjusting the NDIS Processor Affinity Mask to allow or prevent certain processors from servicing DPCs. This is not recommended unless you are very familiar with this process. A less-than-optimal mask can degrade processor performance. You can change the Processor Affinity Mask by editing the ProcessorAffinityMask value entry in the Registry.
Symmetric distribution. If you are administering a multiprocessor server that distributes interrupts to all processors, whether statically or dynamically, you can improve performance by setting the value of the ProcessorAffinityMask value entry in the Registry to zero. If the value of ProcessorAffinityMask is zero, network cards are not associated with processors, and DPCs remain on the processor that handled the interrupt. ProcessorAffinityMask is located in:
HKEY_LOCAL_MACHINE\System
\CurrentControlSet
\Services
\NDIS
\Parameters
ProcessorAffinityMask is described in the Regentry.hlp file on the Supplement 1 CD. (It is not included in previous versions of Regentry.hlp.)