Dave Leinweber
Mark Ryland
Microsoft Corporation
April 1996
Microkernel Designs and the Windows NT Architecture
The Windows NT 4.0 Kernel-Mode Change
Microsoft® Windows NT® Workstation 4.0 and Windows NT Server 4.0 include a change in the implementation of Microsoft Win32® graphics-related application programming interfaces (APIs). These changes are transparent to applications and users, yet they result in a variety of improvements to graphics performance and memory requirements, as well as simplify the design of the Windows NT Win32 subsystem. The improvements result from the move of certain operating system modules from a user-mode application process into a subsystem within the privileged portion of Windows NT, known as the Executive.
Changes to the code that operates in the kernel or "privileged" mode of any operating system can be of concern to application designers and system architects. Because such changes potentially affect the operating system's compatibility with existing applications, as well as its portability and reliability, such changes should be explained and justified.
This article provides a basic understanding of the issues and justifications surrounding this change and its effect on customers and developers. It explains how this change will provide improvements to the operation of the system—such as performance and memory requirements—while leaving the critical factors of application compatibility, scalability, portability, and reliability unaffected.
The Windows NT 4.0 architecture merges the best attributes of a layered operating system with those of a client/server or microkernel operating system. (Endnote 1) The Windows NT Executive is the generic name for a number of subsystems and components of the operating system that execute in the microprocessor's kernel (or privileged) mode. The remainder of the operating system, such as various server subsystems and their application clients, execute in user (or application) mode.
Within the Windows NT Executive, basic functions of the operating system are integrated in a component and layered fashion, as shown in Figure 1. The underlying layer of the Windows NT Executive is the Hardware Abstraction Layer (HAL), which provides a common software abstraction over such hardware devices as clocks, cache and memory controllers, peripheral adapters, symmetric multiprocessing functions, and system buses. Simplifying Windows NT provides support for a wide range of hardware designs within a given microprocessor family. Various system services at the top of the Executive offer support to the user-mode subsystems and to each other. Residing between these two layers is the microkernel, the heart of the operating system, which provides the most basic operating system functions such as first-level interrupt handling, deferred procedure call and thread scheduling, and synchronization primitives.
Figure 1. Windows NT 3.51 architecture
For this discussion, it is important to clearly understand the difference between the terms kernel mode (Endnote 2) and microkernel. Kernel mode refers to a mode of code execution in a microprocessor in which all memory is totally accessible and all CPU instructions can be issued. By providing the operating system software with a higher privilege level than application software, the microprocessor provides a necessary foundation for operating system designers to ensure that a misbehaving application cannot disrupt the stability of the system as a whole. (Endnote 3)
A microkernel, on the other hand, is the name given to the core portion of a modern, modular operating system. Microkernel operating systems are based on two fundamental principles. The most basic principle is one of modularity, encapsulation, and data hiding. In this aspect of the design, there is one and only one portion of the operating system that has system-wide responsibility for a particular function. All other parts of the operating system (as well as applications, naturally) access that function through a well-defined interface. There is no duplication of function and no "back doors" to critical data structures; all access to system data structures is through software interfaces. This approach makes it possible to upgrade or replace entire modules within the system without disturbing the remainder. A secondary principal of the microkernel design, related to the first but focused more on the implementation strategy, is that large portions of the operating system which traditionally run entirely in the kernel mode of the microprocessor can now be executed in user mode, with only the microkernel itself, along with a relatively small amount of hardware device-specific code, executing in kernel mode.
Operating systems that follow both of these principles are often called "pure microkernel" systems. Operating systems that follow the first principle only—strict modularity and strong encapsulation—but not the second are sometimes called "modified microkernel" or macrokernel operating systems.
No commercial operating system is based on a pure microkernel design. (Endnote 4) The reason is simple: the pure microkernel design is commercially impractical because it is too computationally expensive—that is, it is too slow. The simple fact is that there are insufficient benefits from running core portions of the operating system in user mode to make up for resulting dramatic losses in performance. Consider the following: suppose the file-system code of an operating system has a bug that causes it to crash from time to time. In a traditional operating system or a macrokernel operating system, a bug in kernel-mode code such as a file system may crash the entire operating system. In a pure microkernel operating system the file system implementation is run in user mode, so theoretically it can't crash the entire operating system. That's all well and good, but the hypothetical buggy file system will still fail from time to time. What is the practical difference in real-world use between an operating system that crashes from time to time and one that continues running but loses all access to persistent storage? (Endnote 5)
From the start, the Windows NT architecture has fallen squarely into the macrokernel camp. On the one hand, Windows NT is extraordinarily modular and well encapsulated in its design. For example, the Windows NT operating system as a whole does not depend on a particular size for the microprocessor's virtual memory page. (Endnote 6) The virtual memory page size is determined dynamically at boot time by the HAL and thereafter all operating-system code, including the memory manager itself, works happily with whatever page size the HAL has determined as the operating system booted. Thus, a microprocessor architecture could switch from using 4K pages to 16K pages and Windows NT could run despite this dramatic change in the hardware environment.
On the other hand, from its first release, Windows NT has come down squarely on the side of implementing high-performance, operating-system subsystems in kernel mode, where they can interact with the hardware and each other without performance-robbing transitions of processor mode along with thread/process transitions. From its earliest days, Windows NT has implemented its memory manager, integrated cache manager, file systems (including network redirectors), object/security manager, network protocols, network server, and all thread/process management as kernel-mode subsystems while retaining a highly modular design. All of these subsystems are fully protected from errant applications, of course, as applications do not have direct access to the code and data of the privileged part of the operating system.
The result of the tradeoffs in the Windows NT design speaks for itself: Windows NT 3.51 is recognized throughout the industry as having hit a "sweet spot" in operating system design. It is both incredibly robust and stable as an application server and workstation platform, yet fast and nimble from the perspective of core operating system services such as virtual memory management, file input/output (I/O), networking, and file and print sharing.
There is one area, however, in which Windows NT has, until now, followed the more pure microkernel approach in its implementation of the window manager and graphics subsystems that implement the graphical user interface (GUI) aspects of the Win32 API. Those subsystems—the Window Manager (also known as USER) and Graphics Device Interface (GDI)—have been implemented in Windows NT 3.51 and previous releases as a separate user-mode process called the Client-Server SubSystem (identifiable with process monitoring tools as CSRSS.EXE).
In the earliest days of Windows NT, the Win32 server was designed as an equal to the OS/2, POSIX, and other planned operating system environments. Each of these subsystems was intended to operate as a stand-alone environment. However, this design was leading to a large amount of duplication of system functions that, ultimately, would have negatively affected both system size and performance. Moreover, from a market perspective it became clear toward the end of the initial development of Windows NT that Win32 would be the critical programming interface for the entire operating system. To alleviate much of this duplication while responding to market trends, the final Windows NT 3.1 Win32 subsystem was a specially privileged "application" that was required by all other subsystems and the operating system as a whole.
The Win32 server in Windows NT releases prior to Windows NT 4.0 contains five modular components, as shown in Figure 2:
Each of these components reside in a separate DLL.
Figure 2. Windows NT 3.51 Win32 subsystem
Having a separate server process that contains the Win32 graphics subsystem requires substantial memory overhead to hold data passed between the client and server and back, to marshal the information, and to manage the required multiple threads. The design also requires multiple thread and process context switches, (Endnote 7) which require considerable CPU cycles and memory resources. In normal operation, potentially hundreds of graphics support calls per second can dramatically affect the overall performance of the system.
Nevertheless, the present design is highly optimized. For example, for each thread on the client side there is a paired thread in the Win32 server waiting in a special interprocess communication facility called Fast LPC. Unlike normal thread context switches, transitions between paired threads via Fast LPC do not cause a scheduling event in the microkernel, thereby enabling the server thread to run for the remaining time slice of the client thread before having to take its turn in the kernel's preemptive thread scheduler. Moreover, shared memory buffers are used to allow fast passing of large data structures such as bitmaps, and clients have direct but read-only access to key server data structures to minimize the need for thread/process transitions between clients and the Win32 server. Finally, GDI operations are batched and cached. Batching means that a series of graphics calls by a Win32 application is not actually "pushed" over to the server and drawn on the output device until a GDI batching queue is filled. The size of the queue can be set by Win32 applications, and the queue can be flushed at any time. Conversely, read-only properties and data structures of GDI are, once they are obtained from the Win32 server, cached on the client side for nearly instantaneous subsequent access.
Still, because Windows NT is a Windows-based operating system, in which all other environmental subsystems utilize the basic functions provided by Win32, and because the graphics and windowing subsystems have a very high rate of interaction with hardware (video drivers, mouse and keyboard drivers, and so forth), the Windows NT 4.0 design team decided to move that common functionality from user mode into kernel mode. In this way, the functions can be performed without shared memory buffers or paired threads, and with fewer thread and process context switches. This change as implemented in Windows NT 4.0 results in faster operation and reduced memory requirements, both visible benefits to the end user. And there is no loss of reliability, since (a) the kernel mode implementations of Win32 are fully protected from direct access by applications; and (b) the Win32 server process is considered a critical system component of Windows NT 3.51, such that if the CSRSS.EXE process faults the entire Windows NT operating system is shutdown.
With this new release, the Window Manager, GDI, and related graphics device drivers have been moved to the Windows NT Executive running in kernel mode. Both Window Manager and GDI now exist as Windows NT Executive system services. The console, shutdown, and hard error handling functions remain as part of CSRSS in user mode, as shown in the following figure.
Figure 3. Windows NT 4.0 Win32 service
Moving the Window Manager and GDI functionality to kernel mode has removed a large amount of complexity from the code. Applications can now access the GUI implementation subsystems in exactly the same way that they currently implement the base (non-GUI) parts of the Win32 API such as file I/O and memory management: by making a kernel-mode trap call at which time the calling thread is switched to a kernel-mode stack and all call parameters are validated before being used by the privileged subsystem. Thus, many thread and process transitions are eliminated. Still, kernel-mode transitions are relatively expensive compared to direct code/data access at the same privilege level so some of the performance tricks of the old design—such as a more limited degree of GDI batching and caching in the user mode portion of the client's address space—are maintained to maximize performance.
Once applications have called into Window Manager and GDI, those subsystems can now access other Windows NT Executive systems directly without the cost of user/kernel-mode transitions. This is especially important in the case of GDI calling through video drivers, which involves interaction with video hardware at high frequencies and high bandwidths.
All in all, the change reduces memory consumption, thread count, and process/thread context switches substantially.
There are a number of questions that naturally come to mind when evaluating a change such as this. How will this change affect Windows NT architecture? Will this affect reliability or security? What about memory requirements? In this section of the article, we'll attempt to answer these and other questions related to this change.
From an overall architectural view, Windows NT looks pretty much the same. The microkernel is still the center of the Windows NT Executive—for the most part, it remains the same. The architecture of Windows NT has always provided for an extensible kernel-mode operation. The Executive was designed from the beginning to allow services to be added without redesigning the system. In the current change, a set of services previously implemented in user mode are now operating in kernel mode. And because none of the APIs have changed, there is no effect on existing Windows NT-based applications.
Figure 4. Windows NT 4.0 architecture
The objective of any high-end operating system, such as Windows NT, is to protect the system and other applications from errant applications.
Windows NT uses the features offered by today's microprocessors to provide two ways of accomplishing memory protection—process-space memory management and CPU-processing modes. A thread running in user mode inside any given process cannot read or write outside of its own memory space, nor can it read or write to the kernel-mode portion of its own address without first transitioning to kernel mode. In Windows NT 3.51 (and previous releases) Window Manager and GDI are protected from applications writing into their space by the fact that each resides in a separate process.
On the other hand, the shared services provided by the Windows NT Executive reside within the processor's dedicated kernel-mode memory addresses, and in this way are protected from applications and other user-mode code. The kernel-mode services can access any memory location, including all applications' memory spaces. However, applications cannot directly access the memory of any service located in the kernel mode.
Figure 5. Windows NT 3.51 process protection
In order for an application process to access any of the Win32 Server processes in Windows NT 3.51, the application uses the User32 or GDI32 interfaces to pass information to the Win32 server. Because the Win32 server is located in a different process, User32 and GDI32 accomplish this action using a shared memory window and the process manager service of the Executive. While this design does indeed protect the Win32 server function from the application, it does so with a fair amount of overhead, as is shown in Figure 5.
Figure 6. Windows NT 4.0 service protection
In the Windows NT Workstation 4.0 release, the Window Manager and GDI processes are still protected because applications cannot write to memory locations occupied by kernel-mode code and data, as shown in Figure 6. Also notice how the interface between the application process and the Window Manager or GDI has been greatly simplified with the elimination of the shared-memory window and multiple threads. The same thread is used for both the application code and the operating-system service without exposing the system to an errant application process because the operating-system code is able to intervene and validate access when the thread tries to cross the user/kernel boundary.
Consequently, there is no change in stability or reliability resulting from poorly behaved applications because kernel-mode code and data is protected by the Windows NT architecture and the processor's memory-protection system.
Note that in this respect of total isolation of critical operating-system data from user-mode application code, Windows NT Workstation 4.0 remains unchanged in being architecturally more robust than other PC operating systems such as Microsoft Windows® 95, IBM OS/2 Warp, and Apple® Macintosh® operating systems. All of those systems make a trade-off for greater performance and smaller memory footprint that involves putting critical system data structures in memory locations that are directly accessible to application code without transitioning through operating-system code to the privileged mode of the processor. That tradeoff is entirely appropriate for today's low- and medium-range platforms, but not in a high-end platform such as Windows NT. With Windows NT 4.0, it remains true that if application code can crash the system, Windows NT has a bug, period. In other typical PC operating systems, because of architectural choices inherent in their designs, application code can crash the operating system even when the system code is flawless.
One of the side effects of this change is that now Window Manager, GDI, and graphics drivers have the potential to write directly to other spaces within the Executive, thereby possibly disrupting the stability of the whole system.
However, from the user's point of view, that potential to disrupt the system has always existed. If the GDI process in Windows NT 3.51 should fail for any reason, the user would be presented with a system that appears to have crashed. The fact that the kernel is still operating is invisible to the user, because it simply appears that the system is not responding. Such is the critical nature of Window Manager and GDI. The change in Windows NT 4.0 has no effect on this kind of problem. However, Microsoft ensures through its development and testing standards that these services are highly reliable and trustworthy.
Another area of possible impact can come from the move of the graphics drivers into kernel mode. Previously, some portions of a graphics driver ran within CSRSS, while other others ran in kernel mode. Now, the entire driver runs in kernel mode. While Microsoft does not develop all the graphics device drivers supported in Windows NT, Microsoft does work directly with hardware manufacturers to ensure that they are able to produce reliable and efficient drivers. New kernel-mode GDI drivers must be produced to operate in Windows NT 4.0. Fortunately, the required changes are not significant—experienced developers are able to make the modifications in just a few short hours. To ensure that all currently supported graphics devices are still supported in Windows NT 4.0, we are working very closely with manufacturers to make sure that the change doesn't affect the stability of the system. All drivers shipped with the system are submitted to the same rigorous testing as other Executive components.
Finally, it's important to understand that this design is not fundamentally "risky." It is identical to the ones used by existing I/O Manager drivers (for example, network card drivers and hard disk drivers). All of these drivers have been operating within the Windows NT Executive since the inception of Windows NT with a high degree of reliability.
Due to the modular design of Windows NT, moving Window Manager and GDI to kernel mode will make no difference to the security subsystem or to the overall security of the operating system. This will also have no effect on the C2 or E3 security certification evaluation, other than making it easier to document the internal architecture of Windows NT since the majority of Win32 APIs are now implemented in a more similar fashion.
One of the primary goals of any new release of Windows NT to improve its performance. The Windows NT Workstation 4.0 release is no different, and in many ways is even more focused on performance. With the new user shell matching the Windows 95 shell, customers will compare the two operating systems for use on the desktop within their organizations. While the advanced security and advanced operating-system features of Windows NT sometimes mean there are incompatibilities with some legacy programs, it does not mean that the performance need be noticeably different.
By making the Window Manager and GDI subsystems into Executive services, both operations decrease in size and improve in performance. This is largely due to the elimination of the shared memory buffers, as well as a substantial reduction in the code size and thread/process switching from the more complex client/server design. Consider, for example, the client/server transition time on a Pentium 90 of at least 70 μ seconds, whereas a kernel-mode transition is on the order of 4 to 5 μ seconds. For screen graphics calls, this difference can become significant enough to be visible to the end user.
This can be observed with previous releases of Windows NT. Running the 3-D Pinball Table application that comes with the Windows 95 Plus! pack on a Windows NT 3.51 Workstation results in virtually unplayable graphics action. However, on Windows NT Workstation 4.0, users see a performance essentially identical to running the game on Windows 95. While Windows NT is not designed specifically as a games platform, these same results will also be apparent on business applications that place heavy demands on the graphics subsystem. Applications such as Microsoft PowerPoint will display information 15 to 20 percent faster on the Windows NT 4.0 release than on Windows NT 3.51.
Overall, however, Windows NT 4.0 system performance is estimated to remain about the same as Windows NT 3.51 because non-graphic subsystems are not affected by this change. Other changes that improve the functionality, such as the addition of the feature-rich but more complicated new shell, are expected to take up much of the performance gains realized by the change to Window Manager and GDI.
Using a client/server approach to communication between subsystems requires a substantial amount of memory. While some memory is tied up in the code required to support the complex client/server protocol, most of the required memory is used in a 64K shared-memory buffer to accumulate and pass parameters between the client and server.
Moving Window Manager and GDI to kernel mode completely eliminates the buffer and related code complexities because all the data in the client's address space is fully accessible from kernel mode. The total amount of working set memory saved with this change in the release of Windows NT 4.0 is estimated to be between 256K for machines that require few simultaneously open applications and 1 MB for users who consistently have many applications open at the same time.
These memory savings will likely be used in other subsystems or new services such as the new Windows Explorer-based user interface. Consequently, from a total system perspective, the final release of Windows NT 4.0 is expected to have roughly the same memory requirements as Windows NT 3.51.
Some people have speculated that the move of Window Manager and GDI into kernel mode will hurt the preemptive multitasking capability of Windows NT. The theory is that with all the additional Win32 processing time spent in kernel mode, there will be less opportunity for other threads to be run preemptively.
This view is based on a misunderstanding of the Windows NT architecture. It is true that in many other nominally preemptive operating-systems code (such as many UNIX implementations and OS/2 Warp), executing in kernel mode is never preempted by the operating-system scheduler; or (as in AIX) is preempted only at a certain limited number of predefined points of kernel reentrancy. In Windows NT however, threads running anywhere in the Executive are preempted and scheduled alongside threads running in user mode; and all code within the Executive is fully reentrant. Among other things, this capability is necessary to achieve a high degree of system scalability on symmetric multiprocessing (SMP) hardware. This is one of the many subtle but important respects in which Windows NT represents a next-generation operating system beyond the capabilities of most of its competition. The preemptability and full scheduling of kernel-mode threads applies equally to threads running through the new occupants of kernel space: Window Manager and GDI.
Another line of speculation is that SMP scaling will be hurt by this change. This theory is as follows: previously, an interaction between an application and Window Manager or GDI involved two threads, one in the application and one in CSRSS.EXE. Therefore, on an SMP system the two threads could run in parallel, improving throughput.
This analysis shows a misunderstanding of how Windows NT works today. Today, with a single exception, (Endnote 8) all calls from the client to the Win32 server run synchronously. That is, the client thread entirely blocks waiting on the server thread and begins to run again only when the server thread has completed the call. Therefore, no parallelism on SMP hardware can ever be achieved. This phenomenon is easily observable with a busy graphics application using Performance Monitor on an SMP system. The observer will discover that on a two-processor system each processor is approximately 50 percent loaded, and it is relatively easy to find the single CSRSS.EXE thread that is paired off with the busy application thread. Indeed, because the two threads are fairly intimate with each other and sharing state, the processors' caches must be constantly flushed to maintain coherency. This is the reason why with Windows NT 3.51 a single-threaded graphics application typically runs slightly slower on an SMP machine when compared to a single processor system!
As a result, the changes in Windows NT Workstation 4.0 will actually increase SMP throughput of applications by making heavy use of Window Manager and GDI, especially when more than one application thread is busy. When two application threads are busy on a two processor Windows NT 3.51 machine, a total of four threads (two in the application plus two in CSRSS.EXE) are battling for time on the two processors. While only two are typically ready to run at any given time, the lack of a consistent pattern of which threads run results in loss of locality of reference and cache coherency, since the busy threads are likely to get shuffled from one processor to another. In the Windows NT 4.0 design, each of the two application threads will essentially have their own processor, and the Windows NT automatic thread affinity will tend to run the same thread on the same processor indefinitely, thus maximizing locality of reference and minimizing the need to synchronize the private per-processor memory caches.
In general, moving Window Manager and GDI to kernel mode will have no effect on the ability of Windows NT to handle multiple processors, different processor architectures, large amounts of memory, or disk management. This change will have no discernible effect on the microkernel or HAL. The microkernel is responsible for responding to interrupts and exceptions, scheduling threads, and synchronizing the activities of multiple processors. The HAL handles hardware-dependent details such as I/O interfaces, interrupt controllers, and multiprocessor communication mechanisms. The microkernel and HAL will continue to be designed in the same manner with the same number of locks that are independent of each other, as well as being independent of other kernel-mode components. Because neither of these Windows NT Executive components has been affected by the change to Window Manager and GDI, the ability of Windows NT to scale with multiple processors has not changed.
The main advantage of this change to future releases of Windows appears in the area of graphics performance. After three years of fine tuning, the GDI subsystem in Windows NT 3.51 is operating at its maximum potential under the client/server model. With Windows NT 4.0, both the Window Manager and GDI functions operate faster than their predecessors with the potential of additional performance gains in future releases.
So what does moving Window Manager and GDI from their place as user-mode subsystems to kernel-mode Executive services really mean to the user? To the administrator? To the systems developer? Simply put, the move delivers improved performance and increased functionality with a decrease in memory requirements!
Users and administrators of Windows NT 4.0-based workstations and servers will notice a performance improvement in graphics intensive applications. System reliability, security, scalability, and memory utilization are unaffected.
Application developers are not affected by this move. The Win32 APIs are unaffected. And though Window Manager and GDI are now contained within the Windows NT Executive, all Win32 APIs are still accessed with the same User32 and GDI32 interfaces.
From an architectural standpoint, Windows NT continues to be a modified microkernel operating system. In fact, the kernel itself is largely unaffected by this change. It is a testament to the architectural design of Windows NT that a change like this to a critical operating system component can be made without major structural changes.
Ultimately, integrating the Window Manager and GDI subsystems into the Windows NT Executive is simply another step in the continuous improvement of Windows NT. It allows Windows NT to continue to define, within the framework of the Windows 32-bit operating system family, a new standard for personal computing that is both high-end and mainstream at one and the same time.
Note To access information via the World Wide Web, visit the Microsoft Windows NT Workstation 4.0 Web site (http://www.microsoft.com/ntworkstation/).
Disclaimer The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date publication. This document is for informational purposes only.
MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.
Microsoft, Windows, Win32, Windows NT, and PowerPoint are registered trademarks of the Microsoft Corporation.
Intel is a registered trademark of Intel Corporation. OS/2 is a registered trademark of International Business Machines Corporation. POSIX is a registered trademark of Institute of Electrical and Electronics Engineers, Inc. Macintosh is a registered trademark of Apple Computer, Inc. UNIX is a registered trademark in the United States and other countries, licensed exclusively through X\Open Company, Ltd.