Ruediger R. Asche
Microsoft Developer Network Technology Group
November 1995
Following up on the Microsoft® Development Library's "The Little Device Driver Writer," this article presents an overview of virtual device driver (VxD) programming and a comprehensive reading list.
With the introduction of the Microsoft® Windows® 95 operating system, virtual device drivers (VxDs) have evolved from a very special and somewhat playful concept to an integral system component. Almost all device drivers in Windows 95 are implemented as VxDs. Consequently, many tasks that were previously solved by MS-DOS® or Microsoft Windows® user-mode drivers must now be tackled by VxDs. The information about writing VxDs is rather scattered. In this article, I will present a road map for writing Windows 95 VxDs. Every item on the Microsoft Development Library CD that is related to VxD writing should be included here. (Throughout this article, all titles that start with a Q number are located in the Knowledge Base and Bug Lists section of the Development Library. Other Development Library locations are indicated in parentheses after the title.) I have also snooped around in the world outside Microsoft. Whatever I could find about writing VxDs is somewhere in here also.
It is my hope that whatever your experience in writing VxDs, this article will present something new and helpful to you.
This article is designed to provide information to anyone involved in VxD writing in any form. The analogy I chose is the level ranking scheme I experienced in my Tae Kwon Do school. You should assess yourself according to the following scheme to decide where to start reading.
White Belt: You are a white belt student if you don't know anything about PC architecture, let alone Windows architecture, or where to start. After reading this article, you will hopefully pass this state.
Yellow Belt: You have a blurry idea about how to go about things, but you need to figure out how an IBM®-compatible PC works.
Blue Belt: You know the Intel architecture, but you don't know a whole lot about the Windows enhanced-mode operating system (VMM and VxD) architecture.
Purple Belt: You know (roughly) how the VMM (virtual machine manager) and VxDs work, but you have never built a VxD and don't know how to debug one.
Brown Belt: You are familiar with writing VxDs, but you need more information on how to use VMM and VxD services. You want to write a VxD that needs to virtualize a hardware device or that supports an MS-DOS terminate-and-stay-resident (TSR) program that your company has shipped for a long time.
Black Belt: You are an experienced VxD writer and want to enhance the functionality of the VMM through a VxD into an exciting and previously unexplored area.
As happens frequently in the martial arts, you will want to go back and forth in your studies in order to review previous materials and peek into more advanced areas of study.
There is not a whole lot of text in this article. I feel that most of what is out there in the market covers the area of VxD programming fairly comprehensively, and I'd rather let the material speak for itself instead of generating useless noise. However, in a few places I added a paragraph or two to clarify where certain things fit in.
I have not tried to stick to any particular party line while compiling this list. There is a long and continuous discussion about what can and/or should be done with VxDs, and what the problems and advantages of describing undocumented data structures are. I certainly do have an opinion about these issues, but the most important thing in this article was to present you with as close to a one-stop shopping place for VxD information as I could get. Some of the sources I refer you to contain information that is not documented and/or endorsed by Microsoft.
I annotated everything I introduce, assuming that you are a professional developer whose primary interest is to build a stable and working device driver that does something useful (you will notice that a lot of samples demonstrate rather playful little diagnostic tools that may or may not be of use to you). I annotated and sorted all of the sources of information in a way that I feel makes it easier for you to find a safe and efficient way through the jungle of VxD programming.
Information management is not an easy task. I have tried to make this compilation as comprehensive and well organized (in other words, useful) as possible, but I don't say there's nothing that fell through the cracks. I encourage you to submit your own recommendations for articles and other material. The magazines I consulted are Dr. Dobb's Journal, Windows/DOS Developer's Journal, Windows Sources, and PC Magazine.
This section contains items that are useful for beginners as well as advanced device driver writers.
Hazzah, Karen. Writing Windows VxDs and Device Drivers. Lawrence, KA: R&D Publications, Inc., 1995.
My comment: Probably the only really systematic and thorough introduction to VxD writing in existence at the time this article was written. Contains some fundamentals about hardware architecture as well as an introduction to the VMM and a number of working VxD samples.
Windows Device Driver Kit (Product Documentation, DDKs)
Although the DDK is, strictly speaking, not necessary to write VxDs, you'll probably want to refer to the DDK documentation to determine details of the system services.
"The Little Device Driver Writer" (Technical Articles, Windows Articles, Driver Articles)
OK, how's that for some shameless self-promotion? This article has been around for over a year now, but it is pretty much a one-size-fits-all introduction to device driver writing. It contains a little bit of operating system architecture, a discussion of tools and debugging, and even a mention of Santa Claus. So there. You won't find any more self-promotion in this article, I promise.
The VMM is an operating system that, among other things, divides CPU time between tasks. Although Windows 95 refines the smallest schedulable unit to be a thread instead of a process, and the notion of a process itself has also taken on a different meaning with Windows 95, the concept of a virtual machine (VM) is still central to the VMM. Except for some rather specialized types of VxDs, every VxD must be aware of the existence of VMs and be able to arbitrate between VMs.
To understand virtual machines, you must familiarize yourself with the architecture of the Intel® 80386-compatible (and later) processors, to which Windows 95 is tailored. There are many books available that cover the 80386 architecture. I found it most useful to go back to the source:
The 386 DX Microprocessor Programmer's Reference Manual. Intel Corporation, 1990 (order number 230985-003).
Here are a few Knowledge Base articles and specifications that you may find useful for more information on architectural issues:
Q57480, "Intel Phone Number for Information About EMS Specification"
Q99357, "Hardware Interrupt Priorities—Background and Usage"
Q104418, "Direct Memory Access (DMA) Channels and Descriptions"
Q52196, "Serial Communications Parity Definition"
Q71704, "MS-DOS: Reasons for A20 Line Switching"
"Extended Memory Specification (XMS) 3.0" (Specifications)
"Extended Capabilities Port (ECP) Specifications" (Specifications)
"Microsoft Flash File System Media Control Structures Specification 1.0" (Specifications)
You might also want to check another technical article, "The Tao of Interrupts" (MSDN Library Archive Edition, Technical Articles, Windows Articles, Driver Articles, Windows [16-bit] Only), for some general (as well as more specific) information on interrupts.
The market contains a myriad of books and articles about PC architecture. Most of the information is buried in discussions about system diagnostic tools. Here are four pretty random selections I found:
Kyle, Jim and Chip Rabinowitz. "How the 8259 Programmable Interrupt Controller Manages External I/O Devices." Microsoft Systems Journal 4 (May 1989).
Knoblaugh, Rick. "Eavesdropping on Interrupts." Dr. Dobb's Journal (November 1993).
Miles, Troy A. "Writing PCMCIA Software." Dr. Dobb's Journal (June 1994).
Prosise, Jeff. "DMA and Memory Managers." PC Magazine (December 6, 1994).
Windows 95 is an evolved operating system. Thus the architecture of Windows 95 is also the architecture of Windows 3.x as well as the architecture of MS-DOS; there is even a small part of Windows architecture that is relevant to VxD writers. The following list covers the necessary pieces of all of the above operating systems. First, let's look at the material that is on the Development Library CD:
Q32905, "Windows Version History"
Q118331, "How to Search for Windows DDK Articles by KB Subcategory"
Q51737, "Expanded Memory under Windows 3.0"
Q82787, "Meaning of LocalTSRs= in SYSTEM.INI"
MS-DOS Programmer's Reference 6.0 (Books and Periodicals)
"MS-DOS 6 Specifications" (Specifications)
"EMM386 I/O Trapping Extensions" (Specifications)
"TSR Support in Microsoft Windows Version 3.1" (MSDN Library Archive Edition, Technical Articles, Windows Articles, Driver Articles, Windows [16-bit] Only)
"Floating-Point in Microsoft Windows" (MSDN Library Archive Edition, Technical Articles, Windows Articles, Driver Articles, Windows [16-bit] Only)
"Microsoft Windows for Workgroups Version 3.1: Architecture Highlights" (MSDN Library Archive Edition, Backgrounders and White Papers, Operating Systems)
"Microsoft Windows Operating System 3.1 and Improved Hard Disk Access" (MSDN Library Archive Edition Backgrounders and White Papers, Operating Systems)
The following group of Knowledge Base articles elaborates on specific VxDs or parts of the VMM. Several of these articles also present VxD code that you may find useful on the brown and black belt levels.
Q75006, "Virtual Communications Driver Functional Structure"
Q75008, "Virtual Display Device I/O Trapping"
Q75009, "Using Call_Priority_VM_Event"
Q75864, "Async VMM Services May Be Called by Interrupt Handlers"
Q77403, "SAMPLE: User Interaction w/ a VxD Using Message Mode Services"
Q78151, "Using the Interrupt 2Fh Critical Section Services"
Q78325, "Ring Protection Under Windows 3.1"
Q83658, "INF: Version Information Resources in VxDs"
Q83726, "Int 12h and Windows Enhanced Mode (demonstrates Assign_Device_V86_Pages service)"
Q90796, "Instanced Data Management in Enhanced Mode Windows"
Q92407, "API Translation Buffers in Enhanced Mode Windows"
Q100579, "Claiming Critical Sections on Timer Ticks"
This group of items focuses on new elements of the Windows 95 architecture:
King, Adrian. "Windows, the Next Generation: An Advance Look at the Architecture of Chicago" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1994 Volume 9, January 1994)
Pietrek, Matt. "Stepping Up to 32 Bits: Chicago's Process, Thread, and Memory Management" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1994 Volume 9, August 1994)
Oney, Walter. "Extend Your Application with Dynamically Loaded VxDs Under Windows 95" (Microsoft Systems Journal, 1995 Volume 10, May 1995)
"Plug and Play SCSI: Designing to the New Standard" (Technical Articles)
Q117567, "How 16-Bit and 32-Bit Programs Multitask in Windows 95"
"Microsoft Windows and the Plug and Play Framework Architecture" (Backgrounders)
"Networking in Windows 95: Adding Value to Connected Windows-Based PCs" (Backgrounders)
"PCMCIA Card Support in Windows 95" (Backgrounders)
"Plug and Play BIOS Functionality and Windows 95" (Backgrounders)
The following items give you a good idea on how to tackle the mysterious area of DMA programming:
"Virtual DMA Services (VDS) Specification 1.0" (Specifications)
Q124729, "How to Program DMA Transfers in the Protected Mode Windows Environment"
Q63937, "SAMPLE: DMA in Windows 3.0"
"Ask Dr. GUI #13: Safe DMA" (MSDN Library Archive Edition, Technical Articles, Ask Dr. GUI)
Also, the Virtual Device Adaptation Guide (VDAG) from the Windows 3.0 and Windows 3.1 DDKs provides a very good introduction to the enhanced-mode Windows architecture. The Windows 3.1 DDK is in the Development Library under Product Documentation, DDKs.
Believe it or not, the VMM has been around for a long time—to be more precise, since Windows version 2.11. This almost antique article provides some of the pioneer information about the very early days of the VMM:
Duncan, Ray. "Microsoft Windows-386: Creating a Virtual Machine Environment." Microsoft Systems Journal 2 (September 1987).
King, Adrian. Inside Windows 95. Redmond, WA: Microsoft Press, 1994.
The following article is an excellent discussion of the new layered file system architecture and how a VxD can hook into it:
Mitchell, Stan. "Monitoring Windows 95 File Activity in Ring 0." Windows/DOS Developer's Journal 6 (June 1995).
Re-read the "Disclaimers" section earlier in this article, and then check out the following books:
Schulman, Andrew. Unauthorized Windows 95. Foster City, CA: An International Data Group Company, 1995.
Schulman, Andrew. Unauthorized Windows 95: A Developer's Resource Kit. Foster City, CA: An International Data Group Company, 1995.
In the Tae Kwon Do school I used to attend, the promotion to purple belt was very special because it meant that in your promotion test you had to break a board to pass. At your purple level, you have acquired all the knowledge to write a VxD, but you don't yet know how to do it. Let's go over the necessary materials.
To build a VxD, you will need an assembler or a 32-bit C compiler (or both) to build object files, and a linker that can generate VxD executables. The linker that is shipped with Visual C++™ 2.2 and later accepts the directive /VxD, which has the linker emit an LE file that is required for VxDs. If you choose to use Microsoft tools, you will want to check with the product documentation for Visual C++. Also, to use the assembler, you may want to check with MASM documentation:
Macro Assembler 6.11 for Windows NT (32-bit) (Product Documentation, Languages)
However, you do not need to rely on Microsoft tools to build your VxD. I found one article that describes an alternative tool. This is not an endorsement—I do not imply that this tool is any better or worse than any of the other tools out there. I have never used anything but Microsoft tools and therefore do not know anything about the quality and ramifications of this tool:
Nicolaisen, Nancy. "VxDs Made Easy." Windows Sources 3 (April 1995).
To debug a VxD, you can use either WDEB386.EXE (which, ironically, is documented in Chapter 5 of Programming Tools in the Windows 3.1 SDK) or NuMega's SoftICE for Windows debugger. Here are some articles on the Development Library CD that relate to debugging with WDEB386:
Q58010, "WDEB386 Debugger's Use of COM Port"
Q64123, "PRB: Encountering an INT01 with WDEB386"
Q74605, "Advanced WDEB386 Features and Tips"
Q75252, "Tips on Installing WDEB386"
Q76473, "PRB: No Output from VxD Debug_Out and Trace_Out Macros"
Q77987, "Debugging GP Faults with WDEB386"
Q85897, "WDEB386-Compatible Timing Card Available"
Q105275, "Using the 'BR' Command in WDEB386.EXE"
Q73333, "PRB: WDEB386 Does Not Find Driver Symbol Files"
"Ask Dr. GUI #8: Ctrl+Alt+Del Behavior" (MSDN Library Archive Edition, Technical Articles, Ask Dr. GUI)
"Enhancing WDEB386 with External Debugger Commands" (Technical Articles)
An interesting tidbit of information on kernel debugging can be found in:
Bonneau, Paul and Neil Sandlin. "Debugging Hung Applications Revisited." Windows/DOS Developer's Journal 5 (February 1994).
The sample pointed to by article Q76301, "SAMPLE: Basic Virtual Device (VxD) Structure," contains a VxD skeleton on which you can build your own VxD. A good starting point for Windows 95 VxDs written in C is the CVXD32 sample from the Windows 95 DDK (available on the Microsoft Development Platform).
The following sources can be used for information on the VxD development cycle:
Thielen, D. and Bryan Woodruff. Writing Windows Virtual Device Drivers. Reading, MA: Addison-Wesley, 1994.
Prosise, Jeff. "Adventures in Flatland with VxDs." PC Magazine (May 31, 1994), pp. 291-297.
"What's New in Windows 95 for VxD Writers?" (Technical Articles, Windows Articles, Driver Articles)
"VxD Samples: Demonstrate the Use of a Generic VxD" (Sample Code, Book and Periodical Samples, Microsoft Systems Journal Samples)
Finally, here are some items on the Development Library CD that I found to be useful regarding VxD programming in general:
"Relocatable Object Module Format" (MSDN Library Archive Edition, Technical Articles, C/C++ Articles)
Q65673, "Explanation of Device Development Kit (DDK)"
Q118891, "Installing and Using the Debugging Version of Windows"
Q46734, "Where to Find Information on Writing TSRs"
Q84000, "Obtaining a Virtual Device (VxD) ID Number"
And here are outside items I found that elaborate on a system service:
Bonneau, Paul. "Windows Q&A." Windows/DOS Developer's Journal 6 (March 1995).
Matthews, Don. "Porting VxDs from Windows 3.1 to Windows 95." Dr. Dobb's Journal (November 1995).
Oney, Walter. "Building Device Drivers That Conform to the Windows 95 Plug and Play Standard." Microsoft Systems Journal 10 (December 1995).
One of the blessings and curses (at the same time!) of VxDs is that they are extremely powerful. It is a blessing because it makes the operating system very extensible; it is a curse because there are no defined interfaces between VxDs that do more than they are supposed to do, which makes VxD writing messy at times. There are a number of well-defined purposes of VxDs:
On the brown belt level, we will look at the material that helps you with these tasks.
The following material on the Development Library CD deals with the above issues. Most of these items are sample VxDs or TSRs that demonstrate specific programming techniques.
"Understanding the Windows Virtual Pen" (MSDN Library Archive Edition, Technical Articles, Windows for Pen Computing Articles)
"Callback: A VxD Sample That Demonstrates New Windows 95 Features" (Sample Code, Technical Article Samples, Win32 and Windows NT Samples)
"WIN95VXD: Demonstrates Dynamic VxD Loading Capabilities of Windows 95" (Sample Code, Book and Periodical Samples, Microsoft Systems Journal Samples)
"GDOSMEM: Demonstrates Shared Global Memory" (Sample Code, Product Samples, Languages and System Toolkits)
"TDOSMEM: Passes Data Between a Windows Application and an MS-DOS TSR" (Sample Code, Product Samples, Languages and System Toolkits)
Prosise, Jeff. "DOS Q&A" ( Microsoft Systems Journal, 1995 Volume 10, May 1995) [discusses the use of a hidden VM]
Pietrek, Matt. "Under the Hood" (Microsoft Systems Journal, 1995 Volume 10, September 1995) [discusses a performance-monitoring VxD]
Q94702, "Architecture of Windows Pen Drivers"
Q93469, "Complete Text PW0519: Virtual DMA Services (VDS)"
Q124727, "How to Program DMA for Linear Addresses under Windows"
Q67590, "SAMPLE: VxD Helps MS-DOS Application Query Screen State"
Q68250, "SAMPLE: Capturing Global Interrupts in Enhanced Mode"
Q68975, "I/O Device Serialization and Virtualization"
Q74471, "Using Selectors from _Allocate_LDT_Selector Service"
Q74472, "Initializing Virtual Devices and Hooking Interrupts"
Q74516, "Binding a TSR to a VxD"
Q89494, "SAMPLE: Hot Keys in Windows 386 Enhanced Mode"
Also, the Windows 95 DDK (on the Development Platform) contains a large number of VxD samples. If you have the Platform, check them out.
Here are some third-party articles I found that fit into this category:
Olsen, Thomas. "Making Windows and DOS Programs Talk." Windows/DOS Developer's Journal 3 (May 1992). [discusses the possibilities of VMs to communicate via a VxD]
Bonneau, Paul. "Identify the Running DOS Application from Windows." Windows/DOS Developer's Journal 3 (December 1992).
Olsen, Thomas. "WinGate." Windows/DOS Developer's Journal 5 (January 1994).
Tomlinson, Paula. "Virtualizing a DOS Device Driver with a VxD." Windows/DOS Developer's Journal 5 (May 1994).
Tomlinson, Paula. "Yielding from a VxD: A Windows Cold Boot." Windows/DOS Developer's Journal 5 (July 1994).
Mashlan, Robert. "Capturing Hot Keys in Windows." Windows/DOS Developer's Journal 6 (January 1995).
Bonneau, Paul. "A VxD to Monitor DOS Output." Windows/DOS Developer's Journal 5 (May 1994). This is an update in the material covered in the following article:
Bonneau, Paul. "Windows Q&A." Windows/DOS Developer's Journal 6 (January 1995).
Unlike Windows NT kernel-mode device drivers, VxDs have the interesting property that, besides their original purpose as outlined in the brown belt section, they give you virtually unlimited access to the computer's resources. On the positive side, this property of VxDs allows you to accomplish fairly elaborate tasks, such as making very sophisticated TSRs compatible with enhanced-mode Windows, or even extend the scope of Windows (for example, by providing mechanisms that allow virtual machines to communicate with each other). On the downside, however, VxDs make it rather easy and attractive to take a deep plunge into sleaze country and do very bad things with VxDs, such as providing access to system data structures for applications.
The following information is not for the squeamish. I urge you to use extreme caution when utilizing these sources—think twice or three times before deciding to use any of this in your code. For a few of these techniques, there is a legitimate use; everything else is relying on version-specific and, in parts, undocumented information.
Let's first look at bimodal interrupt handlers (BIHs). This is basically a hack that was introduced into the Windows 3.1 VMM to relieve the problem of poor virtualized interrupt performance. A BIH is a user-mode interrupt handler that can be registered with the VMM to bypass the virtualization engine. There are two sources of information about BIHs:
"Bimodal Interrupt Handlers" (MSDN Library Archive Edition ,Technical Articles, Windows Articles, Driver Articles, Windows [16-bit] Only)
Hazzah, Karen. "Fast Interrupt Handling without VxDs." Windows/DOS Developer's Journal 4 (June 1993).
Note that BIHs are not really a VxD programming issue; you do NOT write a VxD or VxD-related code to implement BIHs. Under Windows 95, the BIH support code has not changed; that is, existing BIHs will still work, but you should seriously consider writing a VxD to handle your interrupts instead.
Other areas that are frequently addressed by VxDs are communication between virtual machines, and supporting virtual machines in the enhanced-mode Windows environment. Here are a few locations of VxD samples and articles that cover this area:
"WINPOST and POSTMSG: Send Messages from an MS-DOS Application to a Windows Application," which accompanies the following Microsoft Systems Journal article:
"MS-DOS Questions & Answers" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1994 Volume 9, November 1994)
Q72899, "SAMPLE: Calling PostMessage from a Virtual Device (VxD)"
Q73185, "SAMPLE: Calling 16-bit Code from a Virtual Device (VxD)"
"VDMD: Helps a Virtual Machine Determine Its Window State," which accompanies the following Microsoft Systems Journal article:
"MS-DOS Questions & Answers" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1994 Volume 9, May 1994)
"V1INSTD: Determines Multiple Instances of the Same MS-DOS VM," which accompanies the following Microsoft Systems Journal article:
"MS-DOS Questions & Answers" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1994 Volume 9, July 1994)
"MS-DOS Questions & Answers" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1994 Volume 9, September 1994) [describes a VxD that allows a VM to emulate a hot key into enhanced-mode Windows]
Q89705, "How VMs Can Communicate with VxDs"
Q120565, "How to Read Text Mode Screen of MS-DOS Session in Windows"
This last subsection contains material that is extremely interesting but also very dangerous:
"VXD: Contains a VxD that Uses DPMI and Accesses the CR3 Register" (Sample Code, Book and Periodical Samples, Microsoft Systems Journal Samples, 1992 Volume 7, October 1992)
Schulman, Andrew. "Go Anywhere and Do Anything with 32-bit Virtual Device Drivers for Windows" (MSDN Library Archive Edition, Books and Periodicals, Microsoft Systems Journal, 1992 Volume 7, October 1992)
Schulman, Andrew. "Vexed by VxDs." InfoWorld (February 27, 1995).
Schulman, Andrew. "Undocumented Corner." Dr. Dobb's Journal (December 1993). [a VxD displaying the list of VxDs]
Zytaruk, Kelly. "Undocumented Corner." Dr. Dobb's Journal (January 1994). [a VM control block dissected]
Zytaruk, Kelly. "Undocumented Corner." Dr. Dobb's Journal (February 1994). [discusses a VM explorer]
Shmidt, Alex. "Undocumented Corner: Calling Ring 3 from Ring 0." Dr. Dobb's Journal (March 1994).
Mueller, Klaus. "Undocumented Corner." Dr. Dobb's Journal (April 1994). [think globally, act locally—about instanced data and Device_CB_area]
Okazaki, Taku. "Undocumented Corner." Dr. Dobb's Journal (September 1994). [global EMM interface]
Knoblaugh, Rick. "A Windows I/O Monitor." Dr. Dobb's Journal (September 1994).
Schulman, Andrew. "Windows 90+5." Dr. Dobb's Journal (January 1995). [excerpts from Unauthorized Windows 95]
Bonneau, Paul. "Windows Questions & Answers." Windows/DOS Developer's Journal 5 (August 1994). [undocumented information on how a VxD can create a VM]
Lawless, James. "Tech Tips." Windows/DOS Developer's Journal 5 (November 1994). [autostart of executables using the Int 21h callout interface]
The most ambitious task I have ever seen a VxD attack is to try to make multitasking, timeslicing TSRs compatible with enhanced-mode Windows. Before there was Windows, there was a rather flourishing niche market for TSRs that multitasked DOS applications. Most of these TSRs would hook the timer interrupt and switch the applications' stacks after a given number of timer ticks.
Being a multitasking operating system itself that uses virtualization as the core of DOS application multitasking, Windows enhanced mode does not work with these TSRs very well. Back when I was a support engineer for Microsoft, I had to deal with two projects along these lines, and I have a fairly good understanding of just WHY it is not a good idea to try to tweak the VMM to support such TSRs. Before I present you with a Knowledge Base article that I never finished about this topic, let me refer you to an article in which a colleague presents his observations about and approach to the same problem:
Offen, Dave. "A Windows VxD for DOS Timeslice Multitasking." Windows/DOS Developer's Journal 6 (May 1995).
A valid (and fairly well-known) technique for TSRs to implement limited multitasking under DOS is to hook into the timer interrupt and switch between stacks (and possibly program segment prefixes, or PSPs) of multiple processes (which may be other TSRs, code sequences embedded in the TSR, or TSR client applications), thereby preemptively multitasking between those processes. While this strategy works well under plain DOS, it is generally very hard to adapt to work with enhanced-mode Windows.
When running in enhanced mode, Windows will provide a virtual environment to DOS applications and TSRs, even if they execute in the context of a protected-mode virtual machine such as the Windows environment executing in the system virtual machine. Here are some of the implications that follow from that design:
To do so, the TSR should observe the INDOS flag provided by DOS as well as the Begin Critical Section (Int 2Fh, ax=1681h) and End Critical Section (Int 2Fh, ax=1682h) calls. Note that there is no possibility for enhanced-mode Windows to be sensitive to a mechanism to prevent reentrancy via a TSR-provided flag. For example, if the TSR code keeps a local flag that indicates whether a process can enter critical code or not, the VMM cannot be made aware of this flag and may still switch VMs and reenter the critical code.
Note that the issues discussed here are but some of the more common problems that can arise while trying to adapt those multitasking TSRs to work with the VMM. Many more issues may come up in specific setups.
In general, it is very hard to make two preemptive multitasking operating systems cooperate with each other. Normally, the necessary effort to adapt TSR-based multitasking operating systems to run while enhanced-mode Windows multitasks below it includes at least the following:
It may very well be the case that this effort is more costly than redesigning the TSR-based operating system altogether. In an ideal situation, the TSR would be deactivated entirely while Windows is running, and mechanisms that Windows and the VMM provide would be used to achieve the necessary multitasking. If this approach is not feasible, there are basically two possibilities to redesign the TSR to make it cooperate with the VMM more easily:
In any case, the enhanced-mode startup broadcasts Int 2Fh, ax= 1605h and 1606h, respectively, can be used to inform the task switcher to switch to Windows-aware behavior. Likewise, the termination broadcasts can be used to reverse that behavior. Note that by using this technique, only one TSR can be shipped that is capable of working both with and without support by Windows.
A TSR may create a new VM by monitoring the enhanced-mode Windows TSR identification broadcast Int 2Fh, ax = 160bh. This broadcast allows the TSR to specify an executable image file that is to be executed in the context of the system virtual machine. If this executable is not a Windows-based application, a new virtual machine is created.
Any attempt to keep the multitasking on the TSR level fully functional even while Windows is multitasking below it is very likely to fail. At the very least, it will require a very good understanding of the architecture and the functionality of the VMM as well as a long development cycle for the adaptation.