Enhancing WDEB386 with External Debugger Commands

Manfred Schluttenhofer, Microsoft Corporate Support, Central Europe
With support from Ruediger R. Asche, Microsoft Developer Network Technology Group

October 9, 1995

Click to open or copy the files in the VDYNDEBD sample application for this technical article.

Abstract

This article discusses three VxD-related features: writing dynamically loadable VxDs for the Microsoft® Windows® 95 operating system, loading such VxDs using the Win32® application programming interface (API), and supplying external WDEB386 commands in a VxD. These features are illustrated by VDYNDEBD, which is a VxD written for Windows 95. Although VxDs for Windows 95 may be written in C or C++, VDYNDEBD is written in assembly language to keep the source code as small as possible. The article assumes that you know how virtual device drivers for Windows 3.1 work.

Making Your VxD Dynamically Loadable

There are two ways to load a VxD that is dynamically loadable:

The two techniques are not equivalent. When a Win32 application opens a handle to the VxD, the Win32 virtual device (VWIN32) uses the VXDLDR's LoadDevice service, and then sends a W32_DEVICEIOCONTROL notification message to the loaded VxD.

Consequently, to make your VxD dynamically loadable, the control procedure of your virtual device has to handle at least two system control messages: SYS_DYNAMIC_DEVICE_INIT and SYS_DYNAMIC_DEVICE_EXIT. If your VxD can be opened from inside a Win32 application, the control procedure must also handle the W32_DEVICEIOCONTROL notification as described in the "Handling the WIN32_DEVICEIOCONTROL Message" section below. It is probably a good idea for your VxD to handle the WIN32_DEVICEIOCONTROL message in any case, just to be prepared to be called from a Win32 application.

The VDYNDEBD control procedure handles this minimal set of notifications:

BeginProc VdyndebD_Control
    Control_Dispatch SYS_DYNAMIC_DEVICE_INIT, VdyndebD_Dynamic_Init
    Control_Dispatch SYS_DYNAMIC_DEVICE_EXIT, VdyndebD_Dynamic_Exit
    Control_Dispatch W32_DEVICEIOCONTROL, VdyndebD_DeviceIOControl
    clc
    ret
EndProc VdyndebD_Control  

Handling the SYS_DYNAMIC_DEVICE_INIT Message

The SYS_DYNAMIC_DEVICE_INIT message will be sent only once (namely, when the VxD is loaded), so you may put the code and the data used while processing this message in discardable segments. Any initialization necessary for the VxD should be done in this handler. To succeed in the initialization, EAX should be set to 1, and the carry flag should be cleared before returning from this handler. Here's an example:

BeginProc VdyndebD_Dynamic_Init
    mov eax, 1
    clc
    ret
EndProc VdyndebD_Dynamic_Init

Handling the SYS_DYNAMIC_DEVICE_EXIT Message

SYS_DYNAMIC_DEVICE_EXIT is the last message the VxD receives when it is unloaded. All necessary cleanup should be done in this handler. As in the SYS_DYNAMIC_DEVICE_INIT message, EAX should be set to 1, and the carry flag should be cleared before returning from this handler. For example:

BeginProc VdyndebD_Dynamic_Exit
    mov eax, 1
    clc
    ret
EndProc VdyndebD_Dynamic_Exit

Handling the W32_DEVICEIOCONTROL Message

The W32_DEVICEIOCONTROL message is sent to the VxD right after the SYS_DYNAMIC_DEVICE_INIT message, when the first Win32 application opens a handle to the VxD. Unlike SYS_DYNAMIC_DEVICE_INIT, the message is sent again for each application that opens the VxD and for each application that closes the VxD. To distinguish between control codes, you need to check the dwIoControl member of the DIOCParams structure that ESI points to when the VxD receives this message. The value of dwIoControl is set to DIOC_OPEN when the VxD is opened, and to DIOC_CLOSEHANDLE when it is closed. Application-specific initialization and cleanup can be done here.

To succeed in a request to open, your VxD must set EAX to 0, and the carry flag must be cleared before returning. Upon returning from a request to close, EAX must be set to VXD_SUCCESS, and the carry flag must be cleared. When a Win32 application calls a VxD-defined I/O control function using the DeviceIoControl API, the VxD will receive another W32_DEVICEIOCONTROL message. This technique is not discussed in this article; however, your VxD should be prepared by setting EAX to 50 (ERROR_NOT_SUPPORTED) and clearing the carry flag for unsupported dwIoControl values before it returns. For example:

BeginProc VdyndebD_DeviceIOControl
    mov  eax, [esi+12]
    cmp  eax, DIOC_OPEN
    jz   short DeviceIOControlOpen
    cmp  eax, DIOC_CLOSEHANDLE
    jz   short DeviceIOControlCloseHandle
    mov  eax, 50

  DeviceIOControlDone:
    clc
    ret

  DeviceIOControlOpen:
    xor  eax, eax
    jmp  short DeviceIOControlDone

  DeviceIOControlCloseHandle:
    mov  eax, VXD_SUCCESS
    jmp  short DeviceIOControlDone
EndProc VdyndebD_DeviceIOControl

Loading Your VxD Using the Win32 API

More than one application can open and close a dynamically loadable VxD. No matter how many applications open the VxD, it will be loaded into memory only once: The VxD will be loaded into memory when the first application opens it, and it will be unloaded after all applications that opened the VxD have closed it. The sample source code accompanying this article includes a simple Win32 application called LOADVXD, which can be used to load and unload the VDYNDEBD sample VxD. To open a dynamically loadable virtual device driver from within a Win32 application, you should use the Win32 CreateFile function. The lpFileName parameter to the CreateFile call should point to a string in the form "\\.\vxdname," where vxdname is the actual filename of the VxD to be opened. The dwDesiredAccess, dwShareMode, pSecurityAttributes, and hTemplateFile parameters do not matter when opening a VxD and may, therefore, be set to 0. The dwCreationDistribution parameter should be set to OPEN_EXISTING, and dwFlagsAndAttributes should be set to FILE_FLAG_DELETE_ON_CLOSE. For example:

HANDLE hVxD = 0;

hVxD = CreateFile("\\\\.\\VDYNDEBD.VXD", 0, 0, NULL, OPEN_EXISTING,
                  FILE_FLAG_DELETE_ON_CLOSE, 0);

To close the previously opened VxD, simply call the Win32 CloseHandle function with the handle returned by CreateFile. For example:

CloseHandle(hVxD);

Supplying External Debugger Commands

Every virtual device driver can supply external commands (.DOT commands) that kernel-mode debuggers such as WDEB386 can use, by handling the Debug_Query system control message. When the user enters the command ".vxdname" (where vxdname is the actual name of the VxD) in the debug terminal, the control procedure of the VxD will receive a Debug_Query message. Here is an example of a control procedure that handles this message: 

BeginProc VdyndebD_Control
    Control_Dispatch Debug_Query, VdyndebD_DebugQuery
    clc
    ret
EndProc VdyndebD_Control

Handling the Debug_Query Message

While processing the Debug_Query control message, a VxD may use the Trace_Out macro to send a menu with options to the debug terminal. The virtual machine manager provides services such as In_Debug_Chr and Is_Debug_Chr to read from the debug terminal. Using the Trace_Out macro and these services, you can easily perform tasks specific to the kind of device your VxD is dealing with, and provide information that would not be available otherwise. For example:

BeginProc VdyndebD_DebugQuery
    Trace_Out " "
    Trace_Out "*** VDYNDEBD DEBUG INFORMATION ***"
    Trace_Out " "
    Trace_Out "[1] Option 1"
    Trace_Out "[2] Option 2"
    Trace_Out "[3] Option 3"
    Trace_Out " "
    Trace_Out "Please select an option: ", nocrlf

    VMMcall In_Debug_Chr

    Trace_Out " "
    Trace_Out " "
    jz   short DebugDone
    cmp  al, '1'
    jz   short DebugOption1
    cmp  al, '2'
    jz   short DebugOption2
    cmp  al, '3'
    jz   short DebugOption3
    Trace_Out "Invalid VDYNDEBD debug option"
    Trace_Out " "

  DebugDone:
    clc
    ret

  DebugOption1:
    Trace_Out "VDYNDEBD debug option 1"
    Trace_Out " "
    jmp  short DebugDone

  DebugOption2:
    Trace_Out "VDYNDEBD debug option 2"
    Trace_Out " "
    jmp  short DebugDone

  DebugOption3:
    Trace_Out "VDYNDEBD debug option 3"
    Trace_Out " "
    jmp  short DebugDone
EndProc VdyndebD_DebugQuery

Further Reading

For a complete description of the functions, services, macros, and messages described above, see the Microsoft Windows 95 Device Driver Kit (DDK) documentation in the Microsoft Development Library. Some details, such as the description of the W32_DEVICEIOCONTROL system control message, are not included in the DDK but can be found in the Win32 Software Development Kit (SDK), also in the Development Library.

The following articles provide other useful information on related topics:

Asche, Ruediger R. "What's New in Windows 95 for VxD Writers?" April 1994. (Development Library, Technical Articles)

Oney, Walter. "Extend Your Application with Dynamically Loaded VxDs Under Windows 95." Microsoft Systems Journal 10 (May 1995). (MSDN Library, Books and Periodicals)