Testing Driver, Filter, and Services Applications on Microsoft Windows 2000
Noel Nyman
Windows NT Applications Test
May 1998
Summary: While testing applications on prerelease builds of Microsoft® Windows® 2000, we've found categories of applications that share similar problems (bugs). We refer to the applications as "driver applications" because they have one or more of these characteristics:
- They install a driver during system boot or user log on
- They use a file filter
- They install a Windows NT service
Antivirus, remote control, backup, disk defragmenter and redirector applications are typical driver applications.
This article discusses the types of bugs the Windows NT Applications Test team found and the testing methods that expose them. The test methods described here may help you identify bugs exposed in your applications on Windows 2000. Our goal is to release Windows 2000 with as many applications as possible running bug free on the operating system.
Verify the tests discussed here against your existing application test plans to ensure that the bugs found by these methods are also exposed by your regular tests. We use applications on pre-release builds of Windows NT to find Windows NT bugs. We cannot test applications exhaustively. The tests described in this paper are not meant to be comprehensive for any application.
Bluescreens
The most disturbing bugs a user sees when running Windows NT result in a "bluescreen," the character mode screen with a blue background and cryptic lists of driver names with addresses in hexadecimal. The bluescreen is known colloquially as "the bluescreen of death" because it means Windows NT has crashed, as far as the user is concerned. The only recourse is to manually reset or power cycle the computer. Often, data not saved to disk is lost in the process.
User-mode applications rarely cause bluescreens. Windows NT usually isolates user processes from the operating system to prevent such problems. Driver applications can cause bluescreens because they are more closely linked to the operating system than user applications.
Bluescreens During Startup and Log On
We've seen several driver applications that bluescreen immediately when the computer is rebooted after installation or when the user first logs on. In most cases, these bluescreens are consistent and, if they happen with your application, you've probably already seen them if you've tested it on a Windows 2000 beta build.
Some of the startup and log on bluescreens are not consistent. They may appear intermittently, or on some test systems and not on others. To find inconsistent or infrequent bluescreens, try these test scenarios:
- Install your application on a checked build of Windows NT. The checked build debugging code may catch problems that are handled or ignored by free builds. The additional code in Windows NT checked drivers changes the address relationships of drivers and filters, and has exposed pesky intermittent bugs.
- Install debug versions of your application's drivers. They may expose bugs in ways similar to checked Windows NT builds.
- Install other applications on your test computers. Pick applications that your customers are likely to use at the same time as your application. Common choices might be office suites, Web browsers, utilities, information managers, development environments, and games. Vary the installation scenarios so that your application drivers are not always loaded in the same sequence with other drivers.
- Use a variety of test platforms that reflect the range of your application. Install on FAT, FAT32 and Windows NTFS file systems. Use a variety of video hardware, resolutions, and color depths for applications that may install different files for some configurations. Connect a variety of serial and Universal Serial Bus (USB) devices when testing installations.
Bluescreens During Regular Operation of Your Application
Tests in these categories can help you find bluescreen problems that occur after startup and log on.
File access tests
Driver applications are often intimately involved with file access. Here are some of the tests we use that expose bluescreen bugs:
- Format a disk.
- Save files of varying sizes.
- Delete files, with and without the recycle bin.
- Save several files using the same names repeatedly to the same folders. Overwrite both existing and deleted files.
- Try to change locked files.
- Try to access a drive from a command window or another application after your application has accessed it.
- Dismount disks from drives while your application is active.
Local drives
We see bluescreen bugs exposed by tests on some types of drives, but not on others. Try to include one example of each of these in your local drive test scenarios:
- FAT
- FAT32
- NTFS 4.0 (uncompressed and compressed)
- NTFS 5.0 (uncompressed and compressed)
- Large drives
- Over 2GB and over 4GB. Also check that your application doesn't misreport total drive size and free space.
- Volume sets
- Stripe sets
- Floppy drives
- CD-ROM drives
- Removable media drives, such as Zip and Jaz drives
- Integrated Device Electronics (IDE) drives
- SCSI drives
- Tape
- IDE
- SCSI
- floppy drive controller
- parallel port
Mount points
The drive that hosts a mount point must be a Windows 2000 NTFS drive, but the mounted share can be any type. If your application makes the assumption that all folders will be Windows 2000 NTFS, you may see bugs when the application processes these mount point types:
- FAT
- FAT32
- NTFS 4.0
- Compressed (if the host is uncompressed)
- Uncompressed (if the host is compressed)
- Distributed File System (DFS)
- DOS, Windows for Workgroups
- Windows 95
- Novell
Built-in defragmentation
Windows 2000 is scheduled to release with limited feature defragmentation. The defragmenter is available on prerelease builds as a preview. Test your application both before and after the defragmenter is used on local drives.
Network drives
If your application works with network or mapped shares, test each of these:
- Windows NT FAT
- Windows NT FAT32
- NTFS 4.0 (compressed and uncompressed)
- NTFS 5.0 (compressed and uncompressed)
- DFS
- DOS, Windows for Workgroups
- Win95 FAT
- Win95 FAT32
- Novell shares
- Shares mounted with third party redirectors
Test Your Application while Other Applications Run
Occasionally your application tests bug free, but other applications will have problems because of changes you've made in the operating system. While your application is running, install and test some of these types of applications:
- Applications that access drives and files your application monitors
- Applications that your customers may commonly use
- Applications that access system resources such as:
- a Printer port
- COM ports
- a Game port
- a USB
- a Network Interface Card (NIC)
Use your application on "typical desktops" for several days of user activity. For applications that run on servers, use load and stress tests. Beta testing is also a good source of "typical desktops" user bugs.
Failure to Initialize
We've seen several applications that fail to start a real-time monitor function, but don't warn the user that the function isn't working. Antivirus program users, for example, may assume they're protected against viruses when opening files from a floppy disk or network share, even in the absence of a real-time monitor icon on the taskbar, unless your application specifically warns them about a failure to initialize this feature. The best solution is to make sure the feature properly starts on Windows 2000. However, if the feature fails, check that one of these events happens:
- A prominent message to the user appears during startup or log on when initialization fails
- A failure message is added to the event log
If your application displays an icon on the Windows NT 4.0 taskbar, make sure it appears on Windows 2000 when the functionality is active. Use a separate "disabled" icon when the functionality is not available, rather than just removing your icon from the taskbar.
Other Driver Application Bugs
The bugs in this section seldom result in bluescreens and they may not cause serious problems, such as data loss. But they are obvious to the user and we have found each of them in more than one application. When testing your application on Windows 2000 be sure to check the following:
- Replacing system files. Some applications try to replace system files with older versions. Windows 2000 generally won't allow that to happen and the installation of your application is blocked. There are often workarounds so that the user can install your application, but they are tedious and unnecessary.
- Using version information incorrectly. Your application may fail to install on Windows 2000 because it misinterprets the Windows NT version. Windows 2000 identifies itself to applications from the GetVersionEx function. Some applications assume that Windows NT will always be version 3.51 or 4.0. Other applications assume they won't work on any Windows operating system that's not version 4.0.
- Program group. Make sure your installer created only one program group and all the shortcuts are correct.
- Default installation path. Check the default path the installer displays. We've seen a few installation programs that either repeat the drive letter or use incorrect characters that Windows NT won't accept in a path name. Don't assume that C: is the system drive. Use the environment variables %SystemDrive%, %windir%, and %ProgramFiles% (new in Beta 2 of Windows 2000) to locate the system drive and path for the application.
- Installer errors. Some versions of InstallShield display a "string too long" error and give the uninstaller shortcut icon in the program group a bogus path.
- Video driver issues. Applications that work closely with the video system, such as remote control applications, should check for appropriate integration with the full range of video drivers and resolutions. Your customers will be using the new multiple monitor features in Windows 2000 and you'll want to make sure your application can see and work with all their monitors.
- Changes in disabling tape drivers. If your application uses its own tape driver, be sure to change the message you give to users when you find the system tape driver in use. On Windows 2000 users disable the tape driver in the Microsoft Management Console. If they remove the driver, as they did on Windows NT 4.0, the Windows NT Plug-and-Play feature may reinstall the driver next time the user restarts Windows 2000.
- Application log and report files. Test your application's log file feature thoroughly. Some applications find they can't write to the log file. Others do write to the file, but give users a message saying they can't write to it. In a few cases, the application access violates when users create log files or print report files using an existing file name.
- Application messages. A few applications give users incorrect messages. For example, they report scans canceled by the user when the scan terminated normally without user intervention.
- Boot records. Some applications that scan disk drives report they can't see some or all boot records. Make sure all calls to CreateFile, ReadFile, and WriteFile use sector aligned buffers. See the Platform SDK for more information.
- Different behavior between platforms. Some applications that provide versions for both x86-based and DEC Alpha-based processors exhibit different behavior on Alpha than on x86. Run your full test suite on both platforms.
- Browser integration. If your application uses the installed browser to access local files or connect to Internet sites, test its integration with Internet Explorer 5.
- System resources. Either because of bugs or changes between Windows NT 4.0 and Windows 2000, some applications can't view and display system resources such as the recycle bin, CPU utilization, and the event viewer.
- CD AutoRun. Test AutoRun on a CD-ROM after your application is installed. In a few cases, applications accidentally prevent this feature from working.
- Installing application fonts. Occasionally application fonts are not properly installed in the system. The fonts may be visible until the user shuts down Windows 2000, but they're unavailable when the user starts Windows NT again.
- Help files. Some applications don't display all or some of their own help files.
- Verifying users. Applications that require a valid computer or domain user don't always accept valid users on trusted domains. Test your application with user names from a variety of domain configurations.
- Application process does not close. In a few cases, the application process does not close when the user exits the application even though the parent window is no longer visible. Often the user can't restart the application and receives the mystifying message that the application is already running, even though they can't see the application's windows.
- Long timeouts waiting for resources. Some applications wait an excessive amount of time for resources they expect to see. For example, if your application is installed on a computer with a network card, but the computer is not currently connected to the network, make sure your application doesn't appear to "hang" waiting for the network to become available.
- Uninstalling. Some applications don't register an uninstaller and the user will not be able to uninstall them from the Application Wizard. Even when an uninstaller is available, the application may not uninstall. Some uninstallers hang and others don't remove all the files, folders, and registry entries that they did on Windows NT 4.0.
Migration Issues
We want users to be able to migrate from a Windows 95 or a Windows NT 4.0 installation directly to Windows 2000 with all preferences, privileges, and applications working after migration. Microsoft has provided the Migration DLL feature so that vendors can make changes necessary to allow Windows 95 applications to run successfully on Windows 2000 after migration. Application migration from Windows NT 4.0 to Windows 2000 should not require special code. Test migration for your application by installing on Windows 95 and/or Windows NT 4.0, then installing Windows 2000 in the same %systemroot% folder. Run your full regression test suite after Windows 2000 migration, paying particular attention to the special issues raised in this article.