Windows requires testing tools that can simulate many keyboard strokes and mouse actions.Audrey Watson
Created: April 15, 1992
ABSTRACT
MicrosoftÒ Test is a tool for testing applications for the Microsoft WindowsÔ graphical environment. Microsoft Test simplifies the task of creating high-quality, robust applications by automating most of the test process.
The user community has affirmed its support for Microsoft Windows by licensing over 7 million copies of Windows since the debut of Windows version 3.0 in May 1990. According to Software Publishers Association, the Windows-based applications segment is the fastest growing segment of the software applications market. With the introduction of Microsoft Test, Microsoft is offering a tool that will result in higher-quality applications for end-users.
Both Microsoft Test and Source Profiler, a tool for analyzing code efficiency, expand the traditional edit-build-debug phases to encompass the complete software development cycle. These products are part of Microsoft’s strategy to provide developers with all the tools they need to create top-quality Windows-based applications.
This article introduces Microsoft Test and explains how the tool simplifies the process of testing Windows-based applications.
There are some persons who claim that they can write bug-free programs. There’s a saying among sailors on the Chesapeake Bay, whose sandy, shifting bottom outdates charts before they’re printed, “If you haven’t run aground on the Chesapeake, you haven’t sailed the Chesapeake much.” So it is with programming and bugs: I have them, you have them, we all have them—and the point is to do what we can to prevent them and to discover them as early as possible...
—Boris Beizer1
Most software developers, whether they develop products for external sale or for internal use, are concerned with the quality of their products. Testing is one way developers can validate the quality of a software product and verify that it fully meets specifications. Because testing helps developers catch and correct many bugs that would otherwise be caught by users, it results in increased user confidence in the products. Testing software before shipment or distribution also saves the developer time and money because it reduces the likelihood of multiple product iterations.
Yet many software developers don’t test their applications thoroughly enough or often enough. Why? Because they’re in a hurry to get their software out, and testing can be an extremely time-consuming process. Because products rarely succeed in meeting specifications the first time, tests generally must be repeated over and over again. As bugs are found, new tests must be developed to verify that the bugs have been fixed, and old tests must be rerun to verify that bug fixes haven’t caused new problems. This testing and retesting can add up to months of testing time during the product’s development cycle.
The best solution is to automate the test process as much as possible. To test Windows-based applications effectively, the standard approaches must be customized to deal with the complexities of the Windows graphical environment. Developers need an automated testing tool designed specifically for Windows. MicrosoftÒ Test is such a tool.
The traditional test approach generally relies on expensive, labor-intensive manual methods. Traditional tests are frequently based on random command selections rather than a predetermined test script.
For example, in one commonly used technique, the tester tries various key or mouse actions at random and visually inspects the results. This approach may seem quick and easy because it requires little advance planning. However, because command selections are random, the tester may overlook potential problems. Human error is also a factor: The tester’s visual inspection could miss discrepancies between actual results and expected results.
Another common technique is beta testing—using a broad group of testers who work in real-world situations and who may be unfamiliar with the product. Although these qualifications increase the likelihood of finding bugs, beta testing is still a random process and is therefore inadequate for finding all bugs. Moreover, obtaining results can take anywhere from several weeks to several months, so beta testing is inappropriate for situations that require a quick response.
One of the main benefits of automating software testing is the time it saves. Manual testing is a labor-intensive process that takes the same amount of time with each iteration. Automated tests may take longer to create but, once created, can be run without human intervention.
Because most products require multiple iterations of a test suite, automated testing can lead to significant cost savings. In fact, a company typically passes the break-even point in labor costs after two or three iterations of an automated test suite.
Because automated testing has low labor costs, companies can afford to test more frequently, resulting in improved software quality. In addition, automated tests can run at night on machines used for development during the day, thus reducing hardware costs as well as labor costs.
Automated tests also produce results faster than manual tests because (a) computers can enter keystrokes and compare results faster than humans, (b) computers can run 24 hours a day, and (c) developers can divide a test suite among multiple computers and run them in parallel. As a result, a test suite may take one or two days instead of three or four weeks to run.
Finally, automated testing helps eliminate human error. A human tester may inadvertently skip items or may not notice that the result of an action differs in some way from the expected result. An automated test does not make such errors—it runs exactly the same way, every time.
However, automated testing isn’t the answer to every test situation. Automated testing requires a functional product that is past the early development stage. A role for beta testing will always exist because it provides valuable real-world feedback, although it is a manual process. A test that is heavily weighted toward automated testing can lead to significant cost savings and better software products.
If automated testing provides these benefits, why don’t all developers of Windows-based applications rely on this technique? One reason is that many of the automated solutions available today were designed for testing character-based applications. These solutions lack the special capabilities required to test Windows-based applications. Other solutions that do contain some Windows-specific tools can cost as much as several thousand dollars, placing them beyond the range of affordability for smaller development operations.
For example, a simple automated testing technique is to write a series of tests and run them in batch mode. However, this technique is unsuitable for Windows-based applications because the Windows environment has no batch mode and because Windows allows input from multiple sources whereas batch tests allow for only one input and one output stream.
Another technique is to use special test systems in which one computer runs the test software and another computer runs the application being tested. This approach is expensive because it requires two computers. Furthermore, special test systems are difficult to use for Windows-based applications unless they include Windows-specific tools for writing test scripts.
What is it about the Windows environment that makes automated testing so challenging?
Windows requires testing tools that can simulate many keyboard strokes and mouse actions.
Application windows can move around and change size and colors in a number of ways, making it difficult to compare actual and expected screen results.
Test scripts may have to deal with unexpected events such as unrecoverable application error (UAE) messages that can shut down an automated test suite completely.
Given these challenges, developers who can’t afford the high-priced automated test tools must rely primarily on manual methods. Even the high-priced tools have their limitations; for example, most rely on bitmap screen comparisons and are unsuitable for situations in which windows (or their contents) change position on the screen.
Microsoft Test is a Windows-hosted tool for developing and running automated test scripts for Windows-based applications. This tool lets users create robust Windows applications faster than any other testing solution available today.
Microsoft Test:
Provides Windows-focused capabilities that overcome automation challenges posed by the Windows environment.
Works with any Windows application, regardless of application size or tools used to create it.
Saves time due to extensive automation capabilities.
Is easy to learn and use.
Provides synergy with other Microsoft tools.
These benefits set Microsoft Test apart from other test tools and make it an ideal choice for testing Windows-based applications.
Microsoft Test is designed specifically for testing Windows-based applications; it contains a host of Windows-focused capabilities that address the idiosyncrasies of the environment. For example, Microsoft Test can simulate a number of keyboard and mouse movements, overcoming the input limitations of other automated test tools. Microsoft Test allows for “fuzzy” as well as precise comparisons between actual and expected results, so it can deal with differences in monitor resolution and with windows that change location, size, and color.
As shown in Figure 1, Microsoft Test can intercept UAEs and other unexpected messages from other running applications or from the Windows environment itself. The test developer can specify what the test script should do when such events occur—for example, log the event and continue or end that test and start the next test. This capability reduces the danger of the test process shutting down if an unexpected event occurs when tests are running unattended.
Figure 1. Trapping and Logging UAEs with Microsoft Test
The UAE-trapping capability also lets the application developer trace UAEs back to their source and thus promotes the development of robust Windows-based applications. Using the logged UAE information, the developer can evaluate isolated code segments to determine where UAEs occurred and can modify the code to prevent them from recurring.
Finally, Microsoft Test gives test developers control over the timing of events because tests can verify that an event happened within a specified time frame. For example, if a test developer wants to ensure that a particular dialog box appears within 15 seconds of a keystroke or a mouse action, the test can be set up to wait 15 seconds and then log a failure (including an error code identifying the nature of the failure) if the desired event doesn’t happen. This capability also lets the test developer control the test speed—for example, to avoid overflowing the input buffers of the application being tested.
Microsoft Test can test any Windows-based application, from simple front-ends to full-featured products. It can even test application components such as dynamic link libraries (DLLs) and application programming interfaces (APIs). Unlike other test tools, Microsoft Test requires no special hooks or debug code and no additional hardware; however, it works only with executable code.
Microsoft Test can test applications created by languages and tools from any vendor—for example, Borland or Asymetrix. Because Microsoft Test is optimized for size rather than speed, it can test applications of any size.2
Microsoft Test’s open-access nature allows developers to add customized functions easily through the use of DLLs. Using DLLs also lets the developer access most of the product’s functionality from other front-ends if desired. For example, a C developer who prefers to work solely in C can write test scripts in C rather than in Basic. In fact, any language that can call a DLL, including macro languages used by applications such as Microsoft Excel, can serve as a front-end for Microsoft Test.
Because Microsoft Test for Windows does not require the presence of the tester to run test suites, it reduces labor requirements for software testing significantly. Microsoft Test automates user input by simulating mouse and keyboard actions and automates the comparison of the actual output with the expected output. Microsoft Test’s use of device-independent and device-dependent comparison techniques reduces the number of tests required. In contrast, tools that rely on bitmapped screen comparisons require separate tests for each monitor resolution (such as EGA, VGA, and Super VGA). Finally, Microsoft Test saves time by logging test results automatically, even when the application being tested fails completely. The tester can choose to display results on the screen or route them to a file.
Several factors contribute to Microsoft Test’s ease of learning and use:
The test script language is Basic, a popular programming language known for its ease of use.
Microsoft Test includes a recorder that generates Basic code by recording actions such as keystrokes or mouse movements. The recorder generates editable code and can be used both as a learning tool and as a quick way to enter user input.
New users can work in FastTest, a subset of Microsoft Test’s Basic language that uses defaults and high-level, English-like functions that facilitate writing test scripts. With FastTest, even users who have no Basic programming experience can write test scripts.
Microsoft Test includes an extensive set of learning and reference materials, including a comprehensive user’s guide and context-sensitive online help. The online help provides full-featured sample applications that users can paste directly into a test script.
Test developers can use Microsoft Test with Source Profiler3 to enhance testing capabilities. Running Source Profiler (with the coverage-tracking option enabled) simultaneously with an automated test suite helps determine which areas of the program are not exercised by the test suite. The developer can then write additional, focused tests to round out the test suite.
Microsoft Test is appropriate for two primary user groups:
Corporate DP/MIS organizations, for testing programs developed for internal use and for acceptance testing applications before they are purchased
Independent software vendors that develop commercial applications for Windows, for verifying that products function as intended before product release
Microsoft Test users fall into two categories:
Full-time programmers. This category includes programmers with dedicated testers and programmers who do their own testing. The latter group can now test products more thoroughly because of the time savings Microsoft Test provides.
Full-time testers. This group also benefits from Microsoft Test because it reduces the repetitive elements of testing and leaves time for creativity.
Microsoft Test users should be familiar with the Windows environment and should be capable of writing macros or simple Basic programs. Users need not know any other programming language.
Microsoft Test consists of a Test Driver, utilities for comparing controls and screen bitmaps, and several additional DLLs.
Test Driver is a Windows-hosted Basic interpreter that provides the environment for developing and running test scripts. Test Driver includes a variant of the Basic language, enhanced with special extensions for testing Windows applications, and FastTest, a subset of Test’s Basic language that uses defaults and high-level, English-like functions.
As the test script runs, Test Driver takes the necessary actions to perform the specified functions. In some cases, Test Driver calls the DLL that stores the specified function, for example, it may call Test Dialogs to compare the actual elements of a dialog box with the expected elements or Test Screen to perform a bitmap comparison between actual and expected screens. In other cases (for example, when trapping a UAE), Test Driver interacts directly with the Windows environment.
The recorder is a time-saving component of Test Driver that allows test developers to generate Basic code for parts of a test script by simply recording their actions. Developers can record keystrokes, mouse clicks, mouse drags, and other mouse movements; they can record actions singly or simultaneously (for example, for a mouse click and drag). Although the test script requires additional code (for example, instructions for comparing and logging results), the recorder provides a quick way to enter repetitive keystrokes or mouse actions. It is also an excellent learning tool for new users.
The Test Dialogs utility provides a device-independent method for capturing and comparing controls such as windows, menu structures, and dialog boxes. Device-independent comparisons are faster and easier to make than device-dependent comparisons because they require only one test instead of separate tests for each type of monitor used.
The Test Dialogs utility performs a two-level comparison:
The first level checks for an exact match. For example, it checks to see whether a dialog box is in the proper location, is the correct size, and has all the necessary items with each item in the proper location and state (that is, highlighted or unhighlighted).
The second level checks for a “fuzzy” match, that is, the actual and expected controls match if the proper items are present and are in the correct state even if their sizes and locations differ.
For example, in Figure 2, Test Dialogs compares the Calculator window with a previously saved version of the window and verifies that all 18 controls (Calculator buttons) are present, are of the appropriate class, are in the appropriate state, and so on. The fact that the Calculator window is in a different location than before (as shown by the difference in RECT values) is irrelevant because this is a fuzzy match.
Figure 2. Comparing Calculator Windows with Test Dialogs
Test Screen, a utility for capturing and comparing screen bitmaps, is useful for comparing items other than controls. For example, it can determine whether a picture shows up in the proper place, is sized correctly, and has the right content. Because screen bitmaps vary according to monitor resolution (EGA, VGA, Super VGA, and so on), this comparison technique is device-dependent and must be repeated for each monitor type used.
Test Screen lets the test developer choose between position-dependent and position-independent bitmap comparisons. For example, picture bitmaps can be compared in size and content only, or in size, content, and location. Position-dependent comparisons should include multiple iterations to allow for location differences in the Windows environment.
Test Dialogs and Test Screen are available as DLLs as well as in user interface versions:
The DLL versions are accessed by Test Driver when running a test.
The user interface versions are useful during test development because they provide the test developer with more information.
Microsoft Test also includes the following DLLs:
Test Event: Simulates keystroke and mouse input (including international characters).
Test Control: Lets the tester verify and change the condition of a specific control.
Test View: Creates a “viewport” window for logging test results to the screen.
The DLL-oriented architecture provides two key benefits:
It conserves system resources because DLLs are linked only when needed and can be unloaded from memory when no longer needed.
Users can call Microsoft Test DLLs through other front-ends if they prefer. The front-end may be written in any language that can call DLLs, including C, Visual BasicÔ, and the Microsoft Excel macro language.
The test developer works in the Test Driver environment to write a test script, using the environment’s extended Basic language. (Alternatively, the test developer may use the FastTest subset of the script language or may generate some of the code automatically by recording keyboard and mouse actions.) A test script consists of a combination of user input (keystrokes and mouse actions), logging instructions, and comparison instructions.
When a test script is running, all elements of the environment—that is, the test script, Test Driver, DLLs, Windows, and the application being tested—are active at once (see Figure 3). The test script interfaces with Test Driver, which in turn interfaces with the Windows environment both directly and through the DLLs. The application runs on top of the Windows environment and interfaces with it in the standard fashion.
Figure 3. Microsoft Test Interfaces
Once started, the test script runs automatically. It sends the specified key and mouse actions to the application, looks at the results of these actions, determines if the results are correct, and logs the results.
The test developer specifies what Microsoft Test should do if it finds a discrepancy between the actual results and the expected results; for example, log the failure and continue with the current test or log the failure, end the current test, and continue with the next test. However, if the test script is written in Fast Test (see the sample script below), Microsoft Test automatically logs the failure and ends the test.
In the following sample test script, the Windows Calculator performs a simple subtraction and sends the results to the Clipboard. The test script compares the results in the Clipboard with the expected value.
******************CLIPTEST.MST******************
**** This script tests the capability of the Windows Calculator
**** to send data to the Clipboard correctly. Lines preceded by
**** asterisks are comment lines. Commands beginning with an "X"
**** are FastTest function names. The F4 command closes the
**** Calculator window.
'$INCLUDE 'FTEST.INC'
****Set log options and start the Calculator:
XLogOn
XLogBanner "Starting Clipboard Test"
Viewport Clear
XStartApp "C:\WINDOWS\CALC.EXE".""
****Perform a subtraction:
XType "888"
XType "-"
XType "333"
XType "="
****Select the Edit/Copy menu items:
XAlt "e"
XKey "c"
****Verify that the expected result of the subtraction (555) is
****the same as the value currently held in the Clipboard:
XClipBoardCmp " 555"
XAlt "{F4}"
END
System requirements are the same as those for Windows version 3.0:
A personal computer using the IntelÒ 286, 386 (386 SX or 386 DX), or higher processor
1 MB of memory (2 MB recommended)
One 5.25-inch high-density (1.2 MB) disk drive or one 3.5-inch (720K) disk drive plus a hard disk
A CGA, EGA, VGA, 8514/A, HerculesÒ, or compatible video graphics adapter and monitor (EGA or higher recommended)
MS-DOSÒ or PC-DOS operating system version 3.1 or later
Microsoft Mouse or compatible pointing device (optional but recommended)
Microsoft Windows version 3.0 or later
Microsoft Test for Windows is the ideal test solution for the developer of Windows-based applications. Unlike most test tools available today, it is designed for the Windows environment and can deal with such challenges as UAEs and windows that change position and size. Because it automates running tests, Microsoft Test can save significant time over manual methods. Microsoft Test can make device-independent comparisons, thus reducing the number of tests required, can run on the production version of the application without special hooks or debug code, and is easy to learn and use.
Microsoft Test for Windows gives developers of Windows-based applications an extra edge by helping them create well-tested, high-quality applications in minimum time and at a low cost for the growing Windows market.
1Boris Beizer, Software Testing Techniques, 2nd ed. New York: Van Nostrand Reinhold, 1990, p. 2.
2The product's efficient use of memory stems from the variant of Basic it uses and from its reliance on DLLs, which can be loaded and unloaded as needed rather than residing in memory at all times.
3Source Profiler, a tool for analyzing code efficiency, works with all Microsoft compilers. It is included in the Microsoft C/C++ Professional Development System version 7.0 and can also be purchased separately.