Microsoft Corporation
1996
The Application Performance Explorer (APE) is a software utility written in Microsoft Visual Basic® to aid in the design, deployment planning, and performance tuning of distributed client-server applications. It allows you to easily run automated "what-if" tests to profile the performance of a multitier application in different network topologies, taking into consideration such factors as network bandwidth, request frequency, data transfer requirements, server capacity, and so on.
In addition, APE is itself an example of a well-designed distributed application. Its Visual Basic source code is well commented and structured to serve as a "template" for component-based client-server applications. This source code can easily be used as the starting point for a custom multitier solution.
This document describes the models of remote deployment supported by APE, followed by instructions for setting up and using the tool. Finally, a brief discussion of component-based design and performance-tuning issues is presented.
Multitier client-server applications partition functionality into separate components that can be distributed on the network in a wide variety of physical deployment configurations. The key performance issue for the designers of these distributed applications is not tuning raw code execution, but determining the optimum network topology for a distributed solution. Developers today need design assistance to determine:
The Application Performance Explorer helps answer these questions by allowing you to quickly set up test cases in various network configurations and to measure performance under different conditions. Exploring alternative scenarios with APE can shed light on questions such as:
To understand the significance of the various factors APE allows you to benchmark, it is necessary to understand the basics of remote component deployment. The following section briefly covers the models of remote deployment supported by APE. If you are already familiar with these models, you may wish to skip ahead to the section titled "Using APE," later in this document.
Remote components can be deployed either synchronously or asynchronously. Each of these approaches has its advantages and disadvantages. Different parts of your application can use different models, depending on the needs of the tasks being implemented.
APE provides built-in models for remotely executing and managing objects both synchronously and asynchronously. These models are described below.
Note In the discussion that follows, you will notice that APE models remote tasks using two separate components working closely together: a worker and a service. This is good component-based design, separating the execution context (worker) from the actual computational routines or business services (service). Since remote components are most efficiently packaged as in-process servers, they need a process space on the remote machine in which to execute. The worker component provides this process space and execution thread, while the service encapsulates the application-specific functionality.
The Application Performance Explorer implements synchronous connections as a direct request from a client application to an object that lives on a remote network server. Because the connection is synchronous, the client application waits for the task to be completed and is blocked until the server returns.
Figure 1. The Synchronous Model
As shown in the illustration above, (1) the Client creates a Worker, which in turn creates a Service that performs the work. When the Service is finished, both the Worker and Service are destroyed. (2) Additional Clients create additional Workers and Services, never sharing them with other Clients.
Direct object instantiation and executing repeated calls on a method is most useful for exploring network latency. Changing calling parameters and return value sizes can help determine "over-the-wire" throughput, which can be useful in deciding which components should be run across a network on remote computers and which should remain together on a single machine.
One of the limitations of the direct approach is the overhead involved in starting the Worker and Service objects. When a Client is done with a Worker, both the Worker and Service are destroyed.
A better approach is to keep Workers "alive" and reduce the overhead of initializing them. There are a number of approaches to this, including "Pool Manager" and "Queue Manager" scenarios. APE implements a Queue Manager, as shown below.
Figure 2. The Asynchronous Model
In the simplest case, (1) the Queue Manager creates a given number of Workers, all of them initially having a "not busy" status. (2) When a Client request comes in, the Queue Manager allocates a Worker, marks it as "busy," (3) uses the Worker to perform the Service until it is completed, and changes its status back to "not busy." (4) The Queue Manager accepts all requests and allocates Workers on a first-come, first-served basis:
Because the Queue Manager maintains its queue internally, Client requests are never refused. If all pre-allocated workers are busy, the Queue Manager simply waits until one becomes available and assigns it to the next waiting Client in turn. In distributed solutions, such a queue can be used to keep workers busy. This results in servers running at their optimum performance level—that is, fully loaded.
The Queue Manager illustrated in Figure 2 showed only the minimum steps necessary to accomplish the work. No provision was made for either returning status information to the Client or logging transaction information for administrative purposes.
In general, return information can be conveyed in one of two ways: either synchronously through a return value, or asynchronously through an indirect notification mechanism. As discussed earlier, the synchronous (direct) method results in the Client program being blocked waiting for the return to complete. For asynchronous operations, a separate notification mechanism is required. OLE callbacks, available in Visual Basic, provide such a mechanism. Callbacks allow the Client to continue processing and be notified asynchronously when the Server side has something relevant to report.
To use callbacks in the Queue Model, an additional component—the Expediter—is used to queue return information back to the Client.
Figure 3. The Queued Objects Model with callbacks
In operation, the Client implements an internal Callback class with a predefined method.
This diagram also illustrates the Queue Manager, Worker, and Expediter all feeding into a Logger.
The Application Performance Explorer has been designed to help you explore the distributed application scenarios outlined above in your environment and on your own machines. Building on the above models, the Application Performance Explorer allows you to control a variety of parameters and determine their overall impact on your application. While DPE includes implementations for all distributed models using "empty" routines, DPE is designed to let you easily specify your own component objects as the Service.
The Application Performance Explorer consists of a single application window comprising three areas: the upper area allows you to load or save a "profile" of the connections and settings to conduct a particular test, the middle area contains tabs for controlling the parameters of each of the components, and the lower area contains summary statistics.
Figure 4. APE application window
Three separate setups are provided, as described below:
The Manager installation includes all components necessary to use the Application Performance Explorer, including server-side components. APE is fully functional when run on a single machine; although in this configuration, it is unable to remote server-side components. To quickly use APE, run the client setup on one machine, start the application, choose any provided profile, and click Start.
To perform a remote test, it is necessary to use the APE Server or Remote Client installation on a second machine, to ensure that a network connection is possible, and to set remote automation permissions on the server components.
During setup on the second (or server) machine, permissions will be set to allow remote activation on the classes used by APE:
Note that the permissions can later be modified using the "Remote Automation Connection Manager," which can be found in the "Remote Automation Management" folder or program group. To perform the remote test:
Each of the supported models for distributing components uses the same set of component object types, with user settings to control various aspects of their functionality. The APE Manager interface contains tabs for each of these component types, the settings of which collectively become the "profile" of the scenario to be modeled.
Each component type independently provides the option of displaying status forms and logging event information. These features are enabled by selecting the two check boxes at the bottom of each tab.
When the configuration options for each object type have been entered, click the "+" button to the right of the Profile list box to save your settings as a new profile.
The major Client parameters are the number of Clients to be created, the number of calls that each Client makes, and—if used—the size of the data to be passed. Varying the number of Clients is useful for "loading" the system and for examining the advantages of a queue manager with extremely large numbers of Clients. Note that it is often impractical to setup 100 or more PCs to test the effect of a large number of users connected to the system. It is, however, more practical to set up 10 clients and increase the call frequency to simulate high-traffic conditions.
The number of calls is most useful for examining the effect of "component persistence": when a component stays in memory between calls, the system spends less time creating and destroying objects and can support a much higher volume of work. Varying the size and type of the data can also be useful in discovering major performance drop-offs in network throughput.
If remote clients will be used in your test, select the Remote Clients check box to enable the Configure… command button. This brings up a Client Machines dialog box to specify their addresses and connection information.
Figure 5. APE Manager: Clients tab
The Asynchronous tab configures the Queue and Expediter components, which are only used in the Asynchronous models, as described earlier in this document. If you select a Synchronous model, the tab changes to Synchronous and the options are disabled.
Figure 6. APE Manager: Asynchronous tab
The Worker component presents four configurable parameters.
Figure 7. APE Manager: Workers tab
The Service component included with the Application Performance Explorer simulates real work: it can be made to take a specific amount of time to complete and—when investigating loads on server machines—use up processor time.
Regardless of the work simulation determined by these settings, the only "real" work performed by APE's built-in Service components, is to return the data requested by the Client. For more accurate measurement, you can substitute your own object method as a Service. To do this, you specify a string of the form "Library.Class.Method," where Library is typically the name of your object server, Class corresponds to one of its publicly exposed object classes, and Method is any method implemented by the Class that requires no parameters.
Figure 8. APE Manager: Services tab
The Logger component simply builds a .LOG (text) file to record the step-by-step progress of your test and the actions of each of its components. Because of the performance impact of logging to disk, the log can be maintained in memory and written either when the scenario is completed or when a predefined maximum log size is reached.
Figure 9. APE Manager: Logger tab
Note APE is configured to launch the Windows Notepad program to view the log, in response to pressing the View button. In some cases, however, the log file can grow quite large and exceed the capacity of Notepad. If you encounter a message indicating that the file is too large, manually open the file with another viewing program, or rerun the tests with less logging information or for a shorter duration.
The Connection tab is used to configure the connection to the remote object classes. It can be used for Remote Automation and—if the system supports it—Distributed COM (DCOM). The Connection tab is merely a single interface for remoting all the object classes used by the Application Performance Explorer: it simplifies the remoting process described in the section "Setting Up and Testing APE" earlier in this article.
Note In order to use Distributed COM (DCOM), you must be using Windows NT 4.0.
Figure 10. APE Manager: Connection tab
The Application Performance Explorer is a flexible and powerful tool, and there are a number of interesting questions you can answer with it. Most importantly, you can directly examine the effects on throughput of varying the major input parameters: the number of calls, the size of objects, whether the Service is in-process or out-of-process, and so on.
Other questions APE can help you explore include:
The first and most important question that a designer must answer in considering how best to take advantage of the potential for distributed computing—and whether a specific application task should in fact be remotely executed—is this: Is the overhead associated with the remoting worth the benefit?
To answer this question a number of performance, maintenance, and administrative factors must be considered. No tool can answer all of these questions, but the Application Performance Explorer can go a long way toward answering the most important performance questions.
Considered most broadly, the question of whether or not to remotely deploy a particular task requires considering the following parameters:
| Task size (TS) | How big is the task being considered for remoting? |
| Request frequency (RF) | How often does the task need to be executed? This includes requests from all users, not just a single desktop machine. |
| Network bandwidth (NB) | What is the raw speed of the network cable, network interface card(s), network protocol, RPC, and drivers, as well as the collision/retry overhead caused by network traffic? |
| Data transfer requirements (DT) | How much data needs to be transferred to and from the task? (Note: In some cases this may be negative because the location of the task on a remote server may actually reduce data transfer requirements to the desktop.) |
| Desktop system capacity (DC) | Primarily, what is the raw bandwidth of the desktop system's CPU(s), Memory, Disk I/O, and other I/O devices that are needed to process the task? |
| Remote system capacity (RC) | Primarily, what is the raw bandwidth of the remote system's CPU(s), Memory, Disk I/O, and other I/O devices that are needed to process the task? |
| Desktop opportunity cost (DO) | What could the desktop machine be doing if it were not running the task? |
| Server opportunity cost (SO) | What could the server being doing if it were not running the task? |
| Desktop resource burden (DB) | Primarily, what burdens do the OS and other applications impose on the desktop system? |
| Shared resource burden (SB) | Primarily, what burdens do the OS and other applications impose on the shared system? |
| Application design & implementation (AD) | How efficiently are the above factors considered in the design and implementation of an application? (Note: Because of the simplicity of Visual Basic 4.0 and Remote Automation, the programming question of how hard it is to use the resources of a remote system vs. the resources of a desktop system is not considered a factor.) |
| Centralized administration | What is the benefit of administrating or updating the task from a centralized location rather than each desktop. |
From the parameters listed above, it is possible to compose a general "equation" to answer the question of whether to remote a task or not:
If DO or CA is "big enough" or if (DO > SO and f(TS, DC, DB) > f(TS, RF, NB, DT, RC, SB, AD)) then the task should be remoted.
Obviously, it will usually be impossible to find a simple quantitative answer to this equation. Its purpose here is primarily to expose the issues so that the factors that are important in any particular scenario can be examined in sufficient detail. The Application Performance Explorer, along with prototyping and other analytical tools, can provide meaningful insights and measurable answers to many of these questions.