David Willson
Microsoft Developer Network
August 1999
Summary: Discusses the methodology used to calculate the performance goals for Phase 4 of the Duwamish Books sample application and provides a sample calculation to determine performance goals for a hypothetical business scenario. (6 printed pages)
In this article, I will explain the methodology we used to calculate the performance goals for Phase 4 of the Duwamish Books sample application and walk through the actual calculations for a sample business scenario. Because Duwamish Books is a scalable application, the data we used in this scenario can be easily substituted with data more applicable to an actual business scenario. To read more about how we tested the Duwamish Books system for performance, including test results, see the article "Performance Testing a Scalable Application."
The Duwamish Books system is built to record and render data related to our fictitious bookstore, including the bookstore catalog and the sales and order information. Because Duwamish Books is a scalable system, the number of database requests the system can handle is limited only by the hardware used to run the system. Therefore, setting performance goals for a system is an important part of planning a system installation.
The process of defining performance goals is an exercise in gathering and organizing information. In establishing performance goals for Duwamish Books, we need to determine the business goals and determine the areas of the application that are critical to these goals.
We also need data on how the system will be used, specifically customer behavior patterns in regard to online book sales. This is where we can use market research to help us determine the relationship between the amount of traffic to our Web site and the amount of sales we can expect.
With a scalable system, we can meet almost any performance goal by supplying more physical resources. However, to calculate performance goals, we need to start with the overriding business goal. In our case, we have a system that is designed to generate revenue from sales. So, to begin our performance goal calculations, we start with the revenue we expect to generate from the system and then convert this figure into performance goals for each applicable functional area.
Because different parts of an online system offer different functionality—some to be used by the customer, some by employees—performance expectations will vary from one part of the system to the next. So, we first need to identify the relationship between a functional area and our performance goals.
For example, with Duwamish Books we really care about the performance of the search engine because we expect our customers to interact with this part of the site the most. In contrast, we should consider separately the amount of time needed to update the data in the catalog. The rate that updates are completed in these tables won't affect our annual revenue nearly as much because these updates will occur much less frequently than searches. Because customer searches are more important, we have optimized these tables for data selection rather than updates. Hence, our performance goals for the sample are based on the speed of data retrieval for these tables.
On the other hand, the order entry part of the system stores all information about customer orders. In a healthy business, the tables in this part of the database will be constantly updated. Therefore, performance goals for this part of the system are set on the speed of inserts and updates.
Customer behavior changes with time and from one retail category to the next. To improve the accuracy of your performance goal calculations, use the most recent and reliable market research figures.
For the purposes of testing, the performance goal must be expressed in a way that is measurable in our testing routines. For example, the Microsoft Web Stress tool we used in testing the sample measures performance in units of requests per second. The hypothetical Duwamish Books business goal is expressed in dollars per year. The main exercise in calculating the performance goal is to choose conversion ratios that translate the business goal (dollars per year) into the units of our performance goal (requests per second).
The precision of the formula relies on the source of information. For example, we will need to know how many requests and what type of requests are made in order for the system to yield some number of sales. This information requires a combination of market research and a study on the way the customer will use the system.
We should expect the outcome of the formula to vary with respect to the figures we supply. For the formula that converts annual sales into requests per second, we should expect that if we double our annual sales figure, the requests per second figure should increase accordingly.
We have prepared an example calculation to determine performance goals for a hypothetical business scenario.
Given this scenario, we'll show how to translate an arbitrary business goal into a certain number of Internet server requests. We will also calculate the performance goal for a particular type of search.
The Duwamish Books business scenario we created for our sample performance calculations assumes the following:
Data on user behavior is harder to come by and requires a combination of market research and user studies. We used a recent market research report to help us determine the ratio of number of visitors to our site to the number of orders. The report titled "The State of Online Retailing," a Shop.org Study conducted by the Boston Consulting Group, November 1998, concludes, "only 5% of all unique visitors become customers and only 2% of all visits result in purchases." So, for sample calculations we used a ratio of 50 visitors for every one order. (The authors of the report warn that these figures are averages across all retail categories and caution that category-specific data should be used when actually making decisions for your business.)
As we stated earlier, we are most concerned about the performance of the search engine. For the data describing customer use of search, we made the following broad assumptions and guesses that, in a real situation, we would replace with well-researched figures:
Because each search type puts a different strain on the system, we had to determine performance goals for each one. To start, we convert our business goal into the minimum number of orders our system will need to handle per second. Then we determine how many searches per second the system will need to handle to generate that many orders. (Keep in mind that only a small percentage of searches will result in orders.) From this figure, we can determine how many requests the system will need to handle, which we can break down by request type.
To calculate the minimum orders per second, we begin with the business goal of six million dollars gross profit per year). Divide this figure by profit per order to convert to orders per year. We can then divide the figure by the number of seconds in a year (31,500,000) to get a figure that will express minimum orders per second:
$6,000,000 1 order 1 year
-------- × ------------ × -------------- = 0.127 orders per sec.
1 year $1.50 profit 31,500,000 sec.
This figure can also be expressed as one order every 7.9 seconds.
To calculate the minimum number of searches per second, we can use the result of our first calculation. Multiply the orders per second ratio by the visitors to orders ratio to produce a ratio of visitors per second. The visitors per second ratio can be converted to searches per second by dividing by (multiplying by the inverse of) the visitors to searches ratio:
0.127 orders 50 visitors 2 searches
------------ × ----------- × ---------- = 12.7 searches per second
1 second 1 order 1 visitor
Bridge engineers will be quick to notice that we used averages in these example calculations. Just as a bridge engineer should account for the effect of high wind speed and traffic congestion, we need to plan for peak activity. Market research should be used to determine the deviation from average that can be expected for peak activity in the appropriate retail category.
The Microsoft Web Stress tool is a configurable testing tool that we have adopted for measuring Web performance in terms of requests per second. So, to use this tool we must convert our data from orders per second to requests per second.
To calculate the requests per second ratio we had to account for every request that would be made to the Internet server. Based on the market research that only one out of 50 visits will result in a sale and our assumptions about search behavior, we made the following performance calculations.
| Search type | Number of the 50 visitors using each search type | Get/post requests per search type | Total requests per search type |
| Category | 30 | 4 | 120 |
| Keyword | 15 | 1 | 15 |
| Natural Language | 5 | 1 | 5 |
Adding up the requests per search type, we get a total of 140 requests. However, each of the 50 searches will theoretically result in a request for a details page, which means there will be an additional 50 requests. There is also one request involved with the purchase.
Therefore, for each order our system must handle 191 requests. Using our goal of 0.127 orders per second, our Internet server must handle 24.2 requests per second.
We can now break down the number of requests by search type. For example, it is simple to determine the performance goal for minimum of keyword requests per second :
0.127 orders per second x 15 keyword requests = 1.91 keyword requests per secondWe can also derive the performance goals for the other request types:
To see how Duwamish Books performed against these goals, read "Performance Testing a Scalable Application."
Here we have identified the performance goals for a particular business scenario. The numbers reflect the specific information we gathered and the conversion ratios we calculated for these performance goal formulas. For other business scenarios, the numbers may change, but the process remains the same. You can easily substitute the figures that we started with to describe the needs of another business goal, producing entirely different performance goals.
A reliable performance goal for a system is an important tool to guide the development, testing, and installation of a data-driven system. Much of the work in calculating a performance goal is gathering reliable information about the system and the way that it will be used. This includes market research to discover customer behavior and trends and analysis of the way that data is stored and retrieved from the system in critical performance areas.
Comments? We welcome your feedback at duwamish@microsoft.com.