Business Systems Division
Microsoft Corporation
March 1996
The following paper analyzes the performance of the Microsoft® Internet Information Server in comparison to other Microsoft Windows® NT®, UNIX®, and Novell® NetWare®-based Web servers. This analysis covers both performance of static HTML page serving, and dynamic Web page serving via the Common Gateway Interface (CGI) and the Microsoft Win32® extension for its Web server called the Internet Server API (ISAPI). The Internet Information Server, now an integrated part of Windows NT Server, provides Web-site managers with a Web server that's easy to install and manage and provides both high performance and a platform for a new generation of Web applications. Microsoft Internet Information Server easily outperforms other PC Web servers by as much as 400 percent and even outperforms more expensive UNIX/RISC-based Web servers.
Note The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, this document should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
The key aspect in which the Microsoft® Internet Information Server (IIS) most dramatically differs from its competitors is performance. In addition, IIS is the only Web server integrated with the Microsoft Windows® NT® Server operating system, and thus IIS derives a wide array of benefits for its users. Installations running IIS find its ease of use, scalability, portability, security, and extensibility superior to any of the alternatives. To demonstrate the overwhelming performance advantages of IIS, a series of benchmark tests were run by independent testing laboratories. The results are compelling and absolute: no matter how the tests were run—using different workloads, a variety of competitive Web servers, multiple operating systems (including UNIX) and even running some tests on expensive, proprietary hardware platforms—IIS delivers up to 500 percent of the performance of all of the systems tested.
The performance of a Web server is critical. Rapidly increasing communications speed, the use of multipurpose servers, the delivery of dynamically generated information, and the explosion of internal Webs ("intranets") combine to make the need for high-performance Web servers mandatory. Only IIS delivers the absolute performance that today's sites require, on a platform that is easy to install and manage, scalable, and secure and extensible by using a similarly high-performance application programming interface (API) called the Internet Server API (ISAPI).
Please visit http://www.microsoft.com/intdev/ for more information on developing Internet applications using Microsoft tools and platforms.
This document is designed to give a performance comparison between the Microsoft Internet Information Server and other Windows NT, Novell® NetWare®, and UNIX®-based Web servers. Three general comparisons were made; two were new benchmarks run by independent laboratories and one is a comparison of publicly available performance information from a variety of hardware and software vendors. The two Web server performance benchmark tools that were used are the National Software Testing Laboratories, Inc. (NSTL) Web benchmark tests and WebStone version 1.1 from Silicon Graphics, Inc. (SGI). The NSTL tests were run by the NSTL at its offices, and the WebStone tests were run by Shiloh Consulting and Haynes & Company at the offices of Shiloh Consulting. The sources for the cross-platform comparisons are WebStone results published by different hardware and software vendors.
Because there is not a standard metric for publishing Web server performance, the graphs in this document depict different performance characteristics of Web servers that the benchmark tools measure. Note that some of the tests are configurable—whenever appropriate, the configuration of the test run is supplied for reference. In all instances, the conclusion is consistent: the Microsoft Internet Information Server running on Windows NT Server delivers the highest performance available.
Until now, Web servers did not focus on performance as an intrinsic design premise. This was acceptable during the early adopter phase of the Web, as most Web sites serviced a limited load over low-capacity links and less demand was placed on the servers. Today, the feature set for Web servers has stabilized, commercial implementations are available, and the importance of Web server performance is increasing. Several reasons that contribute to the increasing focus on performance include:
The WebStone version 1.1 benchmark was used to generate the workload and measure the performance of the Web server. All tests were conducted by Shiloh Consulting at their facility. For complete details on the WebStone test configuration, please refer to the report from Haynes & Company and Shiloh Consulting "Performance Benchmark Tests of Microsoft, Netscape, and Novell Web Servers," which can be found at http://www.microsoft.com/infoserv/.
Shiloh tested various Web servers, including:
For these WebStone tests, Shiloh used a Hewlett-Packard® NetServer LS server, with an Intel® Pentium® 133MHz CPU, 32 megabytes of RAM, two 1-gigabyte hard drives, and a DEC™ 10/100 megabit Ethernet card. The one exception is that for the BSDI UNIX Netscape™ Communications Server tests, 128 megabytes of RAM were used. When only 32 megabytes of RAM were installed with BSDI UNIX, the server repeatedly failed to complete the tests and the WebStone tests could not be run to their conclusion. Even with 128 megabytes of RAM, the WebStone tests were not able to complete the tests at the 128 client load without the server crashing. The other Web server tests on this platform were run with 32 megabytes of RAM.
A wide variety of characteristics determines the overall performance of computing systems. Web servers are no different in that respect. The tests that assess the relative performance characteristics of IIS and contrast them with the same attributes of competitive products, measure the widest possible array of relevant statistics. In these tests, you will find analyses of the following:
For details on the results, see the complete Haynes/Shiloh report at http://www.microsoft.com/infoserv/.
Throughput measures the maximum rate at which the Web server transfers data to its clients. Throughput is reported as megabits per second. For reference purposes, the capacity of an ISDN line is 64 kilobits per second (Kbps) (or 128 Kbps if two channels are used), a T1 line is 1.54 megabits per second, and a standard Ethernet has a capacity of 10 megabits per second. Many local area networks now take advantage of new technology, such as concentrators, hubs, switches, or FastEthernet media, which is capable of delivering data on the order of 10 to 100 megabits per second.
Figure 1. Throughput
As the graph clearly shows in Figure 1, the Microsoft Internet Information Server is capable of delivering as much as four times more data to clients than the Netscape Communications Server version 1.12 on Windows NT Server, and almost three times more data than Netscape on a BSDI UNIX. When compared to Novell's Web server NLM, IIS is more than 50 percent faster.
The capacity for high performance—that is unique to the combination of Windows NT Server and IIS—is mandatory for today's Web sites. The additional capacity and capability extending from a high-performance architecture includes:
Connections per second represents the sum of successful interactions across all the clients. An interaction consists of creating a TCP/IP connection between the client and the server; sending a request from the client to the server over this connection; receiving the correct data back from the request; and closing the TCP/IP connection. A higher number indicates better performance and therefore better client service. Note that this test was not run using Connection Keep-Alives, which is a new HTTP feature. Connection Keep-Alives are fully implemented in IIS; IIS could go faster using them, but to keep the tests fair they were not used, because Keep-Alives have not been implemented in all browsers or servers.
Figure 2. Connections per second
Another way to look at this is Web transactions per second: how many transactions (accepting the request from the client and sending back the requested data) a server can drive per second. The results in Figure 2 clearly demonstrate that IIS can handle as many as four times more transactions than the Netscape Communications Server on Windows NT Server, three times more transactions than Netscape on BSDI UNIX, and easily handle 50 percent more transactions than Novell's NLM Web server.
In real-world terms, IIS would be able to support four times the number of user requests than the Netscape Web server, thus allowing Web-site managers to support a greater number of users on similar hardware. It also means that users will experience a quicker response time. This advantage also manifests itself by allowing the site administrator to better prepare for peak bursts of user requests.
Average response time measures the amount of time required to complete an operation once it is started. The WebStone benchmark measures both connection response time and transfer response time. Connection response time measures the time taken to establish a connection. Transfer response time measures the time to complete a data transfer once a connection has been established. User-perceived response time includes the sum of the connection plus transfer response times, plus any network latency due to WAN/LAN connections, routers, modems, and so forth.
Figure 3. Average response time for client load
For this set of results, a lower number is better. Results in Figure 3 show that the measured average response time for IIS is significantly lower than that of the Netscape Communications Server. Although the Netscape server did not demonstrate any client errors, this graph indicates that it was forced to queue client requests, thereby imposing a frustrating delay on the end user to complete a request. This delay increased steadily as the number of client requests increased. Note that the average response time results for the 128 client load tests for the Netscape Communications Server for BSDI UNIX are not part of this graph because the server was unable to complete the test at this load.
This graph is important because it also clearly demonstrates that the Netscape Communications Server, on both Windows NT and BSDI UNIX, imposes a delay on users even when the Web server is not heavily burdened. Looking at the average response time for the 16 client load tests, you can see that Netscape servers can take up to seven times longer to successfully respond to a client request. This clearly demonstrates that even for Web servers being used by a relatively small number of users, IIS is clearly the better solution.
Figure 4. Average response time for each file
Keeping in mind that the typical Web page is made up of several files—usually an HTML file, and several GIF/JPEG or other graphics files—the average response time results in Figure 4 becomes even more important—because this is the response time for each file on the Web page. So, if a complex Web page with a lot of text and graphics files is accessed on IIS, the page will appear extremely quickly, while a user accessing the same page from a Netscape server will be forced to stare at the hourglass for several seconds.
Many Web sites are moving beyond the delivery of static HTML files. To deliver engaging and high-impact content, Web sites need to be dynamic and have the ability to generate pages that are filled with information targeted specifically at each user (as an example, see http://www.msn.com/ and the custom home page capabilities). In addition, customers and independent software vendors seek ways to improve site and content management, to use the Web as an extension to their own network, and to deliver information technology founded on the Web as a platform. To deliver dynamic interaction and value-add extensions, Web servers must offer a high-performance, easy-to-use programming model.
The method for delivering this new level of functionality is to extend the Web server itself programmatically. There are a variety of techniques available in today's Web servers: The most common approach is called the Common Gateway Interface (CGI). CGI, which is supported by nearly all Web server implementations, is one of the simplest and most straightforward ways of extending the server. Netscape uses a proprietary method called NSAPI, that is available on only a limited number of Netscape's Web server implementations. (For example, Netscape Communications Server for BSDI UNIX did not include support for NSAPI at time of this publication.) Microsoft and other Web server vendors offer an open method called the Internet Server Applications Programming Interface (ISAPI). The performance of a Web server extension API becomes an important factor in servicing high-impact content to the broadest possible user base.
The first data that needs to be examined is a comparison of the performance of a CGI application to an equivalent ISAPI application running on IIS. This test is configured to generate 100 percent CGI or ISAPI pages, with no static HTML pages. This will give a good baseline for the performance that users will see when using ISAPI applications rather than CGI applications on the IIS platform.
The chart below compares WebStone results, at 64 clients, for 100 percent CGI or ISAPI, 0 percent HTML configuration. The charts show connections per second and throughput in megabits per second.
Figure 5. ISAPI vs. CGI
From these charts, it is easy to see that in a direct comparison, ISAPI is close to five times faster than CGI at performing the same tasks on Microsoft Internet Information Servers.
For Web-site managers, this means that by adapting CGI-based applications to the ISAPI interface, they will immediately see a decrease in the time that users have to wait to have their requests for data filled by the server, and the server will be capable of handling more requests. Moving CGI applications to ISAPI applications will also allow the Web-site manager to install more Web applications on a single server, thereby improving the impact of the site without adding additional server resources.
Once it was determined that ISAPI was considerably faster than CGI, the question remained: Is ISAPI faster than Netscape's proprietary Web server API (NSAPI) at the same tasks? To determine this, Shiloh ran the WebStone tests (with 64 clients, configured for 100 percent API, 0 percent HTML page generation) against IIS using an ISAPI port of the API test and against the Netscape Communications Server for Windows NT running the NSAPI port of the API test. Note that this test could not be used to compare NSAPI on BSDI UNIX—Netscape does not currently support NSAPI on BSDI UNIX. Also, it should be noted that Novell's NetWare Web Server was not included in the CGI or API testing because Novell does not currently support this functionality. Novell's Web server does support a proprietary extension called Remote CGI (R-CGI) that allows a CGI application to be run on a server separate from the Web server (which means that for the Novell solution, a Web site would need one Web server and one CGI server to match the capabilities of a single Web server, either from Microsoft or the other Web vendors). Because this is not the standard way Web sites are configured, Novell's Web server was not included in this portion of the testing.
Figure 6. IIS ISAPI vs. Netscape ISAPI
The results, in both throughput per second and connections per second, clearly show that for performing the same tasks, ISAPI is considerably faster than NSAPI—almost three times as fast. Once again, this indicates that users would receive a quicker response from their database query or other Web application when the application is written in ISAPI rather than using NSAPI. This also means that as more and more dynamic Web applications are added to a server, IIS can handle more applications and users on a single server, without degrading the performance as perceived by the users.
While the tests described above demonstrate the overwhelming performance characteristics of IIS, these evaluations all hold the tested hardware platform constant. The vendors of expensive UNIX systems sometimes claim performance as their unique value-add. To learn whether or not these claims can be sustained with a Web server as the application, a variety of publicly available performance information has been compiled and appears in the chart below. This chart uses the results of WebStone data running on IIS on low-cost PC servers, contrasted to the competition. Vendors who have published WebStone 1.1 results include HP (HP 9000 Group) (see results at http://hpcc998.external.hp.com/csopress/ and Spyglass http://www.spyglass.com/products/).
Figure 7. Connections per second
The results are telling: Windows NT Server and IIS, running on a Hewlett-Packard®, Intel Pentium-powered Windows NT Server, performed two to four times faster than any of the other Windows NT or UNIX Web servers—including the "high-performance" UNIX systems. In addition, the Microsoft Internet Information Server significantly outperformed Novell's NLM-based Web server.
For the Web site manager, the Information Systems or department manager implementing an intranet Web server, this means that they can use the lower-cost, easier-to-set-up-and-manage Windows NT Server-based IIS to provide a significantly faster response time to their customers. Combined with the performance data on ISAPI applications, it is clear that the best Web server, for standard HTML publishing or hosting Web-based applications for both the Internet and the intranet, is Windows NT Server and the Microsoft Internet Information Server.
Below are descriptions of the Web servers tested in the chart above:
| Descriptions of Web Servers Tested for Connections per Second | ||
| Microsoft Internet Information Server Windows NT Server v3.51
Microsoft Internet Information Server, HP® NetServer LS (133-MHz Pentium) 1 MB L2 Cache, 32 MB RAM 2 x 1 Gigabyte Hard Drives (Tested by Haynes/Shiloh) |
Netscape Communications Server for Windows NT Windows NT Server v3.51 Netscape Communications Server v1.12 HP NetServer LS (133-MHz Pentium) RAM Cache Hard Disk (Tested by Haynes/Shiloh) |
|
| HP/UX Netscape HP/UX Netscape Communications Server HP 9000 model HP D200 (Tested by HP) |
Sun® Netra Netscape Solaris Netscape Communications Server Netra SPARC™ server Model i600 (Tested by HP) |
|
| Sun Brand X SPARCserver-20, Solaris 2.5 Unknown Web server (Tested by Spyglass) |
Sun NCSA SPARCserver-20, Solaris 2.5 NCSA Web server (Tested by Spyglass) |
|
| Sun CERN SPARC Server-20, Solaris 2.5 CERN Web server (Tested by Spyglass) |
Sun Apache SPARC Server-20, Solaris 2.5 Apache Web server (Tested by Spyglass) |
|
The Spyglass numbers used above were for test results for 60 clients, where as the HP and Microsoft tests were conducted for 64 clients. We believe that this difference in clients between tests does not diminish the comparative value.
It should be noted that the performance numbers for the Spyglass™ Web server were not included in Figure 7 because the Spyglass performance numbers were obtained using Connection Keep-Alives, which significantly improves the results of benchmarks such as WebStone's. Spyglass's benchmark results with Keep-Alives were good (over 16 megabits per second throughput and almost 300 connections/second at 60 clients), but they should be compared with results from benchmark tests where all servers use Keep-Alives. IIS also includes support for Keep-Alives, and IIS could go faster using them; but in fairness, the benchmark tests were run without Keep-Alives because they are not implemented in all browsers and servers. Microsoft plans to publish benchmark results for IIS with Keep-Alives at a future date, when a model has been developed to use it appropriately for benchmarking.
The National Software Testing Laboratory (NSTL) is a well-known software laboratory that has long been involved in testing and benchmarking PC and LAN software. NSTL uses an internally developed Web server benchmarking tool to test various Web servers, including:
The NSTL benchmark tested the ability of the Web, or HTTP server, to transfer HTML files. The results are expressed in transactions-per-minute score. A completed transaction in the NSTL benchmark consists of the client making a request for data from the server and then receiving the correct data. This test measures how many of these transactions a server can complete per minute.
The results from the NSTL reinforce the results that Haynes and Shiloh obtained using the WebStone tests. IIS is the fastest Web server by a large margin. In the NSTL tests, IIS was over three times faster than the Windows NT-based servers from Netscape, O'Reilly, and Process software, and the BSDI UNIX version of the NCSA Web server. IIS was twice as fast as the Netscape server for BSDI UNIX, and bettered by one-and-a-half times the performance of the new NetWare Web Server from Novell.
This performance translates into substantial benefits for Web site managers that install Microsoft Internet Information Servers:
Figure 8. Transactions per minute
Complete results for the NSTL tests will be published by NSTL in the near future.
The results of the WebStone 1.1 and the NSTL Web performance tests are very clear. When comparing Windows NT Web servers, IIS is up to four times faster than competitive Web server products, including the Web servers from Netscape, Process software, O'Reilly & Associates, and Novell. In comparison to more expensive UNIX solutions, the Microsoft Internet Information Server is the clear performance leader.
While it is not possible to draw direct comparisons between the numbers from these tests and the specific number of "hits" or users that can be supported on a single Internet or intranet Web server; it is possible to draw the conclusion that Web sites that use IIS and develop applications using the ISAPI will:
Therefore, better customer service will be provided than by sites that use other Web server products.
For sites presenting information that is generated dynamically or that wish to use the Web server as an application platform, this performance analysis demonstrates that the combination of Windows NT Server, IIS, and ISAPI is the best solution for developing and deploying Web applications and Web server extensions.
It is also clear that the myth that the fastest Web servers are UNIX/RISC Web servers is a fallacy. The Microsoft Internet Information Server, on low-cost PC Server hardware, is substantially faster than the much higher priced Sun Netra and HP 9000 Web server solutions. Organizations considering the installation of Web servers for virtually any application will find their requirements optimally addressed by IIS, the only Web server integrated with Windows NT Server.
The CGI and API tests are part of the WebStone tests: By means of CGI and API interfaces, WebStone requests a small program be run on the Web server to generate a random string of characters that is sent back to the client. Typically, a CGI program is an *.EXE file and an API program is a *.DLL file. The file is loaded when the Web server starts up under Windows NT. Based on the WebStone 1.1 specification, the operation of the test program (either CGI or API) is identical for all Web servers; although the actual code will differ between ISAPI and NSAPI, and between a Windows NT platform and a UNIX platform.
The WebStone version 1.1 benchmark was used to generate the workload and measure the performance of the Web server. The WebStone benchmark was developed by Silicon Graphics Incorporated (SGI) as a tool for measuring Web server performance. SGI makes WebStone freely available in source form to whomever wants to use it. For more information regarding this benchmark, see: http://www.sgi.com/Products/WebFORCE/WebStone/.
Microsoft recognizes that the SGI WebStone performance tests may not be the best solution for comparing performance of Web servers, but in the absence of a single, widely accepted Web benchmarking standard, Microsoft used WebStone for the comparative analyses that customers demand.
The WebStone Tests were run by Shiloh Consulting and Haynes & Company at the facilities of Shiloh Consulting.
Please note that several of the links below point to servers that are not under Microsoft's control. Please read Microsoft's official statement regarding other servers.
For more information and to download the Microsoft Internet Information Server, see http://www.microsoft.com/infoserv/.
For Additional Performance Information, see http://www.microsoft.com/infoserv/haynes1.htm and http://www.microsoft.com/infoserv/.
For Internet Developer Information, see http://www.microsoft.com/intdev/.
For General Microsoft Information, see http://www.microsoft.com.
For the Microsoft Internet Information Server Benchmark Report, see http://www.microsoft.com/infoserv/haynes1.htm.
For the Netscape NSAPI vs. CGI on SGI UNIX Server Report, see http://www.tedhaynes.com/.
For Shiloh Consulting, see http://www.tedhaynes.com/haynes1/shiloh.html.
http://www.sgi.com/Products/WebFORCE/WebStone/
http://www.ncsa.uiuc.edu/InformationServers/Performance/CGI/
http://hpcc998.external.hp.com/csopress/
http://www.spyglass.com/products/
For the latest information on Microsoft Internet Information Server, check out our World Wide Web site at http://www.microsoft.com/infoserv/.
For more information on ISAPI, visit the Internet developers section of the Microsoft Web server at http://www.microsoft/com/intdev/.