Microsoft Site Server 3.0 Personalization Components Capacity and Performance Analysis

April 1999

Microsoft Corporation

Definition of Terms

Chapter 1 Overview

Analyzing the Personalization Components

Personalization features allow content providers to offer different content automatically to different users, matching stored information about available content to user needs and preferences.

For sites that require personalized Web pages, Personalization & Membership (P&M) provides Rule Builder and a set of Design-time controls (DTCs) for rapid content development with the Microsoft® FrontPage® 98 Web site creation and management tool or the Microsoft® Visual InterDev™ Web development system. For sites requiring heavy customization, the Active Data Object (ADO) and the Active User Object (AUO) are available, along with other system objects for use by software developers.

Projecting Total Resource Cost

A model is designed in which total resource cost is calculated for a variable number of concurrent users. Essentially:

C = n × K

When C is resource cost, n = number of concurrent users, and K is the sum of the costs of each user operation.

Separate equations are created for each type of resource (CPU, memory, disk, and network). Thus, it is possible to intelligently predict the maximum number of users a particular hardware configuration can support, or conversely, the hardware resources required for a given number of users.

Verification Testing

For credibility, the accuracy of these calculations was confirmed by comparing the results of the calculations using a series of verification tests. A test script is created that simulates a typical user behavior defined in the User Profile. The predicted resource costs calculated from the model are charted against actual resource costs generated by running the verification script.

System Configuration

The following configurations were used in testing.

LDAP Processor Scaling

1. One Web server was used with different processor settings for processor scaling tests.

Personalization & Membership Description

Introduction to Personalization

Rule Builder Overview

Rules define a relationship between member attributes and content attributes. For example, a rule that specifies that all users with, for example, level 4 in their clearance attribute, should receive news articles from the content database (DB) that have the value high in the privacy attribute. Rules thus allow a site builder to deliver personalized content to each site visitor.

Administrators build rules with an application called the Rule Builder. The Rule Builder retrieves both the content source schema (the fields in the content source that could be used in a rule) and the member schema from the Personalization &Membership directory service. Rules contain condition, exception, and action clauses that map on to the expression If X AND NOT Y then DO Z. The administrator creates such an expression by adding and modifying expression fragments which encapsulate the behavior of the condition, exception, and action clauses in quasi-English language clauses. Multiple rules are organized into rule sets; rules are then evaluated based on their priority in the rule set. The Rule Builder ultimately emits a rule set in the form of a Microsoft® Visual Basic® development system (VBScript) that can be inserted into an Active Server Page (ASP).

Rule Testing Overview

For the purposes of testing, the default member schema will be augmented with three attributes with the respective types of string, integer, and date. The content source schema will similarly include attributes of type string, integer, and date. All rules will operate on these attributes. Each record in the content source will also include a content field, which contains the string to be output to the HTML stream that the client sees when triggered by a rule.

The directory service will be a Microsoft® SQL Server™ database containing 200,000 users. The SQL content source database will contain 5,000 records. The content field in each database will contain strings between 1 and 500 characters in length. The Microsoft® Site Server Search content source will be an index of approximately 200 MB worth of .html files.

Tested Personalization Transactions

User Profile

The following user profile was created to simulate a user visiting a personalized Web site. Eighty percent of the pages are lightly personalized, in this case only using their whole name in appropriate description fields. The other 20 percent of the time the user goes to a page where the linked content is displayed in some way, based on the known user information. In this scenario, it is randomly selected between the six different classifications. The user is active on the site for 10 minutes on average, and retrieves five pages. There are no additional objects on the pages.

This user profile generates this transaction frequency table.

Summary of Scalability and Performance

Based on the data collected in this document, the following assertions can be made about scaling and performance:

• The number of elements returned in the content query affects the overall cost of the system more than the complexity of the rule.
  
  
• The authentication service caching mechanism saves additional costs to the LDAP server and the Microsoft SQL Server by storing the entire user object on the Web server.
  
  
• Properly configuring and tuning the content source is important in performance and scalability.
  
  
• The AUO itself scales well to multiple processors.
  
  
• Revisiting the same Web server in a multi-server environment takes maximum advantage of the AUO cache.
  
  
• The graph does not enter into the context switching realm on any of the servers.

Personalization Service Scaling

Every user who accesses the Web server increases the hardware utilization on that Web server. This chart shows the expected number of concurrent users based on the processor utilization calculations derived from the previous table. This graph stops at the practical maximum limit (10,000) of Microsoft® Windows NT® Server’s built-in Web server, Internet Information Server (IIS) connections.

Chapter 2 Detail Discussion of Scalability and Performance

Processor Usage

The PPEM costs of the transactions for the personalization service are not all constants. They vary with the rate of the transaction. When the transaction rate is low, the costs tend to be higher, and likewise when the transaction rate is higher, the costs of some of the transactions rise. This table represents the least squares approximation of the PPEM versus Transaction Rate that were measured in our test environment. The relative accuracy of these measurements is confirmed in the following verification section.

The PPEM at a given rate on a given processor: PPEM = r x (C₁ + C₂r + C₃r²)

An example of PPEM calculation is given in Appendix B.

Note   Simple AUO scales to multiple processors. The underlying search mechanisms do not scale as well. Also note that the ODBC scaling will improve with the Microsoft Data Access Components (MDAC) version 2.0 update.

Memory Usage

Each concurrent connection uses approximately 24 KB on the Web server. Thus, 500 users x 24 KB is 11.7 MB. There is an additional overhead associated with running the AUO code, the Site Server Search code, and the ODBC code. However, this is a static cost of less than 3 MB for each of these services. Hence, it should not radically affect the memory cost of configuring a server.

SQL Usage

The AUO PM transaction does a simple base lookup for the user information when it authenticates. The information is then stored in the Authentication Service cache and when AUO requests the user name; it does not have to go back to SQL Server.

The Site Server Search tests actually use the Search service for its queries. The search catalog is on the Web server, so there is no Microsoft SQL Server query.

The ODBC request is resolved by an SQL query. These costs are measured and represent some level of caching. The results vary with the complexity of the query and the arrangement of the SQL tables of the content data. The matching user property for determining content personalization does not need to be read from the database because it was in the Authentication Service cache. The broker cache contains the entire user object.

SQL Server Resource Utilization per Transaction Type

2. Site Server Search is a separate search engine.

Resource Usage Calculations

Resource usage calculations are created to answer the question, “What is the cost to support a given number of users?” And given this information, the next question is, “What is the maximum number of users this configuration of resources can support?”

In order to determine capacity for each resource, equations can be designed that measure resource cost against number of users.

Profile Calculations

Before we look at specific components of the CPU, disk, and network calculations, we will examine the user profile. Using the number of concurrent users to determine maximum capacity is difficult if we do not take into consideration user behavior.

T= n x f

Using the Site Server Search 1 example, the equation becomes:

T = n x .000333

If we wanted to find out how many Site Server Search 1 transactions are generated per second for 5,000 concurrent users per connection, the equation becomes:

T = 5,000 x .000333 = 1.6667

The following is an expansion of the entire table.

Processor Calculations

Calculate the total CPU cost at 5,000 users for all seven transactions in the user profile. Suppose we use the Site Server Search 1 transaction again and arbitrarily choose the 4-processor numbers.

PPEM_NL1 = Rate (PPEM C₁ + Rate x PPEM C₂)

PPEM_NL1 = 1.67 (5.27 + 1.67 x 0) = 6.25 PPEM

The following is an expansion of the entire table for a 4-processor.

The following is an expansion of the entire table for a 2-processor.

The following is an expansion of the entire table for a 1-processor.

Note   The projected CPU load bounds the single processor system at 3500 maximum concurrent users.

SQL Server Disk Calculations

The SQL Server input/output (I/O) for AUO membership transactions assumes no caching. The SQL Server I/Os for the ODBC calls are measured using the generated database. Actual results may vary.

In order to calculate the rate for any number of users, multiply the frequency of each transaction by the number of I/Os for each call.

At 500 users:

Extract the frequency from the table. AUO Membership is 3.33 per second. Each AUO generates 2 I/Os per second from the previous table for 6.66 I/Os.

The frequency of the Site Server Search pages are all zero because the requests do not use SQL Server.

Frequency_{ODBC 1} 0.1667 x SQL I/O _{ODBC 1} 8 = 1.33 I/O/s

Frequency_{ODBC 10} 0.1667 x SQL I/O _{ODBC 1} 43 = 7.17 I/O/s

Frequency_{ODBC 100} 0.1667 x SQL I/O _{ODBC 1} 250 = 41.67 I/O/s

Add all the elements together for a total of 56.83 I/O per second.

This table is the expansion of this equation for all users.

To support 10,000 concurrent users, the SQL Server system must support 1136.67 SQL Server I/O transactions per second on the primary data device. In this scenario it is all read traffic. This is a worst-case scenario where there is no data overlap. Furthermore, it is assuming 0 percent SQL Server caching, which is highly unlikely. The steps to calculate the RAID array to support this are as follows.

Disks for:

RAID Level 0 =ceiling ((Read_Max/sec + Write_Max /sec)/60_{average maximum disk I/O speed})

RAID Level 1 = ceiling (Read_Max/sec + (2 x Write_Max /sec))/60_{average maximum disk I/O speed})

RAID Level 5 = ceiling ((Read_Max+ (4*Write_Max /sec))/60_{average maximum disk I/O speed})

The denominator of 60 assumes a 4.3-GB disk drive in accordance with the Compaq performance documentation. Faster disk drivers and a newer controller yield different performance.

RAID Level 0 = ceiling (1137 + 0) / 60 = ceiling(18.95) = 19

RAID Level 1 = ceiling (1137 + 2 x 0) / 60 = ceiling(18.95) = 19

RAID Level 5 = ceiling (1137 + 4 x 0) / 60 = ceiling(18.95) = 19

Sample Configuration

Site Configuration Example

This example, based on the same user profile used throughout this document, illustrates how to determine an optimal configuration for a given number of users.

Profile Description

The Web site has 200,000 total users. On average, each user connects for 10 minutes and visits five Web pages. There are four light personalized pages. One page has content associated with it. All pages are authenticated with Membership Directory Basic Authentication.

Two-percent of total users are online simultaneously at peak time, with 4,000 concurrent users.

Total light personalized pages per second during peak time is 4000_Users x .006667_rate = 26.67 per sec.

The last page is randomly chosen in an equal distribution between the remaining six content generation Web pages (for example, the ODBC 10 element page is hit 4000_users x .000333_rate = 1.33 per sec).

Profile Calculations

The number of requests per second is given by

N x F

The number of concurrent users N is computed as above and the frequency F is the number of queries per user per second.

This table of rates is generated for all transactions.

Sample Processor Calculations

For each of the transactions, compute the expected CPU cost. The PPEM model was developed by assuming the cost of running the individual .asp. The model assumes that the total cost of running all the ASPs together will not bring the system into the area of context switching. Context switching on multi-processor systems running .asp files is expensive at higher transaction rates.

It is important to note that the PPEM costs are only accurate to predict approximately 80 percent utilization on a processor. Above these levels, running a computer with greater CPU capacity is recommended. (The best choice is to increase the clock speed.) The other solution is to add another server to distribute the load.

Furthermore each of the PPEM equations is only valid up to the maximum range measured.

In order to calculate the cost for the simple AUO measurement for a 4-processor:

PPEM_{AUO simple} = Rate (PPEM C₁ + Rate x PPEM C₂)

26.67 x (2.71_PPEM + 26.67*0) = 72.27 PPEM

for the remaining 6

1.33 x (3.75_PPEM + 1.33*0) = 5.00 PPEM

1.33 x (4.6_PPEM + 1.33*0) = 6.13 PPEM

1.33 x (21_PPEM + 1.33*0) = 28.00 PPEM

1.33 x (8.4362 - 0.2865 x 1.33 + 0.0088*1.33² ) = 10.76 PPEM

1.33 x (7.7212 + 0.1341 x 1.33) = 10.53 PPEM

1.33 x (31.0121 + 0.234 x 1.33) = 41.77 PPEM

For a total of 174.46 PPEM or 21.81 percent CPU utilization on a 4-processor.

Appendix A:  Testing Methodology

Calculating the Per Transaction Cost

To calculate the PPEM from observed data, the load was simulated on a per transaction basis. This was performed using Site Server Search single-return-valued scripts. An automated script was run that cycles through a changing user load. The values were captured in a perfmon.exe log file. This log file was then exported to a tab-delimited text file and imported into Microsoft® Excel. Then the log was distributed through an Excel macro, and averaged. Each row represents several minutes of the transaction load. The authentication counter is observed in order to confirm that the entire process is working.

To establish a valid range of samples, we observe context switches. The context switch rate must be below a certain threshold to be considered running optimally. However, the processor should be exercised across a full range.

The total processor time must be multiplied by the clock capacity of the system, which in this case is 4 x 200, and then divided through by the number of requests/sec. This generates the PPEM values at different transaction rates.

The overall PPEM is then calculated by taking a polynomial fitting using a least squares approximation of the PPEM value at the transaction rate. This generates the formulas presented in Chapter 1 for each transaction.

In this case, the polynomial equation is PPEM over rate = 8.4362 - 0.2865 x T + 0.0088 x T².

Verifying Actual versus Calculated

2-Processor Verification

4-Processor Verification Table

Appendix B:  Scaling Summary

Table of Maximum Rates

3.The Site Server Search subsystem was assigned 32 threads. This number of running threads was the bottleneck for throughput.

4.The SQL Server computer that supported the ODBC queries was configured to have additional open connections to support the queries.

5.The disk subsystem that supported the content database was the bottleneck. The maximum throughput for the system will vary greatly based on the content database used.

Appendix C:  Critical Monitoring Counters

All counters noted can be found in the Microsoft® Windows NT® Performance Monitor. These counters will be distributed among the computers in the Personalization & Membership service group. The counters in the system and memory objects can be used to monitor capacity.

Site Server LDAP Service

Connection Current

Static Search per sec

Site Server Authentication Service

Automatic Cookie Auth Successes per sec

ClearText Auth successes per sec

DPA Auth successes per sec

HTML Forms Auth successes per sec

Site Server Search Catalogs

Results rate

SQL Server

SQL I/O transactions per sec

System/% Total Processor Time

Physical Disk

Disk Writes/secDisk activity should not sustain maximum transaction rate

Disk Reads/sec

ASP Object

Requests/sec

Current requests

Request Execution Time

Web Object

NonAnonymous Users/sec

Get requests/sec

System Object

Context switches/secshould be less than 15,000 on a 4-processor system

%Total Processor  should be less than 80%

Processor Object

%Processor Utilization (average)should be less than 80% (for each processor)

Memory Object

Available Bytesshould be greater than 4 MB

Appendix D:  Scripts

This is called by the InetMon script

AuoPure.asp

<%
set user = server.createobject("membership.userobjects")
response.write("the full name is " & user.get("cn") & "<br>")
%>

Get_auo.txt

USER tstRANDNUMBER(1,10000) password
GET url:/auo_pure.asp
SLEEP RANDNUMBER(300,800)
LOOP 9
GET url:/auo_pure.asp
SLEEP RANDNUMBER(300,800)
ENDLOOP
DISCONNECT

The .asp files and the .prf files do most of the work, but they were all generated using the Rule Builder.

Verification script

USER tstRANDNUMBER(1,10000) password
LOOP 4
GET url:/auo_pure2.asp
SLEEP 24000
ENDLOOP
%84 SKIP 2
GET url:/u2perfproj/rp1odbc.asp
SKIP 13
%80 SKIP 2
GET url:/u2perfproj/rp2odbc.asp
SKIP 10
%75 SKIP 2
GET url:/u2perfproj/rp3odbc.asp
SKIP 7
%67 SKIP 2
GET url:/u2perfproj/rp1nl.asp
SKIP 4
%50 SKIP 2
GET url:/u2perfproj/rp2nl.asp
SKIP 1
GET url:/u2perfproj/rp3nl.asp
SLEEP 24000
DISCONNECT

Information in this document, including URL and other Internet web site references, is subject to change without notice.  The entire risk of the use or the results of the use of this resource kit remains with the user.  This resource kit is not supported and is provided as is without warranty of any kind, either express or implied.  The example companies, organizations, products, people and events depicted herein are fictitious.  No association with any real company, organization, product, person or event is intended or should be inferred.  Complying with all applicable copyright laws is the responsibility of the user.  Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document.  Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

© 1999-2000 Microsoft Corporation.  All rights reserved.

Microsoft, FrontPage, Visual Basic, Visual InterDev, and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the U.S.A. and/or other countries/regions.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners.