This document will demonstrate how to design, set up, and administer a very large user base using the Personalization & Membership (P&M) component of Microsoft® Site Server 3.0. This document is not intended to be an introduction to Site Server, Microsoft® SQL Server™, or Microsoft® Internet Information Server (IIS). Rather, this document outlines many of the issues that are critical for corporations or independent software vendors (ISVs) to consider when planning a large-scale deployment of Site Server.
Before you install any software, read this section and the following section, Planning Administration and Security Strategies. To get the most out of Site Server with a large system, you must plan the system carefully.
In a very large installation with potentially millions of members, thousands of simultaneous users, gigabytes of member information, and thousands of transactions per second, the first step in the planning phase is to design an architecture that is capable of supporting such a load.
Microsoft® Site Server 3.0 addresses the challenge of having a distributed, scaled system using what is known as the three-tiered model. This model is aimed at distributing the load required for the different Site Server components across multiple computers and in different proportions depending on what type and variety of load is anticipated on the installation.
The three-tiered Personalization & Membership (P&M) architecture has three layers of system components:
Each layer of the system can be split into multiple instances across many different computers. At the application tier, the architecture might be split into multiple Web server computers, each connecting to multiple LDAP Service computers, in turn connecting to multiple SQL Server computers. The diagram below depicts an example of this scenario.
In order to support large user loads and to provide system redundancy, you can set up the services that make up the LDAP tier as multiple services tied to the same DNS name. Because all of the Membership Directory’s persistent data is stored in a SQL Server database, you can set up multiple LDAP services to support a single Membership Directory and use any of them to retrieve the same Membership Directory information.
For example, if the expected load requires four LDAP Service instances, each running on a dedicated computer, then each of these computers might be registered in DNS as LDAP.DOMAIN.COM. Then, when an application-tier service attempts to connect to an LDAP Service, the round-robin DNS server will return the IP address of the least-recently-used LDAP Service from the pool of four as the LDAP Service for the application tier to use. In the case of the Site Server Authentication Service (part of the application service tier), the service then maintains that connection to the particular LDAP Service until the Authentication Service is restarted.
If an LDAP Service malfunctions, it is crucial for the DNS entry for the affected computer to be cloaked out of DNS. This way, existing application tier computers will only attempt to connect to the surviving Services. If the LDAP Service fails to respond to Authentication Service requests, the connection will time out and the Authentication Service will attempt to connect again. The DNS server can then provide the address of a live service.
Another multiple-service solution is provided by using a Single Internet Protocol (Single IP) solution (also known as Virtual IP), available from third party vendors. The Single IP solution is a hardware/software component which maps a single IP address across multiple services. The solution monitors the services and removes ailing services from its list. This method provides uninterrupted service, even if there is a system failure on a computer.
The next topic to consider is whether the infrastructure in the existing data center is adequate for or conducive to the proposed system. The data center’s topology, the presence of firewalls, and so on, can impact how Site Server is installed and configured. Major issues to consider include:
There are a variety of parameters to consider in each of the individual tiers. By considering these factors early, it will be possible to deploy the system with greater scaling power and a predisposition toward high performance.
Analyze the type of operation that is expected on the site you are planning, especially the following characteristics:
Based on the outcome of your initial analysis, you can use any of several tools to help you target the hardware class (processor, disk, memory) and number of computers necessary in each tier to support the targeted site profile.
Important As a general rule for all the computers, install only the services that you need in each tier. Try to keep each computer dedicated to only the tasks it is designed for by stopping or uninstalling any unnecessary software and services.
For more information about targeting the computers that will be required to power the site, see the Online Service Deployment Guide in the MCIS 2.0 Resource Kit, version 1.1.
The first task is to get a general idea of the character of the P&M-based application services the Site Server installation will be hosting. The planning process outlined previously should provide some insight into the critical site characteristics. Beyond those, the following sections discuss other areas to consider.
The number and datatype of attributes that each member possesses can have a dramatic effect on the performance of the site and the number of resources it uses. A user in a basic P&M installation, using only the required default attributes, has nine attributes. The site probably will require more, but how many more depends on what the site does. In a very large installation, it is important to reduce the user attributes to only those required and to be precise about the datatype that each attribute will use. This keeps SQL Server disk space usage low, keeps the number of rows in the Object_Attribute database table low, and minimizes the amount of memory that the Authentication Service requires to cache users and all of their attributes.
Consider the impact of ambiguous searches of Membership Directory objects and try to limit them, if feasible. This is especially important if your Membership Directory is partitioned across multiple databases. There are three types of Membership Directory searches:
Important If you construct a sub-tree search query to search a specific container or set of containers, be aware that in Site Server 3.0, such queries are limited to a relatively small number of containers (about 200). A QFE hotfix is available at www.microsoft.com. The Site Server Service Pack 2 contains an update that increases this limit, and additional support will be available in future versions of Site Server. To avoid such limits, you can use sub-tree search queries that search the entire Membership Directory, but are constrained to a particular object class (such as the class Member or MemberOf). Remember that in a partitioned Membership Directory, this technique will result in searches that take more time because the entire Membership Directory will be searched.
The more partitions that the Membership Directory has, the greater the cost for one-level and sub-tree searches, because the search has to span more partitions.
The Authentication Service caches up to 10,000 objects in memory on a temporary basis. Memory usage depends on the size and number of attributes per object. The accurate way to analyze total memory is to add up the datatype sizes of the various attributes on user objects.
The Design-time Controls (DTCs) that ship with Site Server are aimed at assisting the new AUO user in writing Web sites that use Active User Object (AUO) code. For that reason, they are designed for the general case. Using the DTCs in high-volume, production environments can be less efficient than writing well-crafted AUO code by hand. For sites where performance is paramount, consider carefully optimizing AUO code. For information about optimizing AUO code, see the Site Server 3.0 Software Developer’s Kit.
Plan group memberships carefully. Each member-to-group mapping is an object in the Membership Directory, and can affect the size of the Membership Directory.
Under some circumstances, it may be possible to plan your site so that it can take advantage of read-only LDAP Services. For example, if your Web site is split into a member registration site and a member service site, the registration site could use a read/write LDAP Service while the member service site uses an LDAP Service that is read-only. This configuration shifts the read load away from the service that is performing writes. Read-only LDAP Service instances can handle higher transactions per second, on average, because read operations are less resource-intensive.
After consulting the Online Services Deployment Guide, you should know how many LDAP Service computers (LDAP servers) you need for the proposed system. As a rule, the LDAP servers can be the lowest in hardware power of all the computers in your installation. There are several considerations that affect the performance of the LDAP servers, however.
Partitioning is the art of dividing up the objects in the Membership Directory into many different SQL Server databases. For a full explanation of partitioning, see Creating a New Partition in the Membership Directory in the P&M section of the Site Server 3.0 documentation.
Important You must plan and set up partitions before you add any objects to the containers that you will partition.
An in-depth analysis of the benefits and costs of partitioning is not the subject of this paper, but is more comprehensively addressed in the white papers Site Server 3.0 Membership Directory Configuration and Tuning Guidelines and Hosting Multiple User Communities with a Membership Directory. However, when planning how partitions will be used, remember the following guidelines:
For example, if you want to partition on ou=Members, and you expect to have about 750,000 users within 24 months, consider using two value partitions. Your Membership Directory will be stored in three databases: the Membership Directory root database, and two Membership Directory partition databases.
P&M supports replication of Dynamic Directory data between LDAP Services that serve a single Membership Directory. This facility, combined with an AUO provider, is sometimes used to solve the problem of maintaining user session state data across Web servers that may be set up to round robin. For information about this AUO provider, see the white paper Using the Membership Directory and Active User Object (AUO) for Session State Data.
Using the Dynamic Directory to store session data is more practical and much faster than storing this information in the SQL Server database, However, as you might suppose, this use of the Dynamic Directory requires more memory than LDAP Service operations would otherwise require. It also requires more overall resources, such as processor capacity, because the level of user activity is higher. Furthermore, the peak number of concurrent sessions impacts the number of dynamic objects that each LDAP Service must store at any single time. Each LDAP Service computer (LDAP server) can support a configurable maximum number of concurrent objects. If you need to conserve resources, you can configure a lower maximum after you deploy your system.
The load level of each LDAP server that supports the Dynamic Directory will depend on a number of different factors, including:
One of the most critical aspects of the deployment effort is properly planning and configuring the SQL Server computers that will house the Membership Directory partition databases. This section will present the various concepts that are important in planning the SQL Server deployment so that P&M will keep performing well.
Those who are not experienced at using SQL Server should seek the assistance of a qualified SQL Server database administrator or architect to assist in deploying the system according to plan.
For information about configuring and tuning SQL Server, see the topics SQL Server Database Tuning for Performance and Advanced Database Tuning Considerations in the white paper Site Server 3.0 Membership Directory Configuration and Tuning Guidelines. These topics contain exhaustive information about tuning SQL Server. When you are working with large numbers of users, keep the following factors in mind:
As an added fault-tolerance mechanism, you can use a controller that supports hot-pluggable disks. This type of controller allows you to insert a new disk (or replace a bad disk) on the fly, without shutting down SQL Server. Data is automatically restriped across all the disks (Compaq Smart-2, Dell PERC, and HP NetRaid support this). For detailed information about the different RAID options and about tuning the I/O subsystem, see Compaq’s SQL Server 6.5 Performance Tuning white paper, available at http://www.compaq.com.
You can also use multiple network cards to separate the Application server-LDAP Service interactions from the LDAP Service-SQL Server interactions.
An issue that you should consider early in the planning process is the administration model for the installation. For example, who will be in charge of administering which containers? Who will monitor and maintain which computers? In very large installations, there are often multiple administrators, often with varying levels of rights to make administrative changes. In conjunction with your administration model, you will need to create a security model for your system that maps out who has the rights to perform what functions, both at the computer level and at the Membership Directory level.
You can combine several tools to secure your system:
For more information about these and other important security issues, see the section Site Server Security in the Site Server 3.0 documentation, and the section Security in the P&M section of the Site Server 3.0 documentation.
If your users are segregated into community containers within the ou=Members container, it may make sense to have different people administer the different containers using separate accounts, while a senior administrator manages overall schema changes and other DIT-wide operations. Adjust the ACLs on each container to allow the different administrator accounts the rights to add and change objects in the respective containers. However, be aware that the P&M ACL model is quite powerful and sophisticated, so practice manipulating ACLs on a test site before working on your production installation. For information about ACLs, see the topic Managing Access Control in the P&M section of the Site Server 3.0 documentation and the white paper Hosting Multiple User Communities with a Membership Directory.
For detailed instructions, see Installing P&M in the P&M section of the Site Server 3.0 documentation. This documentation includes recommendations for options to use when you install SQL Server.
Remember the following considerations when deploying a large system:
After you install Site Server and create the SQL Server databases, you can create an LDAP Service Membership Server (on the LDAP server) and the Membership Directory. Then you can create any additional Membership Servers that you will need (for details, see Creating Membership Servers in the P&M section of the Site Server 3.0 documentation).
Once the Membership Servers have been configured, you can create the Membership Directory partitions (for details, see Creating a New Partition in the Membership Directory in the P&M section of the Site Server 3.0 documentation).
Once you have installed and configured the software, implement your administration and security plans. For details, see the following topics in the P&M section of the Site Server 3.0 documentation:
Once the site is deployed, there are several things to do to keep it running smoothly on an ongoing basis, particularly in the areas of LDAP Service and SQL Server administration.
For a complete list of LDAP Service performance counters, see Appendix A: Miscellaneous Performance Monitor Counters in the white paper Site Server 3.0 Membership Directory Configuration and Tuning Guidelines.
For information about improving search performance for the Membership Directory, see Appendix D: LDAP Search Optimizations in white paper Site Server 3.0 Membership Directory Configuration and Tuning Guidelines.
Remember that the performance of the LDAP Services is tied to the performance of the underlying databases and to the types of searches that the LDAP Services are being asked to perform. To optimize performance, focus on keeping SQL Server healthy and minimizing expensive searches from applications.
If your LDAP servers are consistently overloaded, you can ease each server’s load by increasing the number of LDAP servers (for criteria, see Add More LDAP Servers in the white paper Site Server 3.0 Membership Directory Configuration and Tuning Guidelines).
If you are using DNS round robin to route LDAP Service clients, simply install P&M and its supporting software on the new computer, add its IP address to DNS under the same name as the rest of the LDAP servers, and create an LDAP Service Membership Server on the new computer. One caveat is that the Authentication Services won’t use the new LDAP Service until they are stopped and restarted, since each Authentication Service is tied to a particular IP address once it is started. If possible, stop and restart all Authentication Services. This way, the new LDAP Service is cycled into the DNS round-robin pool and shares the load with the other LDAP Services efficiently.
In order to keep your SQL Server computers well-honed for high volume-operations, it is important to keep an eye on them. At the very least, the following scheduled operational tasks are recommended.
For more information about the following guidelines, see Database Maintenance Considerations in the white paper Site Server 3.0 Membership Directory Configuration and Tuning Guidelines.
DBCC NEWALLOC (MyPartitionDB)
DBCC CHECKDB (MyPartitionDB)
The problem of ongoing operational tuning can be approached from several directions. You may wish to collect and analyze performance statistics for your computers on an ongoing basis and then periodically review the collected data to see if the various parameters are meeting your expectations. You may also wish to set out defined times (such as weekly or monthly) to monitor the counters and collect statistics for analysis. Finally, you may just want to set Perfmon alerts to make notations in the event log when certain counters are not within your targeted range so that you can determine the cause. Whatever approach you choose, it is important to realize that the system will need attention as time goes by and users, applications, usage, and other considerations change. Plan time, particularly when the system is new, to work on tuning away any bottlenecks; it is rare that you will have gotten the configurations exactly right the first time.
Perhaps the most important performance measure to watch on an ongoing basis in the area of SQL Server disk health is the LogicalDisk object counter Current Disk Queue Length. Be aware that if your computers use hardware RAID, Windows NT cannot determine the number for actual physical disks, so divide the counter number for the logical disk by the number of physical disks participating in the array. This number should always be less than or equal to two. You can also look at the following counters for the LogicalDisk object:
For information on other tuning parameters to watch, see Compaq’s white paper SQL Server 6.5 Performance Tuning, available at http://www.compaq.com.
If you suspect that you are not getting the performance you should be, determine whether the bottleneck occurs in log or data files. This should be easy to determine by checking the disk queue length, so long as your transaction logs and data files are isolated onto different logical drives, as recommended. If the bottleneck is a particular logical data file, then SQL Server devices can be moved (using sp_movedevice) from the overtaxed drive set to a less busy drive set to help reduce the load.
The overall goal of watching the processor counters is to keep the server’s processors busy enough to maximize performance, but not so busy that processor bottlenecks occur. Two important counters to watch are the Processor object counter % Processor Time and the System object counter Processor Queue Length. The % Processor Time value should consistently be below 95% and the Processor Queue Length value should generally be less than or equal to two. The command DBCC SQLPERF(threads) can give you a running look at the amount of processor capacity each of the user threads is consuming.
Several SQL Server counters help you tune memory as well. The most important are Cache Hit Ratio and I/O – Page Reads/sec.
If, after turning off SQL Server’s Read Ahead Manager, you find that the Cache Hit Ratio value deteriorates and falls below 90-95%, then you need more memory in the SQL Server data cache for optimal performance. Either allocate more of the existing RAM to the data cache or add more physical RAM to the computer. For information about Read Ahead Manager, see SQL Server Books Online for SQL Server 6.5.
In the optimal case, the I/O – Page Reads/sec value would be zero. This occurs if all pages are loaded into the data cache. The only way to reduce this counter is to add more memory to the data cache, or reduce the page demand by tuning queries (not an option for the Membership Directory).
When you first deploy a Membership Directory, you will have extra database capacity to account for future growth. Since the databases are not full, you can probably house more than one database on each SQL Server computer. As your Membership Directory grows, monitor the critical hardware resources (memory, disk, processor). When any one of these resources is overloaded, consider adding another SQL Server computer and moving some Membership Directory databases (typically partition databases) to the new computer to distribute the load.
Important All partitions must be set up before you add data to the partitioned containers. You can only add SQL Server computers to your system if one or more of your current SQL Server computers supports multiple Membership Directory databases. At most, you can have one SQL Server computer per Membership Directory database.
Moving existing partition databases from computer to computer is not very difficult. To do so, make a copy of the database using one of the standard methods (BCP, replication, or database/log dump) and apply the new database to the new computer. For information about reconfiguring the LDAP Services to use the database in its new location, see Changing Membership Directory Database Connections in the P&M section of the Site Server 3.0 documentation.
This document has provided a variety of information that is relevant to hosting large user communities with a Microsoft® Site Server 3.0 Membership Directory. The following approach is recommended:
For the latest information on Microsoft® Windows NT® Server, visit our World Wide Web site at http://www.microsoft.com/backoffice or the Windows NT Server Forum on The Microsoft® Network (MSN™) (GO WORD: MSNTS).
Information in this document, including URL and other Internet web site references, is subject to change without notice. The entire risk of the use or the results of the use of this resource kit remains with the user. This resource kit is not supported and is provided as is without warranty of any kind, either express or implied. The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
© 1998-2000 Microsoft Corporation. All rights reserved.
Microsoft, MSN, Windows and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the U.S.A. and/or other countries/regions.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.