The Membership Directory is the central data repository for Microsoft® Site Server version 3.0, which is a feature of Microsoft® Commercial Internet System (MCIS) version 2.0. You can use the Membership Directory to register users and manage user data, protect and share user data, verify users’ identities, and control access to site content.
This paper discusses issues associated with managing the Membership Directory when a complex structure of users or products is required. It presents the internal Membership Directory architecture used to support large numbers of groups or containers or complex structures. It also describes limitations of Personalization & Membership (P&M) RTM code, identifies hot fixes to the code to remove limitations, and presents architectural solutions for these demanding user applications.
Note Personalization & Membership (P&M) RTM code is the software that shipped with Site Server 3.0 and MCIS 2.0.
Complex Web applications are being developed to take advantage of the flexibility of the Membership Directory. Large product families can be modeled in the Membership Directory. It can also be used to organize users into multiple small communities for applications such as family accounts, business-to-business commerce, large numbers of user-interest communities, and controlled numbers of personal Web pages.
Administration of portions of the Membership Directory can be delegated securely. This is advantageous when the Membership Directory is organized to model a real-world structure like those described earlier. For example, an individual product manager may be responsible for a subset of products. In the case of family accounts, a family member will likely determine how to apply parental controls. And in business-to-business commerce, a company employee may be designated to determine who may place orders, as well as their associated purchase limits. The Membership Directory makes it possible to delegate these sorts of responsibilities.
The Membership Directory architecture supports the user applications described in the previous section by creating a secure hierarchical Directory structure. The key components used to accomplish this are containers and Membership Directory access control lists (ACLs). This section describes those structures and how they are used to create a directory to support the outlined user applications.
The hierarchy of the Site Server Membership Directory is organized using structures called containers. There are two pre-defined container objects in the Directory: organizationalUnits and mgroups.
In the Membership Directory, containers can be created under ou=Members,o=Root or under ou=Groups,o=Root containers.
By default, a parallel Windows NT group is created by the Site Server Authentication Service each time an mgroup is created. This is useful if Windows NT ACLs will be used on Windows NT file system content (such as Web pages, data files, or ASP pages) or on your site’s Web pages. If administration of very large numbers of communities will be delegated on your site, refer to the section titled Windows NT Group Creation later in this document.
In the Membership Directory, mgroups may be found anywhere under the ou=Groups,o=Root container branch of the Directory Tree. It is also possible to create mgroups under the ou=Members,o=Root container branch of the Directory Tree, but features such as automatic Windows NT group creation are not available.
Both organizationalUnit and mgroup containers are structures in the Membership Directory Information Tree (DIT). The DIT structure is considered to have changed whenever one of these containers is added, deleted, or renamed.
By default, each object in the Membership Directory has an access control list (ACL), which is a list of security IDs (SIDs) and their permissions for that object. ACLs are added to Membership Directory objects when it is necessary to secure the objects. To secure an object in the Membership Directory, an mgroup is created under the ou=Groups,o=root container. The mgroup is then added to the ACLs on an organizationalUnit container to provide appropriate access control. Administrators are then added as a MemberOf object to the mgroup.
By default, all objects created in the Membership Directory inherit the ACL placed on the root node. This is convenient if security is needed on Directory objects. However, there are performance implications associated with proliferation of ACLs in the MembershipDirectory. If this inherited ACL is not needed, it should be removed using Microsoft® Management Console. See the section titled DIT Caching Performance later in this document for more information.
By default, the Membership Directory creates a Windows NT group on the server running the Site Server Authentication Service each time an mgroup is added to the ou=Groups,o=Root container. Automatic creation of Windows NT groups is particularly useful if your application uses ACLs on content, because Windows NT ACLs are used to protect content.
The Membership Directory can maintain security principals on objects within the Directory without creation of Windows NT groups. Administrators of sites with very large numbers of mgroups may want to disable automatic creation of Windows NT groups to avoid the performance impact of having correspondingly very large number of Windows NT groups. In extreme cases, the number of Windows NT groups can become so large that they exceed the limits of the Windows NT directory database.
The Lightweight Directory Access Protocol (LDAP) service uses a memory cache to optimize performance when finding objects and determining group memberships for the authentication service to use in building security contexts. The cache consists of:
Due to the fact that P&M groups are represented as mgroup container objects, all groups are cached in both the DIT container cache and the groups cache. However, the DIT cache includes all container objects in the directory (including any customer-defined object classes), not just group objects. Note that all container-class objects are cached, regardless of whether they actually contain any child leaf objects.
The DIT information is cached when the LDAP service instance starts. It is recached whenever a change to the DIT structure is made by the local LDAP service, unless re-caching is disabled using the appropriate registry key. Refer to Appendix C: Registry Key Information for more information.
At a site with a single LDAP server, any change to the DIT is automatically and immediately reflected in the LDAP cache. At a site with multiple LDAP servers, DIT changes can be made by any of the LDAP servers. To ensure up-to-date DIT information in the cache, each LDAP service periodically polls a timestamp in the Directory database to see if any DIT changes have been made by another LDAP server since the last local DIT cache. If so, then it will recache the DIT information to bring it up to date.
The table below summarizes re-caching behavior on the LDAP server.
| RE-CACHE MATRIX | RE-CACHE BEHAVIOR OF MULTIPLE LDAP SERVERS | ||||
| Operation | Mgroup | orgUnit | GroupMemberOf | MemberOf | |
| Add | Offside DIT & Group | Offside DIT | All | None | |
| Modify | All | Offside DIT | All | None | |
| Delete | All | Offside DIT | All | None | |
| ACL Change* | Offside DIT & Group | Offside DIT | Offside | None | |
* Any change to security (adding or removing an ACL on a container)
A hierarchical Membership Directory with many small communities is built by adding containers under the ou=Members,o=root container. Figure 1 depicts a hierarchical Membership Directory with an individual (Lefty), a single level organization (Comp1), and a multi-level company (Comp2). (The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person or event is intended or should be inferred.)
Figure 1: Membership Directory Communities
In Figure 1, Lefty is an individual or child leaf object. Comp1 is an organizationalUnit container representing a sub-community containing several child leaf objects (Joe, Mary, and Frank). Comp2 is a nested organizationalUnit container with two additional organizationalUnit containers, each representing a sub-community (Comp2a and Comp2b). Comp2a is an empty organizationalUnit container. Comp2b has child leaf objects (Sally and Sue). Any of the organizationalUnit containers included in Figure 1 may have additional organizationalUnit containers added at a future date.
Administration of these accounts can be delegated by setting ACLs on the organizationalUnit containers under ou=Members,o=Root to control use of administrative functionality. To do this, using Directory ACLs, first create an ACL for each organizationalUnit container you want to administer separately. By default, the Membership Directory creates ACLs on all organizationalUnit containers under ou=Members,o=Root. (See the section titled Number of Access Control Lists later in this document for a discussion of performance implications.) Then mgroups are created under ou=Groups,o=Root. Mgroups are added to ACLs on the organizationalUnit containers under ou=Members,o=Root. Finally, users are placed in the mgroups as MemberOf objects.
In Figure 2, Frank is designated as the administrator of Comp1 and Sue is the administrator of Comp2b. (If there is only a single administrator for each organizationalUnit container, it is more efficient to add the member directly to the container’s ACL rather than doing this through a group.) Frank and Sue have been added to an mgroup on the ACL, which allows them to gain access and perform administrative tasks for their respective groups.
Figure 2: Membership Community Administration
When retrieving user data for authentication and personalization operations, the LDAP service normally performs a direct lookup of a user object from the full distinguished name (DN) of the object. The DN is similar to a fully-pathed file name in that it includes the complete hierarchical path information from the root of the directory to the specific object. The DN for Lefty in Figure 1 is cn=Lefty,ou=Members,o=Root, while the DN for Sue is cn=Sue,ou=Comp2b,ou=Comp2,ou=Members,o=Root.
In addition to the direct lookups, the LDAP service also supports one-level and subtree searches where the exact DN of the object is not available, but one or more attributes of the object are known. If the path to the object’s parent container is known, the search is called a one-level search. If only some attributes of the desired object, but not the parent container, are known, the search is called a subtree search.
A subtree search can be initiated either from the root of the directory, in which case the entire directory is searched for objects meeting the designated criteria, or from some lower node (group or container) in the directory and search only in the starting group/container and any containers below that. A subtree search starting at some container below the root is called a partial subtree search because it does not include the entire directory tree.
If a site has a large hierarchy (more than 5,000 containers), partial subtree searches that include these containers will be slow and should generally be avoided. Subtree searches from the root provide better performance.
The following are examples of subtree searches from the root:
The following are examples of partial subtree searches:
This section describes the behavior of the Personalization & Membership RTM code. This is the software that shipped with Site Server 3.0 and MCIS 2.0.
The original Membership Directory RTM code limited the number of subcontainers that can be included in a directory OU container when doing a partial subtree search. A full subtree search (a search that starts at the root directory) is not affected by this limitation.
The technical reason for the limitation is that, in the case of a partial subtree search, the LDAP service must specify the portion of the directory to be searched. Only the LDAP service understands the hierarchical tree structure of the directory. The representation of the Membership Directory in the SQL Server database only contains a parent object ID to indicate where in the directory an object exists. Formulating an SQL query for a partial subtree search requires the LDAP server to add a clause to the SQL query specifying all of the possible parent containers that the target object might have (for example, WHERE ParentContainerID=x1 or ParentContainerID=x2 or ParentContainerID=x3…).
The LDAP service uses a set of pre-allocated buffers to build the SQL queries. The original RTM code had a buffer length that only allowed a clause specifying approximately 200 parent objects.
Many customers who have large numbers of groups in their directory are not affected by the partial subtree search issue because they use custom tools or do not do partial subtree searches.
There are three issues in Membership Directory RTM code related to DIT and group caching performance:
The automatic creation of Windows NT groups with each DIT mgroup is a useful feature if you need to protect content on your site. However, if your site has a very large number of groups or containers, be aware there is a limit on the number of groups in the Windows NT directory database. That limit is approximately 80,000 groups per server. If you anticipate having more than 80,000 groups or containers in your site, see the Content Access Rights in Large Communities section of this document.
Hot fixes are available to remove the previously-described limitations. Contact Microsoft Product Support Services at http://support.Microsoft.com/support for information about how to obtain the hot fixes. Note that in order to download all hot fixes, you must have already installed Microsoft Site Server 3.0, Service Pack 1.
All hot fixes will be included with Microsoft® Site Server, Service Pack 2 when it is released in late 1998. Site Server, Service Pack 2 will ship with the Microsoft® Commercial Internet System (MCIS) 2.5 in Q1CY99.
With Membership Directory RTM code, several factors aside from hardware configuration and total system load affect performance. These factors are:
This section discusses performance impacts associated with each factor.
As noted earlier, there are two types of LDAP cache. The entire DIT is cached on each LDAP server in the MCIS installation. The LDAP Service also separately caches all mgroups. LDAP Server memory requirements go up in proportion to the number of containers in the DIT, as well as the number of mgroup containers. In addition, startup time is adversely affected by the number of containers that are being loaded into the two caches. See Appendix B: Performance Data for more information.
The size of the LDAP Service cache can be reduced using Active Server Pages (ASP) code. See the Writing Custom ASP Code section of this document for more information.
Each LDAP service periodically polls a timestamp in the Membership Directory database to see if any DIT changes have been made by another LDAP service since the last local DIT cache. If so, then it will recache the DIT information to bring it up to date. A registry key setting (dsPollThdRate) controls the polling interval. The interval is reset when the LDAP Service cache has been reloaded.
While re-caching is underway, the DIT can be read, but it is not possible to make changes such as adding an organizationalUnit container or an mgroup container. If the structure of the DIT changes frequently (as might be the case at a site where users can self-register their community), LDAP services could spend a significant amount of time re-loading DIT and group caches. This effectively prevents other users from registering while re-caching is underway. If this behavior is undesirable, alternate solutions such as pre-creation of containers should be considered (see the Architectural Solutions section of this document).
It is also possible to use a single LDAP server for adding containers to your Membership Directory. This would prevent the latency that takes place when re-caching occurs, because re-caching would only occur on the other (offside) LDAP servers and not the server managing structure modifications.
The Membership Directory makes it easy to use ACLs for authorization control. Membership Directory ACLs are stored in the LDAP cache to make it possible to quickly determine who is authorized to access organizationalUnit and mgroup containers. Large numbers of ACLs in a directory significantly increase memory requirements on the LDAP server and lengthen the time spent by the LDAP server loading its cache. If your site has very large numbers of containers, consider writing custom ASP code to control authorization, which will minimize caching and re-caching time. This is particularly important at sites where the structure of the DIT changes frequently. See Appendix B: Performance Data for performance information.
This section discusses several approaches that can be used to enhance Membership Directory performance for sites with large numbers of communities or complex product hierarchies.
Generally, the best way to avoid performance problems associated with large numbers of organizationalUnit and/or mgroup containers is to reduce the number of containers. ASP code can accomplish the same grouping and authorization functions provided by the Membership Directory.
Writing ASP code can provide one or both of the following benefits:
Note Anyone who connects directly to the Membership Directory can alter values in this attribute for their member record using the SELF security principal. A user security principal is a placeholder that grants/denies access for the current user to that user’s own attributes. The security principal is used to determine which user attributes a user will be allowed to see and/or change. For instance, you could set the ACL on the user password attribute for an entire organizationalUnit container to read/write for SELF and no access for anyone else. Then, without having to set individual ACLs for each user’s profile, each user could change their own password. Care should be taken to ensure that only ASP pages that enforce the delegated administration are allowed to access the Directory. Alternatively, access to these attributes can be explicitly denied.
The following sections describe the sequence of operations for registering a company, logging on, and administering a system using this architecture
If you anticipate hosting a large number of separate user communities, and the business of your site requires that communities be allowed to register in real time, system performance may degrade significantly due to the amount of time spent re-loading the LDAP cache. To alleviate this situation, consider either writing ASP code to administer the communities, dedicating a single LDAP server to registration, or creating an architectural solution similar to the one described in the following paragraphs.
This solution is based on the assumption that you want to avoid re-caching whenever a new community registers at your site, and that you are willing to pay service startup penalties to cache the DIT. You should also consider disabling automatic creation of Windows NT groups to avoid Windows NT directory database limits on the number of Windows NT groups. See the topic “PMAdmin Set AuthSvc” in the P&M section of the Site Server version 3.0 documentation for information about how to turn off automatic creation of Windows NT groups.
With this solution, you pre-create batches of containers before they are needed, at times when re-loading DIT changes will have minimum impact on Membership Directory performance (such as during scheduled downtime). You will need to create the following system constructs:
After the system constructs have been created and a single company named ABC has been added, the directory and the list of unused containers might look like the diagram
in Figure 3.
Figure 3: Membership Directory Structures to Avoid Run-time Recache
After the containers have been pre-created and the list of unused containers established, the run-time procedure is as follows:
Normally, the Membership Directory uses ACLs to selectively control access to content based on community membership. However, if a very large number of Windows NT groups is needed to protect content on a site, there is a risk of exceeding the maximum number of groups supported by the Windows NT directory database. To alleviate this situation, consider either writing ASP code to control access to content or creating an architectural solution similar to the one described in the following paragraphs.
This solution is based on the assumption that you want to provide selected content access rights to very large numbers of sub-communities. It is also assumed that you want to use Windows NT ACLs to simplify coding and operations, but expect to extend beyond the Windows NT directory database group limit of approximately 80,000 for a single computer.
With this solution, you create multiple, separate Membership Directories running on separate computers, and use a custom mapping table to direct users to the correct logon page. The key constructs are:
After the constructs have been created, the MCIS installation and Membership Directory may look like the diagram in Figure 4.
Figure 4: Large Numbers of Communities with Access Control Lists
Once the containers have been pre-created and the list of unused containers established, the following run-time behavior takes place:
Use this appendix to help determine problems associated with Membership Directory Group Scalability.
The tables in this appendix show preliminary results of testing performance with the Membership Directory hot fixes in place. The following hardware and software was used to obtain the results listed.
| LDAP Servers: | Dual Intel Pentium II processors, 400 MHz, 128 MB of RAM, Microsoft® Windows NT® Server version 4.0, Windows NT Service Pack 3, Microsoft® Internet Explorer version 4.01, Windows NT 4.0 Option Pack |
| SQL Server Computer: | Quad Intel Pentium Pro processors, 200 MHz, 512 MB of RAM, RAID 0 disk array, Windows NT Server version 4.0, Windows NT 4.0 Service Pack 3, Internet Explorer version 4.01, Windows NT 4.0 Option Pack, Microsoft® SQL Server™ version 6.5, SQL Server Service Pack 4 |
Note Your performance numbers may vary from those provided here based on your hardware and software platform.
This table shows the amount of time required to add containers to a clean database. This is a one-time cost. Time values represent the number of seconds it takes to create the specified number of containers.
| NUMBER OF CONTAINERS | ORG UNIT CONTAINERS | Mgroup CONTAINERS | ||
| Time | Adds/sec | Time | Adds/sec | |
| 10,000 | 275 sec | 36.4 | 304 sec | 32.9 |
| 25,000 | 960 sec | 26.1 | 932 sec | 26.8 |
| 50,000 | 1983 sec | 25.2 | 2193 sec | 22.7 |
| 100,000 | 4363 sec | 22.9 | 5119 sec | 19.5 |
This table shows the time it takes to load the indicated number of containers/groups into the memory cache. This is a recurring cost at service startup and at each recache. Separate columns show the impact of loading ACLs.
| NUMBER OF CONTAINERS | ORG UNIT CONTAINERS | Mgroup CONTAINERS | ORG UNIT/Mgroup COMBINED * | |||
| w/o ACLs | with ACLs | w/o ACLs | with ACLs | w/o ACLs | with ACLs | |
| 10,000 | 8 sec | 23 sec | 9 sec | 12 sec | ||
| 25,000 | 11 sec | 56 sec | 14 sec | 25 sec | ||
| 50,000 | 18 sec | 64 sec | 20 sec | 46 sec | 21 sec | 56 sec |
| 100,000 | 34 sec | 80 sec | 46 sec | 104 sec | 41 sec | 199 sec |
* Load time for 25,000 organizationalUnit containers and 25,000 mgroup containers (50,000 total containers), without any ACLs is 21 sec. Load time for 50,000 organizationalUnit containers and 50,000 mgroup containers (100,000 total containers) with ACLs is 199 seconds.
This table shows the amount of LDAP server memory needed to cache containers in the DIT. Tests were performed with default ACLs. Performance will vary based on length and complexity of ACLs.
| NUMBER OF CONTAINERS | ORG UNIT CONTAINERS | Mgroup CONTAINERS | ORG UNIT/Mgroup COMBINED* | |||
| Mem Usage (w/o ACLs) | Mem Usage (with ACLs) |
Mem Usage (w/o ACLs) | Mem Usage (with ACLs) |
Mem Usage (w/o ACLs) | Mem Usage (with ACLs) | |
| 10,000 | 14.7 MB | 41.7 MB | 15.1 MB | 19.3 MB | ||
| 25,000 | 21.8 MB | 73.0 MB | 22.4 MB | 34.2 MB | ||
| 50,000 | 33.5 MB | 83.0 MB | 34.7 MB | 54.4 MB | 43.8 MB | 78.8 MB |
| 100,000 | 56.9 MB | 88.1 MB | 59.4 MB | 63.9 MB | 76.9 MB | 87.3 MB |
* Memory usage for 25,000 organizationalUnit containers and 25,000 mgroup containers (50,000 total containers), without ACLs is 43.8 MB. Memory usage for 50,000 organizationalUnit containers and 50,000 mgroup containers (100,000 total containers) with ACLs is 87.3 MB.
The following three registry keys are new with Quick Fix Engineering (QFE) hot fixes 47502 / 47505 / 47597. All are found under HKLM\system\currentcontrolset\services\ldapsvc\parameters on the LDAP server, and any changes to the key values are effective only when the LDAP service instance is started or restarted.
The following are other existing related registry keys that are of interest to the issues of group/container scalability and performance of the LDAP service:
Information in this document, including URL and other Internet web site references, is subject to change without notice. The entire risk of the use or the results of the use of this resource kit remains with the user. This resource kit is not supported and is provided as is without warranty of any kind, either express or implied. The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person or event is intended or should be inferred. Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
© 1999-2000 Microsoft Corporation. All rights reserved.
Microsoft, Windows and Windows NT are either registered trademarks or trademarks of Microsoft Corporation in the U.S.A. and/or other countries/regions.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.