XADM: Slow Intersite Directory Replication

• Microsoft Exchange Server, versions 5.0, 5.5

SYMPTOMS

An organization's intersite directory replication that has previously been running without problems may start to fail to keep up with the changes made to it's (the organization's) directory.

This can be caused by a Directory Replication bridgehead server being unable to cope with the sheer volume of replication messages that it is having to send and receive. This is likely to be because the schedule set on the organization's Directory Replication Connectors is too demanding for the intersite replication topology and messaging infrastructure trying to support it.

The Intersite Directory Replication Schedule

How Intersite Directory Replication Works

   1068 - Ask for updates for naming context (either site or
          configuration)
   1100 -> Message submitted
   1058 - Completed successfully 

   1099 <- Message received from requesting directory
   1070 - The context to get, from the starting USN
   1071 - The number of objects retrieved, and entries up to the USN
   1101 -> Message submitted back to the requesting connector 

   1099 <- Message received back from the remote Directory 

Maximum Objects Sent per Request

The Number of Directory Replication Messages a Day in the Organization

N = number of sites participating in intersite directory replication

M = (N-1) * 2 = number of replication messages sent out (for both naming contexts)

F = The number of times each connector is active a day (24 / 3 by default = 8)

2 = factor for every request must get a reply

N sites * (M * F * 2) = Intersite Replication Messages per day

For example, using the default schedule:

10 sites * (18 * 8 * 2) = 2880 messages a day 20 sites * (38 * 8 * 2) = 12160 messages a day 30 sites * (58 * 8 * 2) = 27840 messages a day

CAUSE

Directory replication will slow up significantly if the Directory Replication Connector becomes active (as according to the schedule) and the replies from the remote bridgehead server have not been processed from the previous cycle.

In this situation, the connector must presume that the replies that have not been processed will not be forthcoming. Thus, the connector will request the same updates (plus any other which have happened) from the remote bridgehead server.

It is possible that in a large organization where an aggressive schedule for intersite replication has been set, these messages may be getting held up in the directory service mailbox on the directory bridgehead server.

NOTE: It might be reasonable for this to happen in organizations that have implemented the widely adopted hub and spoke directory replication topology, where central hub server(s) are responsible for passing directory updates between the spokes.

The process of taking the messages from the directory service's inbox and giving them to the Directory Replication Agent is performed on a single thread in the Directory service (DSAMAIN). The messages in the inbox must also be sorted (TABLE_SORT_ASCEND) on the client submit time (PR_CLIENT_SUBMIT_TIME), this becomes a computationally expensive operation as messages build up.

To see the number of messages waiting to be processed by the Directory Replication Agent, view the "Total no. Items" column for the Directory Service on the mailbox resources page of the Private Information Store object in the Exchange Server Administrator program.

If you believe messages are building up in this mailbox you might want to observe what the Directory Service is doing. To do this using Performance Monitor, add the following counter:

Object: Thread
Counter: % Processor Time
Instance: All the instances for DSAMAIN (use shift key to select)

WORKAROUND

To work around this problem, perform one of the following:

• Reduce the replication schedule for ALL replication connectors on the directory bridgehead servers adjacent to (connected to) the backlogged bridgehead server. (Microsoft strongly recommends that the schedule is NEVER more regular than the default: once every 3 hours).
  
  -AND-



• Reduce the replication schedule on ALL replication connectors on the backlogged server. (Microsoft strongly recommends that the schedule is NEVER more regular than the default: once every 3 hours).
  
  -AND/OR-



• Move some of the directory replication connectors off the backlogged server to another Exchange Server computer in the site.
  
  NOTE: If the third workaround is to be adopted, then it must be understood that a full refresh of all replicated contexts is automatically triggered (both ways). This is likely to compound the problem if the first and second workarounds are not applied first.
  
  NOTE: The third workaround is preferable to removing a directory replication connector. If the directory replication connector is removed, the adjacent connector must also be removed. Before a replication connector can be re-added, a Knowledge Consistency Check must be successfully completed on both directories. This will delete the objects that had been replicated between the two sites.
  
  -AND/OR-



• If the backlog in the Directory Service mailbox has become far too large (10,000 or more messages) it maybe necessary to remove these messages to give the DRA a chance to catch up. This can be done using one of the following two options:
  
  
  1. If the private information store on this computer has no other mailboxes on it other than the Directory Service and the System Attendant mailbox, you can rename the Priv.edb file:
     
     
     1. Stop the MSExchangeIS.
     
     
     
     2. Rename [Drive]:\Exchsrvr\Mdbdata\Priv.edb to [Drive]:\Exchsrvr\Mdbdata\Priv.old
     
     
     
     3. Start the MSExchangeIS. A new Priv.edb will be created when the IS starts, the mailbox for the directory service will be created when the directory service starts.
     
     
     
  
  
  
  2. If you know the Exchange service account user name and password, you can use an Exchange/Outlook client to log on to the Directory service mailbox and remove the messages from the Inbox.
  
  
  

MORE INFORMATION

It is likely that this problem will be initially triggered by extra- ordinary behavior; either in the number of Directory Replication messages generated over a period of time (such as adding sites to the Exchange Server organization), or a failure in the messaging infrastructure supporting the intersite replication (such as Message Transfer Agent downtime).

After a Directory service mailbox has a backlog of messages (which it has to keep sorted on client submit time), it will become increasingly difficult for it to pass messages to the Directory Replication Agent. If none of the above resolutions are adopted quickly, the remote bridgehead server connectors will continue to send their requests into the Directory service inbox, compounding the problem.

Real world experience has shown that after a Directory Service mailbox grows to exceed more that 1,500+ replication messages, it will never be able to recover until at least one of the above resolutions are applied.

For more detailed information on designing intersite replication topologies, please see the white paper "Advanced Backbone Design and Optimization" first published on the August 1998 TechNet CD.

Keywords :
Version : WinNT:5.0,5.5
Platform : winnt
Issue type : kbprb