Polling for Changes Using USNChanged

The DirSync control is robust, efficient, and easy to use. But it has two significant limitations:

• Only for highly-privileged programs: To use the DirSync control, a program must run under an account that has the SE_SYNC_AGENT_NAME privilege on the domain controller. Very few accounts are so highly privileged, so an application that uses the DirSync control can't be run by ordinary users.
• No subtree scoping: The DirSync control returns all changes that occur within a naming context. An application interested only in changes that occur in a small subtree of a naming context must wade through many irrelevant changes, which is inefficient both for the application and for the domain controller.

There's another way to get changes from Active Directory™ that avoids these limitations: uSNChanged querying. This alternative is not better than the DirSync control in all respects – it involves transmitting all attributes whenever any attribute changes, and it requires more work from the application writer to handle certain failure scenarios correctly. But it is the best way available to write certain change-tracking applications today.

Here's the technical background on the uSNChanged attribute:

• When a domain controller modifies an object it sets that object's uSNChanged to a value that's larger than the previous value of uSNChanged for that object, and larger than the current value of uSNChanged for all other objects held on that domain controller. As a consequence, an application can find the most-recently changed object on a domain controller by finding the object with the largest uSNChanged, the second-most-recently changed object on a domain controller by finding the object with the second-largest uSNChanged, and so on.
• The uSNChanged attribute is not replicated; therefore reading an object's uSNChanged attribute at two different domain controllers will typically give different values.

With that background, it is easy to see the general outline of how to use uSNChanged to track changes in a subtree S. First perform a "full sync" of the subtree S. Suppose the largest uSNChanged seen on any object in S is U. Now periodically query for all objects in subtree S whose uSNChanged is greater than U. The query will return all objects that have changed since the full sync. Set U to the largest uSNChanged among these changed objects, and you are ready to poll again when the time comes.

The subtleties of implementing a USNChanged synchronization application are as follows:

• Use the highestCommittedUSN rootDSE attribute to bound your uSNChanged filters. That is, before starting a full sync, read the highestCommittedUSN of your affiliated DC. Then, perform a full synchronization query (using paged results) to initialize the database. When this is complete, store the highestCommittedUSN value read before the full sync query; to use as the lowerBoundUSN for the next synchronization. Later, to perform an incremental synchronization, once again read the highestCommittedUSN rootDSE attribute. Then query for relevant objects (using paged results) whose uSNChanged is greater than the lowerBoundUSN value saved from the previous synchronization. Update the database using this information. When that's complete, update lowerBoundUSN from the highestCommittedUSN value read before the incremental synchronization query. Always store the lowerBoundUSN value in the same storage that the application is synchronizing with the DC's content.

  Following this procedure, rather than the more obvious one based on uSNChanged values on retrieved objects, avoids making the server re-examine updated objects that fall outside the set that's interesting to the application.

• Because uSNChanged is a non-replicated attribute, the application must bind to the same DC every time it runs. If it cannot bind to that DC it must either wait until it can do so, or affiliate with some new DC and perform a full synchronization with that DC. When the application affiliates with a DC it records the DNS name of that DC in stable storage (the same storage it is keeping consistent with the DC's content.) Then it uses the stored DNS name to bind to the same DC for subsequent synchronizations.
• The application must detect when the DC it is currently affiliated with has been restored from backup, since this can break consistency. When the application affiliates with a DC it caches the "invocation id" of that DC in stable storage (the same storage it is keeping consistent with the DC's content.) The "invocation id" of a DC is a GUID stored in the invocationId property of the DC's service object. To get the distinguished name of a DC's service object, read the dsServiceName attribute of the rootDSE.

  Note that when the application's stable storage is restored from backup there are no consistency problems because the DC name, invocation id, and lowerBoundUSN are all stored together with the data being synchronized with the DC's content.

• Use paging when querying the server (both full and incremental synchronizations), to avoid the possibility of retrieving huge result sets all at once. See Paging.
• Perform index-based queries to avoid forcing the server to store large intermediate results when using paged results. See Indexed Attributes.
• In general, do not use server-side sorting of search results, which can force the server to store and sort large intermediate results. This applies to both full and incremental synchronizations. See Sorting the Search Results.
• Deal gracefully with "no parent" conditions. The application may "see" an object before it has seen its parent. Depending upon the application this may or may not be a problem. The application can always read the current state of the parent from the directory.
• To handle moved or deleted objects, you must store the objectGUID attribute of each object you are tracking. An object's objectGUID attribute remains unchanged regardless of where it is moved throughout the forest.
• To deal with moved objects, you must either perform periodic full synchronizations or increase the search scope and filter out uninteresting changes at the client end.
• To deal with deleted objects, you must either perform periodic full synchronizations or perform a separate search for deleted objects whenever you do an incremental synchronization. When you query for deleted objects, retrieve the objectGUIDs of the deleted objects to determine the objects to delete from your database. See Retrieving Deleted Objects.
• Remember that the search results include only the objects and attributes that the caller has permission to read (based on the security descriptors and DACLs on the various objects. See Effects of Security on Queries.

For sample code that demonstrates the basics of a USNChanged synchronization application, see Example Code to Retrieve Changes Using USNChanged.