Troubleshooting Tips

If Status Manager is not correctly processing the status messages it receives:

• Verify that the site’s status filter rules are set to your desired configuration. Rehearse in your mind or on paper how various status messages will be processed using the site’s status filter rules. Review Chapter 20, “Understanding the SMS Status System,” in the SMS 2.0 Administrator’s Guide for more information about status filter rules.

  Systems Management Server   Site Database (site code - site name)      Site Hierarchy         site code - site name            Site Settings               Status Filter Rules

• Using the SMS Service Manager, verify that the SMS Executive service and the Status Manager thread component are running.
• Verify that the drive containing the SMS installation directory on the site server has at least 10 MB of free disk space.
• Examine the Statmgr.log file. Search for the strings “ERROR” and “WARNING” and investigate each occurrence. Each error and warning will include a brief statement of the problem that should point you to what you should investigate next.

If you suspect Status Manager is not processing incoming status messages, either via the in-memory queue or via *.svf files:

• Verify that the site database server is functioning properly and is accessible on the network from the site server using the SMS service account that you specified when setting up the site.

If you suspect Status Manager is experiencing a problem accessing the site database at a primary site:

• Verify that the site database server is functioning properly and is accessible on the network from the site server using the SMS service account that you specified when setting up the site.
• Verify that the Microsoft SQL Server services are running properly on the site database server.
• Using the Microsoft SQL Server 6.5 ISQL/w tool, or the Microsoft SQL Server 7.0 Query Analyzer tool, verify that you can connect to the SMS site database using the SQL administrator account you specified when setting up the site.
• Verify that SQL Server is configured for at least 50 SQL Server user connections, plus five for each SMS Administrator console.
• Verify that the SMS site database has at least 1 MB of free storage space and that the transaction log for that database is not full.
• Verify that the master and tempdb databases on the site database server are not full. Verify that the tempdb database is sufficiently large.
• Status Manager is designed to process incoming status messages even when the SMS site database cannot be accessed. While the site database is inaccessible, Status Manager will store pending database transactions as *.sql files in the SMS\Inboxes\Statmgr.box\Retry directory; these text files should not be deleted or altered in any way. Status Manager will periodically retry the oldest pending transaction. You can force Status Manager to retry the transaction by stopping and restarting Status Manager using the SMS Service Manager. Once the oldest pending transaction succeeds, Status Manager will rapidly execute all of the pending transactions chronologically, from oldest to newest.
• Verify that this works properly by monitoring the Statmgr.log file after correcting a problem related to the SMS site database. When a pending transaction is executed successfully, Status Manager deletes the associated *.sql file. Even after you resolve the site database problem, Status Manager will not write new status messages to the SMS site database until all of the pending transactions have been executed. In fact, while the pending transactions are executing, new status messages will be stored as transactions in new *.sql files. When the SMS\Inboxes\Statmgr.box\Retry directory is empty, the pending processing load has been exhausted and the system has returned to normal.

If you configured status filter rules that instruct Status Manager to pass status messages to programs as command line arguments, and you notice problems with this process:

• When you specify the program to execute in the status filter rule, be sure you specify the full path to the program you want to execute. For example, you should use “C:\Winnt\System32\Net.exe” instead of just “Net.exe.” Using the full path to the program makes your program accessible to Status Manager regardless of the value of the PATH environment variable.
• Status Manager’s processing will be blocked while your program is executing. For this reason, you should always specify programs that finish executing in a very short period of time — ideally, in less than one second.
• If your program takes more than 60 seconds to finish executing, Status Manager will assume your program is “hung,” stop waiting for it to finish, and resume processing other status messages. This can lead to a dangerous situation. If the status filter rules direct Status Manager to run your program for many of the status messages it receives, and your program does not finish quickly, a large number of running processes (instances of your program) can build up on the site server. If this situation continues to grow, the site server will run out of memory and will need to be rebooted.

If you observe a substantial, undesired delay between when Status Manager receives a status message and when it processes the status message or a general slowdown in Status Manager’s processing rate:

• Status Manager receives status messages from two sources: an in-memory queue and the SMS\Inboxes\Statmgr.box\Statmsgs directory. Thread components running on the site server report status messages using the in-memory queue. All other components, including client components, report status messages as *.svf files in the SMS\Inboxes\Statmgr.box\Statmsgs directory. Status Manager always processes all messages waiting in the in-memory queue before processing *.svf files in the SMS\Inboxes\Statmgr.box\Statmsgs directory. After the in-memory queue is emptied, Status Manager will read and process up to 1000 *.svf files before rechecking the in-memory queue.
• Determine the number of *.svf files in the SMS\Inboxes\Statmgr.box\Queue directory. If more than 1000 status messages are waiting in the in-memory queue, or if a status message has been waiting in the queue for more than 60 seconds, all of the status messages in the queue will be stored in the SMS\Inboxes\Statmgr.box\Qeue directory as *.svf files. This is done to minimize the risk of losing status messages due to a sudden power loss. If you discover a buildup of *.svf files in the SMS\Inboxes\Statmgr.box\Queue directory, use the Component Status summary in the SMS Administrator console to see if a thread component on the site server is “flooding” the status system by rapidly reporting the same status message over and over again. If you discover a flood, create a status filter rule that instructs Status Manager to discard the flooding status message and search the Microsoft Knowledge Base for more information about the problem. For more information about creating status filter rules, see Chapter 20, “Understanding the SMS Status System” in the SMS 2.0 Administrator’s Guide.
• Determine the number of *.svf files in the SMS\Inboxes\Statmgr.box\Statmsgs directory. Status Manager processes *.svf files chronologically, from oldest to newest. The more *.svf files there are waiting for processing, the longer the delay before they are processed. An extremely large number of small files (tens or hundreds of thousands of files less than 1KB in size) may indicate that a component is flooding the status system. If you suspect a flood, create a status filter rule that instructs Status Manager to discard the flooding status message and search the Microsoft Knowledge Base for more information about the problem. For more information about creating status filter rules, see Chapter 20, “Understanding the SMS Status System” in the SMS 2.0 Administrator’s Guide.
• A large number of large files (thousands or tens of thousands of files greater than 1KB in size) indicates that Status Manager is receiving many status messages replicated from child sites. If necessary, you can modify the status filter rules at the child sites so that unimportant status messages will not be replicated to the parent site.
• Determine the number of *.svf files in the SMS\Inboxes\Statmgr.box\Futureq directory. Every time Status Manager receives a status message, it compares the time that the message was generated to the current system time on the site server. If the generation time is greater than the system time, the message is categorized as being generated in the future. Status Manager writes the status message to a *.svf file in the SMS\Inboxes\Statmgr.box\Futureq directory. The file will remain in the directory until the system time catches up with the generation time, at which point the message will be processed normally.
• A large buildup of *.svf files in the SMS\Inboxes\Statmgr.box\Futureq directory probably indicates that there are many SMS clients at your site that have system clocks that are set substantially ahead of the site server’s system clock. You should correct this problem by synchronizing the system clocks of all computers in your organization to the correct real time. See the Microsoft Knowledge Base for more information about this procedure.
• To assist you in identifying the problem computers, Status Manager reports one status message every 24 hours for each computer that has a system clock that is at least five minutes ahead of the site server’s clock. Due to natural small delays in the network and processing, Status Manager can only approximate the amount of time by which a particular computer’s system clock differs. Also, Status Manager cannot determine if the time of a computer’s system clock is behind the site server’s system clock.

If you observe a general slowdown in Status Manager’s processing rate:

• Like many of the SMS components running on the site server, Status Manager’s processing rate is constrained mostly by the performance of the site server’s disk subsystem and by the performance of Microsoft SQL Server. You can achieve major performance gains by upgrading the disk subsystem on the site server; for example, you could upgrade from a standard SCSI controller and hard disk to a high-performance RAID array.
• You can also achieve major performance gains by improving the performance of Microsoft SQL Server. Microsoft SQL Server is also constrained by disk performance, so you should consider upgrading the disk subsystem on the site database server. You should also consider improving the network throughput between the site server and the site database server.
• The rate at which Status Manager can write status messages to the SMS site database will decrease when there are more status messages in the database. For example, in one performance test conducted by Microsoft, Status Manager was observed to write status messages into an empty SQL database running on a high-performance machine at an initial rate of about 400 messages per second. But when the database had accumulated 4GB of status messages (about 8 million messages), the rate dropped to about 100 messages per second.
• If you have a large number of status messages in your site database, you can improve the rate that new messages can be written by deleting some of the old status messages. One easy way to do this is to define a Status Message Query that specifies the messages you want to delete, right-click on the selected messages, and then click Delete Messages. You can also tune the system by adjusting the status filter rules, which control how long the status messages are kept in the SMS site database before being deleted by the Delete Aged Status Messages database maintenance task.