Troubleshooting with the IIS Exception Monitor

Aaron Barth
Microsoft Corporation

Updated April 23, 1999

Contents
Introduction
Background
What To Do About These Exceptions and Access Violations
Where to Get IIS Exception Monitor
Using This Information To Solve Your Problems
When All Else Fails: Contacting Microsoft Technical Support
Conclusion

Introduction

This article has been written to give you a look at a tool that Microsoft Technical Support has been using to solve complex problems with Microsoft® Internet Information Server (IIS). Learning how to use it effectively can help you to troubleshoot complex IIS problems under the following categories:

IIS exits and a Dr. Watson error log is generated.
The Web service exits abnormally.
The browser returns an "ASP 0115" error.
The browser returns a "Server Application Error" message.
The browser returns a "Server Too Busy" error.

There are Microsoft Knowledge Base articles on most of the topics above. However, each IIS installation is different. Companies use IIS as the building block on top of which they build their Web applications.

Some Web applications can use Active Server Pages (ASP) technology to connect to ODBC-compliant databases, and some applications can contain compiled business objects running within IIS. COM objects called Active Server Components can be written in many languages, including Microsoft C++, Microsoft Visual Basic® (VB), Java, Delphi, Powerbuilder, and others. These Active Server Components can be written in-house or purchased externally.

System administrators and developers can use the IIS Exception Monitor to diagnose problems within IIS. Once they have determined the cause of the problem, they can contact the developer or vendor of the dynamic-link library (DLL) and provide them with concrete information about the situation.

Background

Many problems with IIS could be caused by exceptions that occur within IIS's process space (Inetinfo.exe). The most common exception is the 0xC0000005 exception, which is called an access violation. Access violations occur if a DLL within the IIS process tries to access memory that is not allocated or to free memory that has already been freed.

This is all good to know, but how does it help with troubleshooting IIS?

Ever hear of the ASP 0115 error? Usually this error is raised to the browser and the IIS log because an access violation occurs. There is usually no one cause of these error messages because each site uses different components in its Web application.

Ever have the Web server stop without you stopping it? This can also be caused either by an access violation or another type of exception being thrown within IIS.

What To Do About These Exceptions and Access Violations

The Dreaded "ASP 0115" Error

error 'ASP 0115'
Unexpected error
/<Web Name>/<ASP file name>.asp
A trappable error occurred in an external object. The script
      cannot continue running.

If you get this error message, you have a leg up on this process. When the error is displayed on the returned ASP page, you now know two things:

what page it was that caused the problem to occur
that an exception occurred in an external object

If knowing the page that is causing the problem is enough for you to figure out the problem, then you don't need to go any further. However, if this is not the case for you, take a look at the second bit of information that this error gives us. This error tells you that an exception occurred in an external object.

Well, that is great information, right? All you need to do is to remove all calls to external objects on the page and see if it fails. This is usually not possible since removing all Server.CreateObject statements in an ASP page would leave your page with only the intrinsic objects, such as Application, Session, Request, and Response. ASP's power comes from being able to create an instance of any Active Server Component or COM object. If you remove all calls to external objects, this leaves you with a pretty boring page. ASP considers any object that it creates with a Server.CreateObject statement to be an external object.

Getting Under the Hood

Microsoft Technical Support has provided a tool that will pry the hood of this car we call IIS open so you can peek inside. This tool is called the IIS Exception Monitor. It attaches to the IIS Process and reports back what is going on. The IIS Exception Monitor can provide an explanation of which type of exception occurred, what thread the exception happened on, and what DLLs were executing functions on that thread.

This may not be enough information to solve the problem 100 percent of the time. However, it usually provides more information than you had before and narrows the scope of the problem down to focus on a certain ODBC driver or component that is being called.

Installing and Running the IIS Exception Monitor

Installing the IIS Exception Monitor should go smoothly provided you obtain the correct EXEs for the version of IIS and processor architecture of the IIS server. Use the link below to download the appropriate version.

Where to Get IIS Exception Monitor (version 6.1.0.0)

IIS Exception Analysis Tools (Intel)

IIS Exception Analysis Tools (Alpha)

Installing IIS Exception Monitor

Once you have downloaded the Ixcptmon.exe file, you can start the installation program by running the EXE.

You will be prompted whether you would like to install the IIS Exception Analysis Tools.

Figure 1. IIS Exception Analysis Tools 6.0 installation prompt

You will be asked for a directory for the IIS Exception Analysis Tools.

Figure 2. IIS Exception Analysis Tools 6.0 directory prompt

Choose a file path that does not contain any spaces or accept the default.
When the installation finishes it will automatically display the readme.txt. Once you close the readme.txt file, the IIS Exception Monitor setup will add all of the tools to the IIS Exception Monitor program group.
Run the IIS Exception Monitor by going to Start, selecting Programs, selecting IIS Exception Monitor, and then selecting IIS Exception Monitor again.

Note The installation program will associate the DBL extension to the readlogs.exe program for viewing logs by double-clicking on the DBL file itself. This setting will be removed when the IIS Exception Monitor has been uninstalled.

Walking Through an IIS Monitoring Session

When the wizard first starts you will see the Welcome window.

Figure 3. Welcome window

From this screen you have the option to click Next, Cancel, or Help. To continue running the IIS Exception Monitor and progress to the next page, press Next.

Note The View Existing Log Files check box can be selected if you want to jump to view log files that you have already created using the IIS Exception Monitor. If you select this check box, you must press Next to jump to the page where you can view the log files.

Using the Check Symbols page

Figure 4. Check Symbols dialog box

The wizard asks if you would like to verify the symbols that you have installed.
If you choose Yes (default), the wizard will look for files used by IIS and determine whether there is a matching symbol file in the \%systemroot%\symbols\dll directory.
If you choose No, you will be brought to the Process Options page. Press Next after you have made your selection.

Note There is an option to choose an alternate symbol path. If you check the Use Alternate Symbol Path for Exception Monitoring check box, type in the alternate symbol path in the space provided.

Using the Verify Symbols page

Figure 5. Verify Symbols window

After the progress bar has disappeared you will see a list of DLLs that either did not have a matching symbol file, or had a symbol file that did not match the DLL.
You can check the Determine which symbol packages can be installed from Microsoft's Internet Site check box to be taken to the Download page, where you can download the symbols from the Microsoft Internet Site (this requires an Internet connection).
Press the Next button after you have made your selection.

Using the Download page

Figure 6. Download Symbols window

After the progress bar has disappeared you will see three columns for each symbol package that can be downloaded from the Microsoft Internet site.
Note The symbol packages in this list can be downloaded from the Microsoft Internet site. You may have applied hotfixes that do not have corresponding symbol files, or the symbol files may not be available publicly for the DLLs in question. For a list of the DLLs with symbols that do not match please refer to the SymDiff.log file in the directory where the IIS Exception Monitor is installed.
Select the left column of the package(s) that you would like to download and press the Download button. If you select multiple files, the IIS Exception Monitor will download and install the files in the order presented. This requires an Internet connection.

Figure 7. Symbol Package installation dialog box
It is important that you download and install the symbols from top to bottom (since this is the same order that you have installed the software).
If you have downloaded a file with an .exe extension, the Wizard will ask if you would like to install the symbol package (this requires an Internet connection).
Press the Next button after you have made your selection.

Note The downloaded files will be placed in the \download directory underneath the directory where the IIS Exception Monitor is installed.

Using the Process Options page

Figure 8. Process Options dialog box

From this page you can choose whether to monitor an In Process (Inetinfo.exe) application, an Out of Process application (Mtx.exe) or any other process on the system.
If you choose In Process you will be brought to the Session Options page.
If you choose Out of Process you will be brought to a selection page to select the name of the Out Of Process application that you would like to monitor.
If you choose Other Process you will be brought to a selection page to select the name of the process that you would like to monitor.
After making your selection, press Next.

Using the Select Out Of Process Application page

Figure 9. Select Out of Process Application dialog box

After the progress bar has disappeared you will see two columns--Process ID and Application Name--for every instance of Mtx.exe. Note that this is the name of the application as defined in the Transaction Server Explorer when the application was marked to Run in a separate memory space.
Select the Process ID column for the Out Of Process application that you would like to monitor.
Press the Next button after you have made your selection. Note If you do not select an Out Of Process ID, the wizard will monitor Inetinfo.exe instead. The information message in Figure 10 explains this situation.

Figure 10. This message box appears if you do not select an Out of Process ID.

Using the Select Any Process Application page

Figure 11. Select Any Process dialog box

After the progress bar has disappeared you will see two columns--Process ID and Process Name
Select the Process ID column for the process that you would like to monitor.
Press the Next button after you have made your selection. Note If you do not select a Process ID, the wizard will monitor Inetinfo.exe instead. The information message above explains this situation.

Using the Session Options page

Figure 12. Session Options dialog box

From this dialog box you can choose whether you would like to run an Automatic Session or a Manual Session.
If you choose Automatic Session (default), a log file will be created for your analysis when an exception occurs.
If you choose Manual Session, you will be prompted for a Port, Username, and Password so that a support engineer can interactively troubleshoot your problem.
Note This option should be used primarily if you are currently working on an issue with technical support. If Manual is selected, then once an exception occurs the Web server will be suspended and will not be able to accept any Web requests until it is either restarted or the monitoring session has been canceled.
Advanced Options:
a. Enable Recursive Mode: Check this option if you want the IIS Exception Monitor to continue to monitor when INETINFO.EXE has written a log and restarted the process. This can be useful for restarting the Web services when trying to troubleshoot a problem with the Web server.

b. Notify Admin: Use this option to receive a popup message when the IIS Exception Monitor encounters an exception or if you have CDO and the SMTP service installed and specify a fully qualified email address (User@company.com) you can receive an e-mail message with the .DBL file as an attachment when an exception is detected.

c. Schedule This Session: enable this box if you would like to schedule an Exception Monitoring session to take place at a specified time or on system startup.
Note This requires the Task Scheduler to be installed. If the Task Scheduler is not installed, this option will be disabled

Using the Manual Session Options page

Figure 13. Manual Session Options dialog box

The Manual Session Options page allows you to choose connection information used for a technical support engineer to connect to the manual session.
Port: you can choose the port that the IIS Exception Monitor will use to listen on for a connection.
Username: the username needed to connect to this session (default: iisdebug).
Password: the password required to connect to this session (default: Microsoft).
Note The username and password specified above do not need to match any username or password that exists in your Windows NT® user accounts database. This username and password will only be used for this monitoring session. It is strongly recommended that you specify a username and password for your session to prevent unauthorized access.
Block Incoming IP Connections: Enable this box to prevent technical support, or anyone else, from connecting until you manually toggle this setting by typing @L at the console. This is a security feature (default: unchecked).
Press the Next button after you have made your selection

Using the Start Monitoring page

Figure 14. The Start Monitoring dialog box

The Start Monitoring page will display your selections from the previous pages. Press the Run this monitoring session button to start the monitoring session.
Note If you have chosen the Schedule this Session option, a dialog will appear. For more information on this feature, see the Advanced Monitoring Options section in this article.
The display will change to indicate the success or failure of the operation.
Once the above button is pressed the Next button will become active. Note You will notice a minimized Command window appear on the Start menu. This is your monitoring session.
Press the Next button to continue to the Status page.

Using the Status page

Figure 15. The Session Status dialog box

After the progress bar has disappeared you will see two sections. The Automatic Session information is on the top and the Manual Session information is at the bottom. If you have run other monitoring sessions you will see them listed, as well as the most recent session from the previous page.
Automatic Session section: In the upper window you should see five columns for every running or completed automatic session. The columns are LogFile, Process ID, Start Time, End Time, and Exception?.
There are two buttons for you to use. The first is the View Log button: Use this button to view DBL files in this list when they have completed. This action will run ReadLogs.exe, which is documented separately within this resource kit. Note that this button is only enabled when a DBL log has been selected. The second is the Refresh Status button: Use this button to get an up-to-date status on all Automatic Sessions (running and completed).
If a DBL file has been highlighted in the left column and the View Log button has been pressed, the Log Analyzer will be run.

Figure 16. The Log Analyzer dialog box
Manual Session section: In the lower window you should see six columns for every running Manual session. The columns are Process ID, Port, Username, Start Time, Current Time, and Exception?.
Note The Exception? column is for future use.
There are two buttons for you to use. The first is the Notify Support button: Highlight a Process ID in the lower window and press this button to send support an e-mail with connection information for this session. The case ID, e-mail, and SMTP servers are used from the previous Manual Session Options page. The second button is the Refresh Status button, which you use to get an up-to-date status on all running manual sessions.

Using This Information to Solve Your Problems

The DBL file that is created in the \ixcptmon\logs directory is a file that was created when the server IIS Exception Monitor encountered an exception. This file contains the output from a list of commands that can be parsed later and displayed by the Readlogs.exe tool.

What to Look for in ReadLogs

Figure 17. This log displays the output of the Readlog.exe tool.

Within the .dbl log file will be the output of a command kv. This is your stack of the thread that was executing when the IIS Exception Monitor encountered an exception. This stack contains the functions that were executing along with the parameters passed to the functions on the stack. This can be shown in the lower window of the Readlogs.exe tool.

If the DLL that faulted within IIS contains symbolic information, then this stack will tell you the name of the function that faulted. If the symbols were applied and match the installed DLLs, then this information on the stack is accurate. If a function within a DLL that does not have symbolic information were found on the stack, then you only see a memory address for the function in question.

The Readlogs.exe tool can utilize the starting and ending addresses of all DLLs loaded in the process at the time of the fault. The tool can also figure out which DLL's memory space the function on the stack lies within. That information will be shown in the Function Name column if the symbols are not present.

Advanced Monitoring Options

If you chose Schedule this Session from the Session Options page, then you will see the following dialog (Figure 18) box when you click on the Run this Monitoring Session button from the Start Monitoring page.

Figure 18. The Set Password dialog box

When the IIS Exception Monitor dialog box pops up, click on the Task tab and set the username (DOMAIN\username if applicable) and password for the account that you would like to run this scheduled session when no one is logged on.

Figure 19. The IIS Exception Monitor dialog box Task page

From the Schedule tab you can change the time that the task is executed. By default Schedule Task is set to execute At System Startup. If you would like to have multiple schedules, you can choose the Show multiple schedules check box and add a new schedule.

Figure 20. The IIS Exception Monitor dialog box Schedule page

From the Settings tab, make sure that you clear the Stop the scheduled task after xx hour(s) check box. This ensures that the task will not be stopped until it has completed.

Figure 21. The IIS Exception Monitor dialog box Settings page

When All Else Fails: Contacting Microsoft Technical Support

If you are able to examine a DBL file from your machine and are still no closer to solving your problem, you may need to contact Microsoft Technical Support to examine the problem further. The IIS Exception Monitor wizard is capable of setting up your system for remote diagnosis by an IIS engineer.

Remember the Session Options page (Figure 11)? This page allows you to choose whether you would like an automatic session or a manual session. In the previous example we chose an automatic session so we could get the log and examine it ourselves. If you would like to set up for remote diagnosis, you can choose Manual session. You will then be prompted for additional information.

Making the Connection

In order for Microsoft Technical Support to diagnose your machine, it must be connected to the Internet or have the ability, through a dial-up connection, to reach the Internet. You must choose an unused port number over which the connection will take place. For security, you should provide a unique username and password for this session. This username and password may be required to connect to this session remotely. All of these options are configurable on the Manual Session Options page.

Securing the connection

On the Manual Session Options page there is the option to choose any port number and specify a username and password for the connection. There is also a check box for disabling incoming IP connections. If you check this box, the IIS Exception Monitor session will not accept any socket's connection until someone presses @L in the console window to enable IP sessions.

Note When this is enabled, the response from this command will be LOCK: No IP Sessions Allowed.

Figure 22. "LOCK: No IP Sessions Allowed": response to disabling incoming IP sessions

Notifying support

Once you have a support incident open with Microsoft Technical Support, you can notify a Microsoft engineer by checking the Notify Support check box. You can then fill in the service request number, the e-mail address of the engineer that you are working with, and the IP address or DNS name of an SMTP server accessible from the IIS server.

Conclusion

The IIS Exception Monitor has been used successfully by Microsoft Technical Support to solve many issues with IIS. This tool does not have all of the answers. However, it can provide more information on the status of a failure than the Web server's logging mechanisms can.

Keep in mind that this tool is primarily used when the server is crashing and the IIS logs and the event logs do not contain enough information to solve the problem.

Aaron Barth is a Technical Lead for the Internet Server Support Team at Microsoft.