Now that you've spent the dollars and resources to develop an eye-catching and highly interactive site, it's time to consider monitoring your site. The user experience simply won't be as glossy as you expect if the site isn't functioning properly. Not surprisingly, as Web sites become more interactive and complex, the task of monitoring them becomes more challenging.
A Web site can be monitored and tested at several levels. For example, network monitors can analyze network traffic and detect outages. There are also tools that can detect when a particular Windows NT service has stopped. The monitoring/testing tool described in this article takes a user-based approach; it acts as a browser, and interacts with the Web site as a typical user would. The degree of interaction depends on the complexity of the site and your special monitoring needs. Using the techniques described here, you can develop a site monitoring tool with parameters tailored for your site to give you an effective, user-based eye.
Web users interact with their browser, which in turn interacts with a Web server. The Web server may, in turn, interact with back-end servers such as a database (see Figure 1). Rather than checking each piece independently, our strategy is to check only the Web server by "simulating" a browser. The idea is that if the check succeeds at that level, then all other systems must be functioning properly. This is a simple, yet effective approach to site monitoring.
Figure 1: This diagram illustrates our site monitoring tool.
The communication between the client and the server is HTTP based. While a typical browser must implement the entire protocol, a small subset will do for our purposes. We'll explain everything you need here, but if you are interested in the details of the HTTP protocol, a copy can be found at http://www.w3.org.
HTTP communication is divided into two main parts: requests and responses. Typically the client makes a request and the server responds. A simple request would be:
GET /index.html HTTP/1.0 <CR><LF>where <CR><LF> indicates a carriage return followed by a line feed. If you telnet to port 80 of a Web server and type the above command, you will get the HTML page (index.html) that would typically be sent to your browser. However, HTTP requests can be more complicated than this. In addition to the GET method, there is also the POST method used to dispatch content of HTML forms to the Web server. Also, a number of fields are populated, and form the Request Header. These fields include authorization information, details about the capabilities of the browser, etc.
By now you've probably guessed where we're going with this. The idea behind our site monitoring tool is to identify a number of URLs that effectively represent key components of the site. In other words, if these URLs are working, we can conclude the entire site is functioning properly. We would then identify key strings (lines) within the pages represented by each URL. That is, if the page returned from the Web server contains the unique line, we know our transaction was successful and that the systems are working properly. If, on the other hand, the page returned from the Web server contains an error message, or it times out, the unique line would not be found and we know that our transaction was unsuccessful.
With this in mind, our task is divided into the following steps:
The remainder of this article shows how a few lines of Java code can effectively accomplish these tasks. So let's get to the details.
Java is heralded for its networking capabilities. We take advantage of such capabilities offered by the jave.net.URL and java.net.URLConnection classes. By doing so, we avoid interaction with the details of the HTTP protocol, and allow Java to handle them for us. The code in Figure 2 shows how Java can be used to implement our four tasks.
Figure 2: Implementing the four tasks.
import java.util.*;
import java.net.*;
import java.io.*;
public class webMonitor {
public static void main( String argv[] ) {
String siteURL = "http://www.xyz.com/index.html";
String keyString = "key string data goes here";
String line = "";
try {
URL url = new URL(siteURL);
URLConnection connection = url.openConnection();
connection.setDoInput( true );
BufferedReader input =
new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
int i = 0;
while ((line = input.readLine()) != null) {
i++;
if (i == 3) {
if (line.equalsIgnoreCase(keyString))
System.out.println("Successful\n");
else
System.out.println("Failure\n");
}
}
input.close();
}
catch( Exception e ) {
System.out.println(
"An error occurred while checking the site.\n");
System.out.println( e );
e.printStackTrace();
}
}
}Here's a run-down of what the program in Figure 2 is doing. This a Java application, not an applet. As a result, the main class is declared. We did this so we could concentrate on the networking aspects of the application without the clutter of applets and HTML.
The program sets the variable siteURL to the URL we want to check. The variable keyString holds the string which indicates a successful request-response sequence. Finally the variable line will be used as placeholder to examine each line received from the server.
Next, we'll create an instance of the URL class and initialize it with the URL we're interested in monitoring. The URL class is part of the java.net package. We can now establish a connection to that URL. To do so, we use the URLConnection class, which is also part of the java.net package. It's intended to handle interaction with URLs. Note that there is another class named HttpURLConnection which is HTTP-specific. We opt to use the more general URLConnection class. The instance of URLConnection is named connection and will be used to manipulate the connection. This instance is set to the result of the openConnection method of the URL class. If a connection cannot be established, an IOException is thrown.
Because we'll be using the result of the connection as input, we must indicate it by calling the setDoInput method with an argument of true. We chose to handle the input via the BufferedReader class; however, there are some alternatives. It doesn't really matter, as long as you can grab the result sent from the Web server and examine it.
Our buffer for the result is simply named input. We read one line at a time, and compare it with keyString using the equalsIgnoreCase method which is part of the String class. Note that we use a counter variable named i to count the lines as we get them from the buffer. This allows us to be a bit more precise in the position of our key string. In other words, we not only require the key string to be present in the result obtained, we expect it to be at a particular line. For sites whose content change often, this may be more of a nuisance, because the line where our key string appears changes frequently. It will be up to you to decide whether this extra "exactness" is appropriate for your site.
Finally, depending on the result of our line-by-line comparison, we print a simple message and close the input stream.
To complete the try block, we add the catch block to handle exceptions that may have been thrown by the URL and URLConnection classes.
What we've just described accomplishes the four tasks outlined in the previous section. It can be used to monitor content-oriented Web sites. However, this covers only a small percentage of today's sites. More and more sites offer interactivity via connections to back-end systems such as database and e-mail servers. Our monitoring strategy must also be applicable to those sites. As it turns out, it is.
Interactions via the Web are usually accomplished via HTML forms. The user fills out a form, submits it, and receives an HTML page in response to that submission. The form itself may be handled via a CGI program, an application server, or a Java servlet; it really doesn't matter. What we have to do is to submit a typical form and examine the result we get. If our key string is found within the result content, then the application and its back-end components must have worked fine.
Unlike the previous example where we asked for a Web page and examined its result, we now have to simulate filling an HTML form. The HTTP method commonly used to submit forms is the POST method. Our strategy is to open a connection to a CGI script (of whatever the URL of the form handler is). You can find out how the form is handled by looking at the HTML source and find the method parameter inside the <FORM> tag. We then open a connection to this script, "post" some data to it, and retrieve the result it produces.
Figure 3 shows this process, which is similar to the previous program. One difference is that the URL we're monitoring is not an HTML page, but a script (in this case a CGI script named some.cgi). Note that we've introduced another variable named postQuery. This variable contains the HTML field values that lead to a particular response from the Web server. They are presented in the name=value format separated by an ampersand ( & ). The data is also URL-encoded.
Figure 3: Filling an HTML form.
import java.util.*;
import java.net.*;
import java.io.*;
public class webMonitor {
public static void main( String argv[] ) {
String cgiscript =
"http://www.xyz.com/cgi-bin/some.cgi";
String keyString = "key string data goes here";
String postQuery = "name1=value1&name2=value2";
String line = "";
try {
URL url = new URL(cgiscript);
URLConnection connection = url.openConnection();
connection.setDoInput( true );
connection.setDoOutput( true );
DataOutputStream output =
new DataOutputStream(connection.getOutputStream());
output.writeBytes( postQuery );
output.close();
BufferedReader input =
new BufferedReader(
new InputStreamReader(
connection.getInputStream()));
int i = 0;
while ((line = input.readLine()) != null) {
i = i + 1;
if (i == 3) {
if (line.equalsIgnoreCase(keyString))
System.out.println("Successful" + "\n");
else
System.out.println("Failure" + "\n");
}
}
input.close();
}
catch( Exception e ) {
System.out.println(
"An error occurred while checking the site");
System.out.println( e );
e.printStackTrace();
}
}
}Note that this time, we also prepare the connection for accepting input from us. We open an output stream and write the content of postQuery to the connection, which in turn sends it to the Web server. The input buffer is used to capture the output of the Web server which is then examined as we did before.
Some Web sites require authentication before they allow the browser to view a page. In this section, we'll show how our site monitoring tool can handle these sites. To implement a solution, we need some additional details about the HTTP protocol. We have talked about the GET and POST methods already.
In HTTP there are two kinds of requests. The "simple" request consists of the method name (GET), a URL, and the HTTP version. The "full" request contains a series of headers and entity body (containing the POST data) in addition to what the "simple" request contains. These headers contain a variety of different information. For example, the User-Agent header specifies the type of browser and its version. Some sites check this field and present different pages based on the browser type. In our case, we're interested in the Authorization header. This is where the username and password information is placed for authentication purposes. A browser usually pops up a window asking for this information. In our case, where out interaction with the Web server is programmatic, we must provide this information by filling in the appropriate header fields.
The simplest authentication scheme supported by HTTP is a basic authentication. The Web server sends a challenge to the browser and the browser must then supply the necessary information, namely the username and the password. The line sent to the Web server is:
Authorization: Basic base64(username:password)Base64 is a well-documented algorithm. There are many implementations of it on the net (e.g. http://www.innovation.ch/java/HTTPClient). If the username is "jones" and the password is "jones123," you will form the string:
jones:jones123 and pass it through the base64 conversion.
The question is how to change the HTTP headers. The URLConnection class has a method named setRequestProperty which takes two parameters: a header field name, and a value. In our case, the header field name is "Authorization," and the field value is the base64 conversion of the username:password string. The following two statements do the trick:
connection.setRequestProperty(
"Authorization", "Basic base64(username:password)");
connection.setAllowUserInteraction ( true );You should place these statements after instantiating the URLConnection class and before you begin any communication. Now when the request is sent, the authentication information is sent with it, so the Web server can do the checking and allow you (i.e. your program) to view the protected page. You would process the page as you would with any other page.
This last section describes an issue that is more pertinent to enterprise users. Depending on your corporate network setup, you may need to go through a proxy server to access a Web site external to your network. Under these circumstances, it may be necessary for your monitoring tool to work with a proxy server.
We don't have a clear-cut solution to this problem. Getting Java and firewalls to work can be tricky. We have found a series of undocumented API functions that allow access to a URL via a proxy server. These were tested on JDK 1.1.x and proved to work properly. However, they may or may not work under other implementations of JVM.
Before you instantiate a URL, you need to add the following three statements to your program:
System.getProperties().put( "proxySet", "true" );
System.getProperties().put( "proxyHost", proxy name );
System.getProperties().put( "proxyPort", port );The first statement indicates that a proxy must be used for URL retrieval. The second specifies the proxy server, and the third indicates the port number used by the proxy server. These settings are similar to what you would do in your browser for proxy communication.
This article has demonstrated simple techniques that can be used to create an effective Web site monitoring system. We have used the capabilities of Java to interact with HTTP, thus simulating a browser. We have shown how to send and receive data to a Web server via simple and full HTTP requests. Additionally, we covered special cases where access requires authentication, or when traffic must go through a proxy server.
The skeleton technique can be incorporated to create more advanced and useful site monitoring tools. An applet can be programmed to periodically check a site and display the result graphically (e.g. green vs. red color). A timing block can be added so that we not only measure whether a site works, but how long it takes it to complete a transaction. Threads can be used to check multiple sites simultaneously. Rather than printing a success/failure message, we could send an e-mail to the person responsible for the site.
As you can see, there are many possibilities for expanding the basic application. You will have to determine which one works best for your needs. Happy coding!
Piroz Mohseni is an Internet consultant specializing in enterprise-wide Java and database integration. His latest book is JavaBeans Developer's Guide from MIS Press. He can be reached at mohseni@iname.com.