Divide and Conquer

The Assertion class is presented in Listing One. Note that a separate method has been provided for pre-condition, post-condition, and assert. As you can see, these methods are functionally equivalent, and have been provided for better readability. Each of these assert methods throws an Assertion.Exception, which is implemented as a RuntimeException. This, of course, implies that the programmer isn't forced by the compiler to explicitly catch them. The following code snippet illustrates typical usage of these assert methods:

public foo(SomeObject arg1)

{
  Assertion.preCondition(arg1 != null && arg1.IsSane());
  // Do something to my state...
  Assertion.postCondition(IsSane());
}

If you examine the Assertion class, you'll notice it's equipped with a static sHaltOnError flag. By toggling this flag true or false (with the static setHaltOnError method), you can cause the program to exit when an assertion fails, i.e. exit-on-error is the default behavior. As you will see in the unit test examples, you must disable the sHaltOnError flag when testing. This is because unit tests deliberately use improper input to verify that the code behaves as expected. You must circumvent the that would normally fire.

Finally, note that when debugging with (and the sHaltOnError flag set to true), it's easiest to always set a breakpoint just before the exit in the assert method. This can be a big time saver. It enables you to immediately walk the stack trace and examine variables before termination. Note that in my Assertion class, I call the Microsoft-specific method, com.ms.debug.Debugger.breakpoint, so I don't have to remember to set the breakpoint.

One of the common questions we want to ask an object is: "Are you in a sane state? Does all of your data make sense contextually?" This is such an important question that I require all objects to supply an answer. To this end, all objects must implement a standard interface named ISanityCheck, which requires the implementation of a single method; boolean IsSane. This simple programming convention enables you to easily test the sanity of any object, which helps you write bug free code:

public interface ISanityCheck
{
  public boolean IsSane();
}

When you implement IsSane, it should not knowingly halt execution, i.e. don't use within IsSane. Rather, it should simply return true or false. The client programmer may, of course, wrap the call to IsSane in an assertion:

Assertion.assert(IsSane);

Sanity checking can be very useful when debugging the release version, as well as when debugging the debug version. An example implementation of IsSane is as follows:

class WhatEver extends SomethingElse implements ISanityCheck
{
  private AnotherObject m_obj = null;

  // ... other code here ...

  public boolean IsSane()
  {
    if (! super.IsSane()) return false;
    if (! m_obj.IsSane()) return false;
    return true;
  }
}

While not a fundamental ingredient in preemptive testing, the "debug print" is a time-proven technique for instrumenting and debugging code. Anyone who has programmed for a while will have some notion of the debug print. To most of us, the debug print are those lines of code we instinctively place in our code to verify that it is indeed doing what we expect.

I'm not simply dropping in a print statement, however. Instead, I use a qualified print statement that can be flipped on and off programmatically. In C and C++ this is typically done with a macro. In Java, you use the DebugPrinter class, shown in Listing Two.

People either love debug prints, or hate them. The latter generally believe that debug prints clutter the code. These people tend to use a debugger to verify their code, and while this should be standard practice, it is not sufficient in the long term. You can leave debug prints in your code forever, which is an advantage. When you revisit some piece of code (as you will surely have to), you simply flip the debug prints back on to instantly verify that code. Thus, debug prints are another huge time saver.

But more significant to the subject of this article, debug prints are also quite useful for regression testing, i.e. they provide us with an audit trail. We can then programmatically monitor any differences between the current output and the validated copy of the output.

As with , when the debug print is turned off, the pure Java version would result in, at the least, a null function call. For performance reasons, it would be hard to find this acceptable. As mentioned earlier, I use Visual J++, and thus can use conditional methods and successfully work around this problem.

Anyone who has worked on a large software project knows that it is an extremely complex endeavor. So complex in fact, that seasoned developers rely on standardized processes to bring it under control; the most powerful of which is incremental testing. With incremental testing, each piece or unit of a program is first tested separately, by unit testing. Then, incrementally, these units are tested together (i.e. integration testing), and finally, cascading outward, the system is tested as a whole (i.e. system testing).

Incremental testing makes it easy to pin down the source of an error. When you test only a single unit, any errors are either in that unit, in the Interface Specification, or in the test code itself. You do not have to look through much code to find the problem. If a new problem shows up when unit-tested modules are tested together, the error is almost certainly in the interface between them. Incremental testing allows you to focus on each unit individually, which generally yields better test coverage.

In an object-oriented language like Java, the smallest, fundamental, contextual unit of code is a class (methods, public data, and constants all require a class to give them context). Thus, in Java, unit testing implies the isolated testing of a single class. It is nothing more than thoroughly testing every public method in a class, the verification of the public interface for that given class.

Unit testing is the process of verifying that the code indeed does what it is reported to do as defined by the Interface Specification. The unit test verifies that a particular class accurately fulfills the contract it has made with the outside world.

When you encounter an error while unit testing, you must establish the true source of the error. This involves reconciling three possible sources of the error: the source code, the written specification, and the test code itself. It isn't hard to see why debugging unit tests can be a complicated, artistic exercise.

Unit testing falls in the realm of glass box testing, which implies you must have an intimate knowledge of the code being tested. You must be aware of:

• All branches of the method under test.
• What inputs should produce what results or exceptions.
• The interplay of affected member variables and arguments.

For this reason, unit tests are best written by the author of the class.

I use a standard programming convention for all unit tests. An inner class is employed to provide a UnitTester for its enclosing object. This is illustrated by the following (incomplete) code snippet:

class Whatever
{
  public void foo()
  {
    // Do something.
  }

  // -----------------------------
  //     Unit Test Support
  // -----------------------------
  public static void main(String[] args)
  {
    // Construct the output PrintWriter.
    unitTest(outFile);
  }

  public static void unitTest(PrintWriter outFile)
  {
    // Note the peculiar syntax for creating our inner UnitTester.
    Whatever outerObj = new Whatever();
    Whatever.UnitTester unitTester = outerObj.new UnitTester();
    unitTester.test(outFile);
  }

  public class UnitTester
  {
    public void test(PrintWriter outFile)
    {
      // Call individual Unit Tests from here.
      try
      {
        foo_test(outFile);
      }
      // Catch any Exceptions not handled, or  thrown,
      // by the Unit Tests. Any Execption that bubbles up to here
      // represents failure.
      catch (Exception ex)
      {
        // Do something with Exceptions here.
      }
    }
    public void foo_test(PrintWriter outFile)
    {
      // Individual Unit test for method foo.
    }
  }
}

Using an inner class for unit testing makes sense for several reasons:

• Since the compiler will generate a separate .class file for the inner class, you don't bloat the bytecode of the enclosing class with the baggage of the unit test code. This is very helpful, particularly because you may have to transport .class files over-the-wire. The unit test .class files can be easily ignored at deployment
• The test code lives with the code it is testing. This provides a convenient usage template. More importantly, the test code cannot easily go out of sync with the code it is testing
• An inner class has access to the private variables of its enclosing class. This provides an opportunity for more thorough validation. Unit testing is, by definition, glass box testing, so having read access to private variables is quite acceptable.
• Using an inner class significantly improves readability, and reduces the explosion of files into your code base.

Note from the code snippet that the class' unit test is publicly accessed using the static unitTest method. Additional public access is provided using the main method, which gives us the ability to execute each unit test separately from the command line. Use of a static unitTest method allows us to specifically encapsulate testing. (It would be presumptuous to assume the main method will only be used for unit testing.)

If you are going to write a good unit test for a class, you should write a test for every public and protected method in that class. (Testing for all other class levels is discretionary.) These tests may be elaborate or simple; it depends completely on what you are testing. You are simply required to fully test each method. You must test for all expected and exceptional behavior.

Here are a couple of good rules of thumb:

• The easiest way to manage your unit tests is to code the corresponding test as a separate method for each public method in the enclosing class.
• If you place all of your test code at the bottom of the enclosing class along with a separating delimiter, you will not need to wade through the test code when looking for something pertinent in the class.

If you want your unit test to be complete, you should generate your test input from two places: the Interface Specification (if you have one) and the code itself. Using the Interface Specification as an input guide ensures that the code does what it's supposed to do. But the Interface Specification may not tell the whole story. You need to examine the code itself to check for hidden ways to break the code, and to ensure that all branches have been tested.

One of the critical tasks you face when writing your unit tests is to reduce the set of all possible input values (a semi-infinite number) to some reasonable set, while still providing the necessary rigor. This is the "art of testing." To do this effectively you must eliminate most redundant tests. A test is redundant if it uses an input value that's expected to produce the same response as an input value that has already been tested. In other words, the method is expected to produce equivalent behavior for either test case.

Input values that are expected to produce equivalent results are called an equivalence class. At a minimum, at least one test should be run for every equivalence class. The basic equivalence classes for each input value can be derived from its data type. If the input data type is an object, there are always at least two equivalence classes: null and not null. For numbers, there are negatives, positives, and zero. Furthermore, if the method expects particular input values, say, a particular text string, then each of these values becomes an equivalence class in its own right.

Once you have determined your equivalence classes, you must next choose representative values for each class. As we all know, given a range of values, errors tend to occur most often near the edges. Boundary analysis is the term for choosing the appropriate values with which to test a particular equivalence class. For numbers, the edge values are the minimum and maximum values the number can represent. In addition, zero should always be chosen, as it is often problematic. It's also a good idea to surround known hard values. For example you may want to surround zero with -1 and 1. For strings, always include at least the empty string.

When writing a unit test, one of your principle concerns is to provide a comprehensive set of input values for the method you are testing. This includes both valid and invalid values. It is important to verify both the method's behavior for invalid values, as well as for values you expect. You must establish that the method fails only in an expected fashion, i.e. that you do not experience any unplanned exceptions.

To establish this, you must know what exceptions a method will throw, and what inputs will cause those exceptions. Your job is to match the exception you receive with the conditions you expect to produce it. For example, a method will typically throw Assertion.Exceptions for several specific reasons. In essence, you are attempting to filter out acceptable exceptions, searching for those exceptions that you cannot explain. Obviously, an unexplained exception indicates a bug.

The algorithm for a basic unit test consists of the following steps (outlined in pseudo code):

Generate argument providers
Loops over all argument providers
  Construct the object under test (OUT)
  Optionally save the state of the OUT
  Call the method under test (MUT)
  If an exception was generated
    Test for unhandled exceptions
  Else
     on the resultant, OUT, and arguments
End loops

Instead of explaining in painstaking detail the unit test process, I have provided an example.

The example shown in Listing Three is a contrived class representing a dog. This dog can eat, be fed, and bark (but it cannot bark while it eats). The code in this class illustrates the preemptive testing concepts detailed above, including , sanity checking, and debug prints, as well as a complete unit test.

I want to highlight several points from the example. First, notice that the test code is far longer than the code it is testing. This is very common. A successful software project will often end up with twice as much (or more) test code as production code. Consider this a badge of good software engineering.

Second, notice that every method's unit test takes a PrintWriter as an argument, and then, minimally, prints a "success" message to it before returning. This is a good practice to follow. It provides a minimal written record of the test. Without such a record, we could not reasonably run the test as a batch job. How would we know the job ran successfully? Or, conversely, how would we know approximately where the software failed? This also applies to regression testing.

Third, examine the eat_test method. It illustrates pure glass-box testing. For the eat method, we can see (by reading the code) that strange things might happen (like an OutOfFoodException) when we mix different combinations of calls to feed with calls to eat. Consequently, this test code is set up to force those situations to occur.

The simple truth is the earlier you catch a bug, the less it's going to cost you. Consequently, you get the biggest bang for your testing buck during the initial stages of development. Once you have committed your design to code, your first line of defense against bugs is preemptive testing. This is the art of bulletproofing your code. Preemptive testing is more of an approach to writing code than a rigid testing procedure. Your second line of defense against bugs is unit testing. Unit testing tests each class individually, verifying that every public method behaves as expected under both valid and invalid input.

Chris Berry has been building large-scale software systems since the early 1980s. He specializes in Metadata-based frameworks and in distributed systems. Chris is currently a software architect at Works.com, a high-volume e-commerce site in Austin, TX. He is also a Microsoft Java/COM MVP. Chris can be reached at cberry@works.com.