Writing good unit tests, Part 2: Follow your nose

Assertions, exceptions, and refactoring in test code

1 2 3 4 5 Page 3
Page 3 of 5

Exception testing -- the tester's notebook

When writing unit tests for a method (or a constructor) it's important that you cover all boundary cases and possible inputs. Some inputs will cause the method to throw an exception, and you'll want to verify that output. There are two types of exceptions: checked and unchecked. Both are of interest when it comes to properly unit testing a method.

Checked exceptions are for expected errors and should be dealt with programmatically. Unchecked exceptions are mainly for programming errors, and are derived from the JDK RuntimeException class. Such exceptions should be allowed to brachiate all the way up the class hierarchy and terminate the test, because they usually indicate situations from which the program is not meant to recover. However, some RuntimeException subclasses, like IllegalArgumentException or NullPointerException, may indicate errors in method parameters handed over to the callee. Of course, if such exceptions are part of the API contract, they have to be checked with a unit test.

In the next sections I'll highlight some of the idiosyncrasies of exception testing with JUnit 3.8.x, JUnit 4, and TestNG, and also offer tips for resolving exceptions and failures in callback methods, based on JUnit 4.

Exception testing with JUnit 3.8.x

If you are still using good old JUnit 3.8.x, or even previous versions, you might notice that the framework acknowledges the difference between test failures and errors. Failures are the result of failed programmed assertions. If, however, your tested objects throw an exception that is ultimately caught by the JUnit framework itself, it is reported as a test error. (JUnit 4.x has dropped the distinction between test failures and test errors -- a step backward, in my opinion.) A test error means that the test did not run to successful completion -- that is, the set of assertions that the test tried to make was not even executed. Listing 5 illustrates a common coding pattern for dealing with expected exceptions when testing with JUnit 3.8.x.

Listing 5. Exception testing pattern for use with JUnit 3.8.x

public void testNegativeAmount() {
   try {
      final double NEGATIVE_AMOUNT = -2.00;
      new Euro(NEGATIVE_AMOUNT);
      fail("Should have raised an IllegalArgumentException");
   } catch (IllegalArgumentException expected) {
      ; // expected (or sometimes even better: assertNotNull(expected.getMessage());)
   }
}

But you can do even better by taking an object-oriented approach, demonstrated in Listing 6. From a conceptual point of view, this is along the lines of Groovy's closures.

Listing 6. Exception testing with JUnit 3.8.x, the object-oriented way

import javaworld.junit.util.AssertExceptions;
...
/**
 * Here we use a custom assert statement to check if an expected exception is thrown and if 
 * the expected exception has a proper toString() format.
 * Throws an 'AssertionFailedError' if the exception is not thrown or if the
 * exception's toString() format is wrong. </p>
 * Use this assert method if you want your JUnit test to proceed if the expected exception is thrown. 
 * This method is more "object oriented" than writing again and again try/catch blocks in your test cases.
 *
 * Furthermore, by using this kind of assert you clearly signal your test case intention to the reader.
 */
public void testNegativeAmount() {
   final Block actionBlock = new AssertExceptions.Block() {
      public void execute() throws Exception {
         final double NEGATIVE_AMOUNT = -2.00;
         new Euro(NEGATIVE_AMOUNT);
      }
   };
   assertThrows(IllegalArgumentException.class, actionBlock);
}

Listing 6 contains a custom subclass of JUnit's TestCase, called AssertExceptions, that provides assert statements specific to exception handling. Using the assertThrows() methods contained therein, you can easily have more than one exception-checking assertion in your test without writing try/catch scenarios again and again. (The AssertExceptions class is provided in the source code package for this article.)

Exception testing with JUnit 4

JUnit 4, like TestNG, is based on annotations. The @Test annotation has an optional parameter, expected, that takes as values subclasses of Throwable. If you want to verify that your code under test throws the correct exception, you would use a pattern like the one shown in Listing 7.

Listing 7. Exception testing pattern used by JUnit 4

@Test(expected=IllegalArgumentException.class) public void checkForNegativeAmountException() {
   final double NEGATIVE_AMOUNT = -2.00;
   new Euro(NEGATIVE_AMOUNT);
}

Exception testing with TestNG

TestNG allows you to check for the occurrence of an exception in a way that's quite similar to the technique you'd use with JUnit 4. (Perhaps it's more fair to say it the other way around: if you look at both testing frameworks' evolution, TestNG was the trendsetter, so JUnit 4 may have borrowed from TestNG's way of doing things.) With TestNG, you use the @ExpectedExceptions annotation, as illustrated in Listing 8. This annotation specifies that the raising of an IllegalArgumentException is tolerated by the framework and therefore should not be considered a failure. In other words, your test will pass only if that particular type of exception is raised inside the test method.

Listing 8. Exception testing pattern used by TestNG

@ExpectedExceptions(IllegalArgumentException.class)
public void checkForNegativeAmountException() {
   final double NEGATIVE_AMOUNT = -2.00;
   new Euro(NEGATIVE_AMOUNT);
}

Note that @ExpectedExceptions takes a list of exceptions that a test method is expected to throw. If an exception is thrown that's not on this list, or no exception is thrown at all, this test will be marked a failure. This is different from JUnit 4, where you have exactly one exception class.

1 2 3 4 5 Page 3
Page 3 of 5