Writing good unit tests, Part 1: Follow your GUTs

Best practices and tools for high-quality test code

1 2 3 4 5 Page 3
Page 3 of 5

Code quality with style

How can you make sure that you're writing high-quality test code? The following style considerations should be at the top of your mind when you're writing your tests.

Boundary conditions

You should test every method of your public API with appropriate boundary conditions (excluding trivial getter/setter methods if they really do no more than just provide or set a value). Here, an IDE can help you reduce coding effort by automatically generating JUnit or TestNG templates for every class under test. Do not just test the happy path, but write tests for invalid domain data and boundary conditions too. For instance, you should provide:

  • Null values to object parameters
  • Empty strings or very long strings
  • Special characters in certain languages, like the German characters ö, ä, ü, and ß, in strings
  • Special non-alphanumeric characters, like !, %, &, $, and ?, in strings
  • Empty collections, collections with exactly one element, or collections with the maximum number of elements
  • Invalid numbers and boundary numbers for numeric int parameters, such as Integer.MAX_VALUE, Integer.MIN_VALUE, and 0
  • Specific dates

Unfortunately, I cannot provide a best practice method for testing objects with a lot of different internal states -- for example, where state is represented as six member variables, two of which themselves contain collections of objects. If you've uncovered a good technique for such a scenario, please let me know.

Risk-based testing techniques

You should make use of risk-based testing techniques to reduce the infinite number of possible tests and to avoid testing trivial code. Your unit tests should test those parts of the code that are likely to have defects; try to imagine how the program could fail, and then try to get it to fail that way. Be sure to think about both where defects might lie in newly developed code, and about where they might arise when the code is changed (see "How to write good unit tests" by Basil Vandegriend for more about this). Risk-based testing also addresses parts of the system that are used often.

Exploratory testing

Make use of exploratory testing techniques to simultaneously process test design and test execution. As testing progresses, you learn more and more about the behavior of the code. With experience and creativity, you can craft more and better tests.

According to Cem Kaner, who coined the term in the book Testing Computer Software (Wiley, 1999), exploratory testing is "a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." In the context under discussion here, that might characterize a methodology where the user writes parameterized unit tests (with TestNG, for example) and lets an automated tool generate parameter values to cover all reachable statements as a kind of exploratory code analysis. (For more on this, see the article "Exploratory test-driven development.")

Scenario testing

You should write scenario tests that tell the reader something about the behavior of the system under test, or SUT. Such tests don't look at functions in isolation, but at the system under test (a class, in the case of unit tests) as a whole. You should collect user stories and then test them using a combination of the public API's methods.

Tests as documentation

Consider leveraging your unit tests so that they even serve as API documentation, as proposed by Brian Button in his article "Double duty." He proposes focusing more on simplicity and naming. You should look at your tests in a different way, making sure that the tests tell the story of your class or subsystem -- and that they tell the whole story. Then the process of creating your code through TDD is the process of creating the documentation too, and evolving the code also evolves the documentation. Button calls this agile documentation.

Reasonable test coverage

Try to create your tests and test data to produce reasonable test coverage for your SUT's code. I will not postulate an absolute value here; every project has to define its own goals. But keep in mind the wise advice that Lidor Wyssocky gives us in his blog post entitled "The illusion of high test coverage": "Test coverage data helps developers identify missing test cases. It also helps development managers to get a clearer picture of the functional quality of the code. However, the test code coverage metric can also create an illusion. The problem is that this number is always related to what's in the code. It can never tell you what is missing from the code. Unfortunately, many bugs are the result of unwritten code."

Andrew Binstock makes that same point in a blog post called "The fallacy of 100% code coverage," when he tells us, "If developers attain 100% code coverage -- even at the cost of writing meaningless tests -- they can be certain they haven't forgotten to test some crucial code. This viewpoint is the real illusion. By basing proper testing on 100% code coverage, the developer has confused two issues. It's what you're testing and how (so, quality) that determines code quality, not numerical coverage targets (quantity)." Matt Harrah takes the same line in a blog post called "How do you know if your tests are good?":

How do you measure the quality of your JUnit tests and put a number on it? This is a very good question with no quick and easy answers, I'm afraid. I can tell you two answers that are not measures of test quality:
  • We have 100% success rate in our test suites every night.
  • We have 100% code coverage.

Harrah is correct.

Tools for measuring test coverage

The table below offers an overview of tools that can help you with test coverage data collection and reporting. You can find links to all of these tools in the Resources section at the end of this article.

Table 1. A selected list of test coverage tools

ToolLicense typeDescription
CloverCommercial, but Clover is free for any open source project to use.

Clover measures code coverage generated by system tests, functional tests, or unit tests. Provides integrated plugins for IntelliJ IDEA 4.x and 5.x, NetBeans, Eclipse, JBuilder, and JDeveloper. Can be integrated into Ant or Maven 2. Uses source code instrumentation. Supports all JDK 1.5 language features.

Types of coverage measured: Statement, branch, method, and total coverage percentage for each class, file, and package, and for the project as a whole. Clover is a pure Java application and should run on any platform that has at least JDK 1.2 installed.

CoberturaOpen source

Cobertura is a free Java tool that calculates the percentage of code accessed by tests. It can be used to identify the parts of your Java program that are lacking test coverage. It is based on jcoverage. Cobertura modifies your Java bytecode slightly by adding bytecode instrumentation. It shows the percentage of lines and branches covered for each class, each package, and for the overall project.

It also shows McCabe cyclomatic code complexity of each class, and the average cyclomatic code complexity for each package and for the overall product. Ant and Maven integration is supported. Cobertura is platform independent.

EMMAOpen sourceSupported coverage types: class, method, line, and basic block. EMMA can even detect when a single source code line is covered only partially. Uses bytecode instrumentation. Can instrument all classes in a JAR file with one command. Ant integration is supported. EMMA is platform independent and works in any Java 2 JVM since 1.2.
CodeCoverEclipse Public License (EPL)CodeCover is a free glass box testing tool developed in 2007 at the University of Stuttgart in Germany. It measures statement, branch, loop, and MC/DC coverage. CodeCover runs in command line mode on Linux, Windows, and Mac OS X, and provides Eclipse and Ant integration.
1 2 3 4 5 Page 3
Page 3 of 5