TestNG: The next generation of unit testing

Leverage TestNG for unit, synchronous, asynchronous, and parallel testing

TestNG, written by Cedric Beust and Alexandru Popescu, is a light framework based on Java annotations (for J2SE 5.0) that allows you to design complex unit testing for J2SE 5.0 and J2SE 1.4. Why bother learning another unit-testing framework when you're already comfortable using JUnit? If you are interested in simplifying your unit-test cases, in leveraging J2SE 5.0 annotations to tag your test classes as well as being backward compatible with J2SE 1.4, in having out-of-the-box support for dependent methods and parallel and asynchronous testing, TestNG is the tool you are looking for.

This article starts with a description of the JUnit annoyances and introduces TestNG through numerous examples. A case study shows how to use TestNG for asynchronous testing.

JUnit annoyances

The JUnit testing framework has been around for a long time and is (hopefully) widely used by Java developers to unit test developed software. However, the framework has some annoying specificities, which the following sections describe.

Multiple instantiations per TestCase

Multiple TestCase instantiations is a well-known issue. The setup() and teardown() methods are called before and after each test method; moreover, the test class that must extend TestCase is also instantiated each time a test method is called. Although multiple TestCase instantiations might prove acceptable for simple test cases, what if you want to set up an object that is to be reused across more than one test method, for example, a Java Database Connectivity connection (or, for that matter, a Java Naming and Directory Interface (JNDI) context)?

PainOneTest.java illustrates the problem:

 

public class PainOneTest extends TestCase { /** * PainOneTest */ public PainOneTest() { System.out.println("PainOneTest"); }

/** * @see TestCase#setUp() */ protected void setUp() throws Exception { System.out.println("setUp()"); }

/** * @see TestCase#tearDown() */ protected void tearDown() throws Exception { System.out.println("tearDown()"); }

/** * testMe */ public void testMe() { System.out.println("testMe"); }

/** * testYou */ public void testYou() { System.out.println("testYou"); } }

This code produces the following output:

 PainOneTest
PainOneTest
setUp()
testMe
tearDown()
setUp()
testYou
tearDown()

The TestSetup class circumvents this issue, as illustrated below:

 

public class PainTwoTest extends TestCase { /** * Test * @return */ public static Test suite() { return new TestSetup(new TestSuite(PainTwoTest.class)) { public void setUp() throws Exception { System.out.println("setUp()"); }

public void tearDown() throws Exception { System.out.println("tearDown()"); } }; }

/** * PainOneTest */ public PainTwoTest() { System.out.println("PainOneTest"); }

/** * testMe */ public void testMe() { System.out.println("testMe"); }

/** * testYou */ public void testYou() { System.out.println("testYou"); }

/** * main * @param _ */ public static void main(String []_) { Test test = PainTwoTest.suite(); } }

This code produces:

 PainOneTest
PainOneTest
setUp()
testMe
testYou
tearDown()

The above solution introduces a static method, and all variables accessed by it must also be declared static. Although I have nothing in particular against static methods, you must take care when implementing test methods for concurrency access (sharing static objects) and static initialization (occurring only once per VM).

Remember, when using Ant, the default behavior is to use the VM that started Ant to execute all the tasks; in the above scenario, you may well have to use the fork attribute of the Java core task.

Furthermore, multiple instantiations of the TestCase class remain. Surely, if we want the test methods to be independent of each other, we would place them into different TestCase classes. So why multiple instantiations? Multiple instantiations allow you to isolate independent methods within the same TestCase class, which is not always the desired behavior. Finally, too much technical code is required for circumventing the multiple-instantiations scenario: I just want a test POJO (plain old Java object) class.

How does TestNG handle multiple instantiations?

TestNG does not require static block initialization and has a flexible configuration scheme for handling test classes based on regular expressions and XML configuration files. TestNG does not instantiate the test class several times.

Multithreaded support

GroboUtils is an addition to JUnit that supports multithreaded unit tests (since the JUnit framework lacks such support). N. Alex Rupp gives an overview of GroboUtils in "Multithreaded Tests with JUnit" (java.net, August 2003). Although very useful, GroboUtils is not to my taste as it requires too much glue code to handle multithreaded unit tests.

How does TestNG handle multithreaded unit tests?

TestNG has multithreaded and parallel unit tests built in its core. You don't need to write specific code to handle multithreaded unit tests as they are just a configuration of TestNG.

Why should I extend a specific class and prefix the methods with test?

I would rather tag any method than be obliged to prefix methods with test, a task that JUnit requires. Admittedly, this is a minor point, but all IDEs now give you a view (like the Eclipse outline view) that helps you quickly browse your methods to find the one you want—that is, if they are not all prefixed with the same four letters. From a practical viewpoint, the Eclipse outline view is useless when you have dozens of test methods on a class that start with the same four letters.

How does TestNG handle test method names?

TestNG uses Java annotations as specified by Java Specification Request 175 to tag methods for testing and grouping, and for expected exceptions, and so on.

Introducing TestNG

TestNG is inspired by JUnit and also NUnit a unit-testing framework for .Net.

TestNG introduces new functionalities to unit testing such as:

  • Support for Java annotations (if unfamiliar with annotations, see sidebar "What Are Annotations")
  • XML configuration file for test configuration
  • No required class extension or interface implementation
  • Support for dependent methods and groups
  • Support for parallel testing
  • Parameters for test methods
  • Arbitrary number of invocations plus success rate

Step by step: Your first TestNG program

Let's get started with TestNG. The remaining sections of this article introduce you to the TestNG features, starting with a basic first unit test: how to configure it and run it.

Here is a simple test class:

 

package org.jyperion.testng;

import com.beust.testng.annotations.*;

public class FirstTest { @Configuration(beforeTestClass = true) public void configure() { System.out.println("configure"); }

@Test(groups = {"exec-group"})

public void exec() { System.out.println("exec");

assert 1 == 2;

} }

Note that no interface or class is required for your test class. The @Configuration annotation is the equivalent of the JUnit method setup(); you can annotate any method in your test class. @Configuration(beforeTestClass = true) means that the annotated method will be called just once before testing any method in the class. The @Test annotation is equivalent to the JUnit methods prefixed with the word test.

Next comes the XML configuration test called testng.xml:

 

<!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" > <suite name="My First TestNG test"> <test name="Hello Test!"> <classes> <class name="org.jyperion.testng.FirstTest" /> </classes> </test>

</suite>

Finally, TestNG's invocation:

 java –ea com.beust.testng.TestNG testng.xml

Note: -ea enables assertions.

The following appears on the console:

 

configure exec [TestRunner] FAILED: org.jyperion.testng.FirstTest.exec [TestRunner] REASON: java.lang.AssertionError

=============================================== TestNG Samples Total tests run: 1, Failures: 1 Skips: 0 ===============================================

An HTML report is also created in a folder called test-output:

Figure 1. Your first TestNG HTML report. Click on thumbnail to view full-sized image.

An interesting feature of TestNG is the test setup mechanism enabled by using the @Configuration annotation in four different scenarios:

  • Run configuration method once before all tests
  • Run configuration method once before every test
  • Run configuration method once after every test
  • Run configuration method once after all tests

Moreover, if you use TestNG's powerful grouping feature (described in subsequent sections), the @Configuration methods can have a determined execution order.

TestNG DTD view

The configuration is defined by a DTD (document type definition). Figure 2 shows the schematic view.

Figure 2. TestNG DTD. Click on thumbnail to view full-sized image.

The test annotation

The @Test annotation is defined as:

 @Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({java.lang.annotation.ElementType.METHOD, 
         java.lang.annotation.ElementType.TYPE})
public @interface Test {
  public String[] groups() default {};
  public boolean enabled() default true;
  public String[] parameters() default {};
  public String[] dependsOnGroups() default {};
  public long timeOut() default 0;
}

The groups element allows you to group methods and groups. By configuring testng.xml, you can include or exclude groups using regular expressions:

 

public class GroupTest { @Test(groups = {"group-one"}) public void init() { System.out.println("init#group-one"); }

@Test(groups = {"group-two"}, dependsOnGroups = {"group-one"}) public void exec() { System.out.println("exec#group-two"); }

@Test(groups = {"group-three"})

public void testMe() { System.out.println("testMe#group-three"); } }

The corresponding testng.xml:

 <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
  
<suite name="My First TestNG test">
  <test name="Group Test">
    <groups>
      <run>
        <include name="group-.*" />
      </run>        
    </groups>
  
    <classes>
      <class name="org.jyperion.testng.GroupTest" />
    </classes>
  </test>
</suite>

Parameters

Parameters defined in testng.xml can be passed to the test methods at runtime:

 public class ParameterTest {
  @Test(parameters = { "jndi-name" })
  public void lookup(String jndiName) {
    System.out.println("lookup("+jndiName+")");
  }
}

The corresponding testng.xml:

 <!DOCTYPE suite SYSTEM "http://beust.com/testng/testng-1.0.dtd" >
  
<suite name="My First TestNG test">
  <test name="Parameter Test">
    <classes>
      <class name="org.jyperion.testng.ParameterTest">
        <parameter name="jndi-name" value="java:comp/env/ejb/ECCSB"/>
      </class>
    </classes>
  </test>
</suite>

Note that parameters can be placed within the suite element, within the test element, or within the class element, so that your parameter is available to all the classes in the suite, to all the classes in the test, or to only the class.

The corresponding testng.xml:

 

<suite name="TestNG Samples" verbose="1" parallel="true" thread-count="10">

<test name="ParallelTest"> <classes> <class name="org.jyperion.testng.ParallelTest" /> </classes> </test> </suite>

The element parallel is set to true to instruct TestNG to run the tests in parallel. thread-count specifies the number of threads in the pool (if omitted, default is 5).

The HTML report in Figure 3 shows that the methods run chronologically and also presents the thread identifier used to execute a test method.

Figure 3. TestNG parallel report. Click on thumbnail to view full-sized image.

Figure 4 shows the threads associated with the test methods.

Figure 4. TestNG parallel report with thread information. Click on thumbnail to view full-sized image.

TestNG allows two running scenarios:

  • Sequential run of tests (default)
  • Parallel run of tests

To select which scenario to use, simply edit the testng.xml.

Another nice feature of TestNG is its support of timeouts, which can be used in parallel and nonparallel modes using the timeOut attribute in @Test.

Ant task

TestNG is also shipped with an Ant task:

 

<testng

fork="yes" classpath="${my.classpath}" outputDir = "${basedir}/test-output">

<propertyfile value="${basedir}/testng.xml"/> <jvmarg value="-ea" /> <testng>

Using this Ant task (which is part of the TestNG distribution), TestNG could be used for continuous integration, for example.

1 2 Page 1
Page 1 of 2