Integrating Arquillian and JBehave

Three Arquillian extensions for behavior-driven development in the container

From an agile perspective, integrating Arquillian and JBehave makes good sense, which is why a number of developers have already written open source extensions that bridge the two frameworks. Get introduced to three such extensions (one of them written by the author) and see how they can incrementally improve the experience of behavior-driven development with Arquillian. Who knows, one of them could be the missing piece you've been looking for -- or the foundation for your next open source project.

Arquillian is a Java EE integration testing framework that executes tests in the container, testing not just your core application but its interaction with business components. While Arquillian makes writing functional tests much easier, I found it lacking in support for behavior-driven development (BDD). Specifically, I wanted to create a bridge between Arquillian and JBehave, the popular BDD testing framework.

I wrote the Arquillian-JBehave extension to create a strong integration between Arquillian and JBehave. The extension is fast -- it cuts some corners to get around Arquillian's standard behavior of deploying once per test class -- and it provides detailed, BDD-style feedback about what happens at every step of the testing scenario and execution.

In this article I briefly introduce both Arquillian and JBehave and discuss the advantages of extending Arquillian to support behavior-driven development. I then introduce two of the existing integration solutions -- Arquillian JBehave TestRunner and JBehave JUnit Runner -- and explain how my extension builds on them. Finally, I offer a demonstration of my Arquillian-JBehave integration project. All three projects discussed in this article are open source and hosted on GitHub.

Integrating Arquillian & JBehave

Arquillian is an integration, functional, and acceptance testing framework that was founded as a JBoss community project. Based on the JBoss Test Harness, Arquillian breaks the golden rule of unit testing (no dependencies) by throwing out mock objects and testing directly in the container. Injecting CDI beans, EJBs, and other container resources directly into test classes and driving tests in the browser enables a richer, more informative, and more user-friendly testing experience. Arquillian is also container agnostic, integrates well with most testing frameworks and IDEs, and was designed to be an extensible platform. Demonstrations in this article were developed using Arquillian as a test runner inside of JUnit 4.

Listing 1 shows a very basic test in Arquillian

Listing 1. A test class in Arquillian


@RunWith(Arquillian.class)
public class GreeterTest {

    @Deployment
    public static WebArchive createDeployment() {
        return ShrinkWrap.create(JavaArchive.class)
            .addClass(MyWebService.class)
    }

    @Test
    public void myWebServiceTest() {
        Assert.fail("Not yet implemented");
    }
}

The two key elements in the above code are the @RunWith and @Deployment annotations. @RunWith tells JUnit to run the test class using Arquillian's own runner, while @Deployment marks which methods should be considered as deployment producers. A deployment is represented by a single Java archive or a multi-module enterprise archive and is the subject of the test.

JBehave is a popular framework for doing behavior-driven development in Java. Similar to test-driven development, BDD is iterative. But rather than running your code through the wringer, BDD focuses on application scenarios and behavior. In BDD, application testing means writing application requirements in executable form. These requirements are comprised of stories, each with one or more scenarios. To describe BDD scenarios, JBehave supports Gherkin, a powerful, simple (in the sense that it uses a small number of keywords) and platform-independent business readable DSL.

JBehave leverages unit testing frameworks like JUnit and TestNG to connect its scenarios to the test code. Listings 2 and Listing 3 show a very basic JBehave test class and its associated story.

Listing 2. A JBehave-JUnit test class


public class CalculatorStory extends JUnitStory {

    private int x, y;

    @Override
    public InjectableStepsFactory stepsFactory() {
        return new InstanceStepsFactory(configuration(), this);
    }

    @Given("X is $X and Y is $Y")
    public void givenMethod(@Named("X") int x, @Named("Y") int y) {
    	this.x = x;
    	this.y = y;
    }

    @When("I add X to Y")
    public void add() {
    }

    @Then("I get $R as result")
    public void testAddition(@Named("R") int r) {
        Assert.assertEquals(r, x + y);
    }
}

Listing 3. The associated story file


Scenario: Addition
Given X is 5 and Y is 3
When I add X to Y
Then I get 8 as result

Listing 2 shows the most basic JBehave test possible. It is compact because JBehave leverages the convention over configuration principle. The key elements in this example are the @Given, @When, and @Then annotations, along with the stepsFactory method. Note that the annotations' values match the text in each of the corresponding steps from the scenario in Listing 3, thus linking the test code and the scenario. The stepsFactory method tells JBehave in which classes to look for @Given, @When, and @Then annotated methods. (I supplied them in the same class for brevity but it would be more usual for them to exist in separate classes, also called steps candidate classes.)

JBehave's implementation of BDD concepts can add much to Arquillian's functional and integration testing framework, extending it to be a more business-driven testing environment. Bridging the two frameworks would enable developers to do BDD inside the container, without the mocking and test isolation specific to unit testing.

Arquillian-JBehave integration solutions and drawbacks

Arquillian has generated considerable interest since it was launched, and many extensions have been proposed or created. Several extend Arquillian to support BDD with JBehave, but none went as far as I needed it to in terms of scalability and granularity.

I'll go through my list of features for a successful Arquillian-JBehave integration, then point out which ones are currently supported and which are not. I'lll also briefly introduce the two extensions that I used as a starting point for my own project.

Feature 1: Executing and reporting scenarios in the container

The most basic feature of an Arquillian-JBehave integration would be the ability to execute JBehave scenarios inside the container. Arquillian JBehave TestRunner supports this functionality. The downside of TestRunner is that it requires that you run the whole JBehave story (or set of stories) inside a single @Test annotated method. The resulting JUnit report contains information only about the top-level test method: you can't drill down to a more granular level to view how individual scenarios and steps performed.

One approach to resolving this issue would be to enhance JBehave's JUnit runner with a better reporter, perhaps one that was linked with JUnit's RunNotifier. The JBehave JUnit Runner project does just that, generating a JUnit run report with a step-level granularity.

Combined, these two open source integration projects execute stories inside the container while reporting on the test's progress; but they still leave a few things to be desired.

For a start, let's consider a test scenario that would leverage Arquillian's persistence extension, which uses dbUnit behind the scenes. In order to test the search functionality of some library software, we would need to pre-populate the database with books, but not without cleaning it up first to ensure a clean test bed. We would need to do all this before executing the @Given step, which is always the first step in a BDD test scenario.

Listing 4 shows how such a test could look if Arquillian's persistence extension and JBehave were tightly integrated.

Listing 4. Arquillian's persistence extension in a JBehave step candidate


…
@Test
@UsingDataSet("datasets/books.xml")
@Given(“a set of books”)
@Cleanup(phase = TestExecutionPhase.BEFORE)
public void books() {
}
…

In this example the database cleanup is achieved via the @Cleanup(phase = TestExecutionPhase.BEFORE) while the population with test data is handled by the @UsingDataSet("datasets/books.xml"). But achieving this level of granularity (i.e., working at step level) is not possible with the existing framework and extensions because:

  1. Arquillian supports database-related annotations (like @Cleanup and @UsingDataSet) only on actual @Test methods; and
  2. the current integration between JBehave and Arquillian means providing a class with a single @Test annotated method that runs all the scenarios at once.

Annotating the only available @Test method instructs Arquillian to prepare the database as requested and then give control to JBehave to run the scenarios. It won't be able to intervene until all of the scenarios have been executed. So no matter how many scenarios we execute, the books inserted before will exist in the database until all of the tests have been executed, and not just for a particular one, which is what we require to properly test the library software.

Without scenario-level data isolation we can't properly test our software. This limitation affects not just the persistence extension but every other feature in the Arquillian framework that leverages before and after test events.

Feature 2: Running multiple stories

As a project grows, it's likely that we'll want to be able to run multiple stories at once. We could run each story separately but that approach leads to time wasted on packaging and deploying each story. JBehave does allow us to run multiple stories at once, but in order to work it must be able to detect the location of the story files, which is a more involved process inside of a container than out of it. An Arquillian-JBehave integration can't rely solely on JBehave's running framework to execute multiple stories at once because JBehave alone is not capable of locating the story files inside the container's virtual filesystem.

In order to test multiple stories, we would typically write multiple steps candidate classes. (Unless you like 3,000-line classes, that is!) In order to get down to the step level, in terms of granularity, and leverage Arquillian's before-and-after test events, Arquillian must be able to consider each steps candidate a test method, which would be annotated with @Test. Arquillian's big limitation in this regard is that it can only execute methods inside of a single class for each deployment. As a result, we have to redeploy after each test class (or JBehave steps candidate class). That isn't a big deal for a small number of stories, but it clearly isn't scalable.

The Arquillian-JBehave extension

In order to work around these issues I have developed a new Arquillian-JBehave integration that builds on the two projects that I mentioned earlier: Arquillian JBehave TestRunner and JBehave JUnit Runner. While my methods aren't quite models of the Gang of Four design patterns (for instance, I used class instrumentation to get around Arquillian's limitation of a single Java test class per deployment) the extension does what I need it to do. Specifically, here's what the arquillian-jbehave extension offers:

  • Arquillian annotations directly on step methods
  • Annotations to bind a test class to a story file
  • Ability to execute multiple stories with a single deployment

The limitation of the project is that steps candidate classes must be supplied to JBehave as static instances. Code cops will agree that this isn't perfect programming technique, but it doesn't limit the actual testing functionality. (See "A note about deployment" at the end of this article for an explanation.)

New Arquillian-JBehave integration demo

First I'll demonstrate how to run a JBehave story using the arquillian-jbehave extension, then I'll explain a few things about how it works.

Listing 5. Arquillian-JBehave integration demo


@RunWith(ArquillianJbehaveRunner.class)
@StoryFile("example")
public class ExampleStory extends AbstractArquillianStory {

	public static ExampleTestSteps steps = new ExampleTestSteps();

	@Deployment
	public static WebArchive createDeployment() {
		//create the WebArchive to deploy – skipped for brevity
	}

	@Override
	public InjectableStepsFactory stepsFactory() {
		return new ArquillianInstanceStepsFactory(this.configuration(), steps);
	}
}

The first thing to note about the example in Listing 5 is the @RunWith annotation, which features a new runner called ArquillianJbehaveRunner. This updated annotation is the main actor of the Arquillian-JBehave integration. It enables us to deploy multiple stories at once, and also gets down to the step level of granularity by leveraging before-and-after test events. The next annotation, @StoryFile, is a simple extension that tells JBehave the name of the associated story file, which is implied but can be configured. In this case, the annotation tells JBehave to look for a file named example.story in the classpath under the folder stories (the folder name and .story extension are default values but can be configured).

Next, the test class extends AbstractArquillianStory. This class just builds on JBehave's JUnitStory class which, as its name suggests, runs a story using JUnit. In combination with the stepsFactory method, AbstractArquillianStory leverages Arquillian JBehave TestRunner, providing the groundwork to enrich steps candidates, for instance by injecting EJBs directly into steps classes.

The steps are executed in Arquillian as distinct test methods thanks to the ArquillianJbehaveRunner from Listing 5, so annotations such as @UsingDataSet and @Cleanup can be used directly on the step methods.

Running multiple stories is just a little bit different than it was before.

Listing 6. Running multiple stories


@RunWith(ArquillianJbehaveRunner.class)
public class AllStories extends AbstractArquillianStories {

	public static ExampleTest1Steps steps1 = new ExampleTest1Steps();
	public static ExampleTest2Steps steps2 = new ExampleTest2Steps();


	@Deployment
	public static WebArchive createDeployment() {
		// create the WebArchive to deploy – skipped for brevity 
	}

	@Override
	public InjectableStepsFactory stepsFactory() {
		return new ArquillianInstanceStepsFactory(this.configuration(), steps1, steps2);
	}
}

We now have AbstractArquillianStories, which does almost the same thing as its little brother, AbstractArquillianStory. The one small difference is that it extends JBehave's JUnitStories instead of JUnitStory, so that we no longer the need the @StoryFile annotation. The location of the actual story files is computed inside the storyPaths method from JUnitStories and overridden by AbstractArquillianStories to handle the virtual file system of the container. By default, it instructs JBehave to search for files inside a folder named stories that must be in the classpath. (You can change the name to whatever value is required.)

A note about development

You've seen that the ArquillianJbehaveRunner class is able to run and deploy multiple stories, scaling as needed so that you don't have to waste time redeploying for each test. As I've noted, the programming technique to implement this solution isn't perfect, but it is functional. In order to understand the mechanics of the workaround, you need to know a bit about how Arquillian runs tests.

In Arquillian the testing process is designed to run in two separate JVMs: one for the "local" JUnit process and one for the container. The local process goes through the list of tests and calls the container side for each test method that must be invoked. Every time this happens, inside the container the JUnit runner for the test class is reinitialized; the test class instantiated but the tests to be run are limited to the one method described by the local JUnit process as being next in line. For 10 test methods, the JUnit runner is invoked inside the container 10 times and only once in the local JVM, because the local JVM is the one that actually keeps track of the progress.

Putting this behavior in context of the Arquillian-JBehave integration, for every step in a scenario the test class (AllStories in Listing 6) is re-instantiated each time along with the call to the stepsFactory method. If the steps1 and steps2 fields weren't static, every time a step method was called, it would happen on a different class instance. No state would be available between @Given and @When. Kind of like stateless session beans -- which we obviously don't want. That's why the steps candidates provided by the stepsFactory method have to be static pre-initialized fields.

Conclusion: What's next for the Arquillian-JBehave integration

A number of improvements could be added to the Arquillian-JBehave integration in the future. Some of them I will handle myself but others depend more on the Arquillian and JBehave community:

  1. Because Arquillian needs all test methods to be annotated with @Test, all step methods also have to conform to this rule. (This is because of that runtime-generated class comprised of all the step methods that is presented to Arquillian as the real test class.) I intend to remove this little inconvenience in the near future.
  2. Another option is to switch to an ArquillianSuite, when that becomes available, in order to completely ditch the runtime-class generation hack.
  3. The integration doesn't behave correctly when executing pending steps unless there is a method (even if empty) annotated with @Pending matching the pending step text. I am working on this fix and it will be soon available.
  4. The integration also required some changes to the JBehave framework which I made on the 3.8 version, which you can find here. To make the integration work with newer versions of JBehave, these changes should be revised and merged into the master branch of JBehave.

I'm sure there are many other improvements that could be added that I've missed in the above list. Any contributions (via suggestions or actual implementations) would be greatly appreciated.

Get the Arquillian-JBehave extensions

  • Get the Arquillian-JBehave integration source code and demo. Follow the steps in the readme of the example project to quickly setup a running environment.
  • The Arquillian JBehave TestRunner, developed by Vineet Reynolds, is the foundation for Arquillian-JBehave Integration's tests enrichment feature.
  • The JBehave JUnit Runner provides more granular JUnit progress reports in Arquillian.
Join the discussion
Be the first to comment on this article. Our Commenting Policies