Selenium 2 and Thucydides for ATDD

From automated web tests to automated acceptance criteria

John Ferguson Smart is an advocate of open source tools that support and automate the software development lifecycle, especially in Agile environments. Now he presents Thucydides, his own testing library built on top of Selenium 2/WebDriver. As he demonstrates, Thucydides rethinks and extends the potential of ATDD; first by turning automated web tests into automated acceptance criteria, and then by documenting that criteria for collaborative, multidisciplinary teams.

Acceptance test driven development, or ATDD, is a collaborative practice where users, testers, and developers define automated acceptance criteria early in the development process. The team then uses that criteria to guide subsequent design and coding. The acceptance criteria document a common understanding of the problem domain and its proposed solution. In multidisciplinary teams, such documentation helps to ensure that everyone agrees about a project's requirements and implementation. When acceptance criteria are automated, they become part of the "living documentation" of the system, always accurate and up to date.

If you are writing a web application, it makes sense to have your automated acceptance tests exercise the application as a user would, via a web interface. Many web testing libraries do just that, including open source tools such as Selenium, Canoo WebTest, WebDriver, and the recent Selenium-WebDriver integration, Selenium 2. These tools focus on testing the details of web page interaction, and they're great at that.

Most automated acceptance tests do not completely fulfill their potential, however, in that they do not drive the development and documentation process as much as they could. As a proponent and practitioner of test-driven development and other lean, collaborative development methodologies, I wanted a testing framework that would let me express and report on acceptance tests at a high level, group tests by stories and features, and then drill down into the details as required. And so I created one.

In this article I introduce Thucydides, an open source ATDD framework built on top of Selenium 2.

Introducing Thucydides

Thucydides is a software testing library designed to help developers write automated acceptance tests and web-based automated acceptance criteria. Thucydides provides extensions and utilities that improve web testing and development on Selenium 2 in three ways:

  1. Organizing web tests into reusable steps, and mapping them back to their original requirements and user stories.
  2. Generating reports and documentation about acceptance tests. Each report contains a narrative description of a given test, including a short comment and screenshot for each step. Reports serve as living documentation, illustrating how an application implements the specified requirements.
  3. Providing high-level summaries and aggregations in each report. These give an overview of how many stories and requirements have been tested by automated acceptance tests, which in turn lets developers gauge the relative complexity of each story.

Different stakeholders need to be able to view acceptance test at different levels. For instance, a developer needs to know how a feature should interact from the perspective of the end user. This will start at a high level, expressing the general business goals behind a feature, but during the development process the developer will progressively drill down until she is specifying detailed page-level interactions.

A tester will have similar needs, and may also need to review test results to see how the application behaved under test. QA staff would generally rather not run the application through each scenario, but can benefit from seeing how features are implemented by reviewing narrative reports, illustrated with application screenshots.

A project manager is not so interested in the finer details. She will be more interested in the relative size and complexity of different features, and in the number of working tests for each feature -- a good indicator of progress. A business representative or project management office might want an even more high-level view: based on the number and complexity of the features defined and implemented so far, is the project still on track?

All of these aspects of ATDD extend beyond the page-centric nature of traditional automated web testing, and they require particular extensions to implement.

Thucydides is built on top of Selenium 2.0, the latest version of the very popular Selenium open source automated testing framework. Selenium 2.0 is the fusion of Selenium and WebDriver, and incorporates many improvements over Selenium 1, including a simpler, cleaner API, built-in support for the Page Objects design pattern, and excellent Ajax support.

Installation and setup

It's easiest to see the benefits of Thucydides in practice, so the remainder of this article will be a hands-on demonstration. If you want to follow along, you will need to clone this demo project from Github onto your local disk. You will also need to have Maven installed in your development environment.

The Thucydides binaries are available from the Sonatype OSS repositories, so you need to add them to your Maven settings.xml file. See the Thucydides project wiki for detailed information about setting up your environment and Maven project for Thucydides.

I've taken my example application and testing scenarios from the jobboard-webtests project, hosted on Github. The application under test, jobboard, was built using the Play framework v. 1.1.1. If you want to run the demo app, first install Play, then go to jobboard-demos-->jobboard, and type play run.

Once you have the jobboard application running, you can load up the jobboard-webtests project in your favorite IDE and start coding.

Defining requirements in Thucydides

Software projects typically aim to provide end-users with a number of high-level features (sometimes called functionalities or capabilities). A feature is a general value proposition relating to something the application can do for the end user, expressed in terms you might put on a product flyer or press release. For example, if you were developing an online recruitment website, you might include features such as "Look for jobs" or "Manage job ads."

Features are usually too big and unwieldy to implement in one step. In fact, software developers typically divide them into manageable chunks. In Agile projects, these chunks are expressed as user stories, which are short sentences that capture the essence of what the user wants from each feature. In other projects, more formal use cases serve the same purpose.

When automating acceptance tests with Thucydides, you can define the features and stories you intend to test in the form of a simple class structure. This lets you map tests back to the requirements that they verify in a lightweight and robust manner. In Listing 1, the @Feature annotation indicates a feature, and nested empty classes within a feature class indicate the stories or use cases that comprise that feature.

Listing 1. Defining a feature

public class JobBoardApplication {

        @Feature
        public class LookForJobs {
            public class JobSeekerBrowsesJobAds {}
            public class JobSeekerBrowsesJobsByCategory {}
            public class JobSeekerAppliesForJob {}
        }
         
        @Feature
        public class ManageJobAds {
            public class EmployerPostsJobAd {}
            public class EmployerRenewsJobAd {}
            public class EmployerReviewsJobApplications {}
        }
        ...
    }

Although you don't have to model your requirements in this way to use Thucydides, it does produce a very useful record of both the progress of the project and the number of requirements currently being tested.

Automating acceptance criteria in Thucydides

It's a great start to define application features and stories, but we usually need more information before we can begin coding. In a traditional waterfall-style project, this would take the form of detailed specifications. For projects based on iterative, agile, or lean development methodologies, features are defined by discussing each story in detail, in order to better understand its requirements. In particular, this involves drawing up a list of acceptance criteria, or conditions that an application must satisfy for a requirement to be complete. Engaging the requirements process as a team -- with product owners, business analysts, developers, testers, and so forth all participating -- is essential. It helps to create a shared vision and common understanding of the requirements to be implemented.

Acceptance criteria can take many forms, but the best acceptance criteria tend to be quite high level, focused on what the user is trying to do rather than how he or she will do it. This makes the acceptance criteria more robust and easier to maintain; it also lets the team stay focused on the core value proposition of a feature, rather than its implementation details. (That part comes later.)

A narrative style often works well for acceptance criteria. Listing 2 shows an acceptance criteria written in the popular "given-when-then" structure.

Listing 2. A given-when-then styled acceptance criteria

    Scenario: A job seeker can apply for a job online
        Given the job seeker is looking for a suitable job
        When the job seeker finds a good job post
        And the job seeker applies for the job
        Then the job seeker should see a confirmation of the job application
        And the employer should be notified

Some teams strive to make acceptance criteria easier to understand by making them more concrete. The criteria in Listing 3 reads more like an example than a specification.

Listing 3. Specification by example

    Scenario: A job seeker can apply for a job online
        Given the job seeker is looking for a job as a Java developer
        When the job seeker find a job post for a Java Developer at Acme Consulting
        And the job seeker applies for the job
        Then the job seeker should see a confirmation of the job application
        And the employer (Acme Consulting) should be notified

In some cases, such as when calculations are involved, a table-based approach could be more concise. BDD (behavior-driven development) frameworks like Cucumber and Easyb provide good support for table-based acceptance criteria, as do parameterized tests in JUnit.

Web test automation

Not all acceptance criteria have to be automated using web tests. Some core business logic, such as financial calculations, can be effectively tested directly against application code. Testing via the user interace is an important part of the testing process for most applications, however, especially those built on rich UIs such as GWT or JQuery. Automated web tests also provides a form of living documentation, illustrating step-by-step how each feature is implemented from the viewpoint of the end user.

Writing automated tests in Thucydides

Automating acceptance criteria involves more than simply writing automated tests for each criteria. It also involves expressing the acceptance criteria in a form that can be easily understood by all team members. Although Thucydides supports conventional testing tools like JUnit, I prefer the expressive style of Cucumber and Easyb.

In Listing 4, I've used the Thucydides's Easyb integration to automate the acceptance criteria from Listing 2

Listing 4. Automating acceptance criteria with Easyb

using "thucydides"
    thucydides.tests_story JobSeekerAppliesForJob

    scenario "A job seeker can apply for a job online",
    {      
        given "the job seeker is looking for a job as a Java developer"
        when "the job seeker find a job post for a Java Developer at Acme Consulting"
        and "the job seeker applies for the job"
        then "the job seeker should see a confirmation of the job application"
        and "And the employer (Acme Consulting) should be notified"
    } 


This is a fairly ordinary story in Easyb, with the exception of two lines: the using 'thucydides' line at the top lets Easyb know that I want to use the Thucydides Easyb plugin; and thucydides.tests_story JobSeekerAppliesForJob maps the automated acceptance criteria to the story defined in Listing 2. This is the logic that enables Thucydides to group acceptance criteria by story and by feature, and thus provide aggregate reporting across all stories and features.

At the moment, the test in Listing 5 (below) doesn't actually do much -- it will be executed during upcoming automated test runs, but will be flagged as pending. Nonetheless, it's now part of the automated test suite. Having each test automatically executed with every build makes it much easier to keep tabs on what features have been implemented, and which are still pending. The test will also appear in subsequent Thucydides reports as a pending step, as shown in Figure 1.

Figure 1. A pending step

Thucydides Steps: Implementing automated acceptance criteria

Once you've automated an application's acceptance criteria the development process can begin. The acceptance criteria act as goal-posts for the software development team, providing a clear set of objectives for each feature.

When implementing a story, one approach is to first write each acceptance criteria in a way that illustrates how the application will fulfill its corresponding requirements. If you're familiar with test-driven development (TDD), this is a high-level variation on that approach, in that you write the automated web test that you would like to have. In practice, implementing acceptance criteria before writing application code is beneficial but not essential. Some teams prefer to write tests only after the UI details have been stabilized. In either case, the test-writing process is similar. Listing 5 shows the Easyb scenario we defined previously, with the details of each step fleshed out, illustrating how each part of this scenario will be performed from the user's perspective.

Listing 5. Easyb scenario defined with Steps

using "thucydides"
    thucydides.tests_story UserBrowsesJobTabs   
    
    thucydides.uses_steps_from JobSeekerSteps
    thucydides.uses_steps_from EmployerSteps

    scenario "A job seeker can apply for a job online",
    {      
        given "the job seeker is looking for a job as a Java developer", {
            job_seeker.called("Joe Blogs").opens_jobs_page() 
            job_seeker.looks_for_jobs_by_keyword("Java")
        }
        when "the job seeker find a job post for a Java Developer at Acme Consulting", {
            job_seeker.should_see_job_posting_called("Java Developer at Acme Consulting")
        }
        and "the job seeker applies for the job", {
            job_seeker.opens_job_posting_called("Java Developer at Acme Consulting")
            job_seeker.applies_for_job()
        }
        then "the job seeker should see a confirmation of the job application", {
            job_seeker.should_see_confirmation_page_for("Java Developer at Acme Consulting")
        }
        and "And the employer (Acme Consulting) should be notified", {
            employer.called("Acme Consulting").opens_job_postings()
            employer.should_see_candidate("Joe Blogs").for_job_posting("Java Developer at Acme Consulting")
        }
    }

You might notice in Listing 5 that I've added a new instruction below the thucydides.tests_story call: thucydides.uses_steps_from. Steps are used in Thucydides to break an automated acceptance test into smaller chunks, while still keeping an eye on (and reporting on) what the test is doing when it is executed. Thucydides steps are high-level, meaning that they don't offer much implementation detail; this helps the team focus on what the feature is trying to achieve, rather than how it will be done.

Next, I define page interactions via Thucydides's step implementation classes. The use_steps_from EmployerSteps method call shown in Listing 6 injects a variable called employer that is an instance of a class called EmployerSteps. This class is where the steps are actually implemented. The next stage of the test-writing process involves implementing step classes. As you see in Listing 6, step classes are just annotated Java classes.

Listing 6. A step class in Thucydides

public class EmployerSteps extends ScenarioSteps {
        public EmployerSteps(Pages pages) {
            super(pages);
        }

        @Step
        @Pending
        public EmployerSteps called(String name) {
            return this;
        }

        @Step
        @Pending
        public void opens_job_postings() {}

        @Step
        @Pending
        public EmployerSteps should_see_candidate(String name) {
            return this;
        }

        @Step
        @Pending
        public void for_job_posting(String jobName) {}
    }

Methods annotated with the @Step annotation indicate a significant step in a test. The @Pending annotation indicates that although this step has been defined, we haven't yet implemented it. Upon execution, each step is recorded; later it will be published in a Thucydides report, appearing alongside a corresponding screenshot. As you saw in Figure 1, even pending steps appear in a report, which gives a clear picture of exactly how the team expects the application to behave for a given story.

Of course the methods in Listing 6 don't actually do anything yet. Next we'll implement each of the methods, in this case by adding WebDriver test code and any other supporting fixture code that is required.

Acceptance testing with Selenium 2

The actual testing will be done using Selenium 2, an open source web testing library that is the successor to the very well-known Selenium 1. Selenium 2 provides a simpler and cleaner API, better support for the Page Objects model (see Resources), and more reliable browser support. It also integrates the WebDriver API.

When you execute a test using Thucydides, whether it be with JUnit, Easyb, or another framework, Thucydides handles many of the Selenium 2/WebDriver infrastructure details for you. For example, it will open a WebDriver session and close it properly at the end of the test. You can opt to open a new browser for each test, or use the same browser session for all of the tests in your class. You can also indicate what browser you want to use by setting the webdriver.driver system property. The ScenarioSteps base class provides the getDriver() method, which lets you use the current WebDriver instance directly in a step:

Listing 7. getDriver() calls WebDriver

@Step
    public void opens_job_posting_called(String jobName) {
        getDriver().findElement(By.linkText(jobName)).click();
    }

For more complex stories, you may need to break a step down into other steps. A step is free to call other steps or methods. If a method is annotated with the @Step annotation, it will appear as a nested step in the Thucydides report. Otherwise, it will be considered a utility method and will not appear in any report. For example, if applying for a job turns out to be a relatively complex workflow, we could implement it using nested steps as shown in Listing 8.

Listing 8. Nested steps

@Step
    public void applies_for_job(String jobName) {
        opens_job_posting_called(jobName);
        enter_qualifications();
        enter_contact_details();
        verify_submission_preview();  
        submit_application();
    }  
    
    @Step
    public void opens_job_posting_called(String jobName) {...}

    @Step
    public void enter_qualifications(String jobName) {...}

    @Step
    public void enter_contact_details(String jobName) {...}
     
    @Step
    public void verify_submission_preview(String jobName) {...}

    @Step
    public void submit_application(String jobName) {...}

Using steps in this way provides a layer of abstraction between the behavior you are testing and the way the web application implements that behavior. This level of abstraction makes it easier to manage changing implementation details, because a desired behavior will generally change less frequently than the details of how it is to be implemented. Abstraction also allows you to centralize implementation details in one place.

Automated web testing with the Page Objects pattern

The Page Objects pattern is an increasingly poplar strategy for implementing automated web tests. You don't have to use Page Objects with WebDriver or with Thucydides, but it can make your code easier to understand and maintain, especially for larger code bases. A page object is a class that masks the more technical details of a web page behind a facade of more business-friendly, implementation-neutral method names.

Selenium 2 supports page objects, and Thucydides extends this support with a range of utility methods. Listing 9 shows a simple Thucydides page object.

Listing 9. A page object in Thucydides

@DefaultUrl("http://localhost:9000/admin")
    public class AdminLoginPage extends PageObject {

        @FindBy(id="email")
        WebElement emailField;

        @FindBy(id="password")
        WebElement passwordField;

        WebElement signin;

        @FindBy(linkText = "Logout")
        WebElement logout;

        public AdminLoginPage(WebDriver driver) {
            super(driver);
        }

        public void login(String emailAddress, String password) {  
            enter(emailAddress).into(emailField);
            enter(password).into(passwordField);
            signin.click();
        }

        public void logout() {
            logout.click();
        }
    }

The AdminLoginPage class extends the Thucydides PageObject class, which provides a number of useful utility methods. The method enter(emailAddress).into(emailField), for instance, makes tests easier to read and maintain. The @FindBy annotations are the standard Selenium 2 technique for identifying components on the screen. And the @DefaultUrl annotation helps Thucydides determine what URL should be used to open a page. It can be modified at runtime using system properties, enabling you to point a given test to different test servers in different test runs.

Ajax methods for Thucydides

Thucydides provides convenience methods for Ajax programming. For instance, Thucydides makes it easy to instruct an application to wait for a signin button before allowing the user to click it:
public void login(String emailAddress, String password) {  
        enter(emailAddress).into(emailField);
        enter(password).into(passwordField);
        element(signin).waitUntilEnabled().then().click()
    }

You usually will invoke page objects from within test steps. Each step class is provided with a page factory object that instantiates and manages PageObject instances. Within a step library class, which is derived from the ScenarioSteps base class, you use the getPages() method to obtain the page object factory. That, in turn, delivers the page object that you need:

Listing 10. getPages()

@Step
    public void logs_in_to_admin_page() {
        AdminLoginPage page = getPages().get(AdminLoginPage.class);
        page.open();
        page.login("admin", "password");
    }

Thucydides reports

In addition to making acceptance tests easier to write and to structure, Thucydides produces reports.

The first reports are generated whenever you execute a Thucydides test case. These reports illustrate both the outcome of the test and the steps executed during the test, including screenshots taken at each step along the way. Reports provide not only information about the test results, but also documentation about how the feature under test has been implemented. You can see this in the report shown below.

Figure 2. Thucydides displays test results in a structured, narrative format

Acceptance tests are also the ultimate progress indicators. An automated acceptance test works or it doesn't -- there is no "80 percent done" in ATDD! In addition, the total number of steps required to implement the acceptance criteria for each user story can be used to estimate the relative size and complexity of that story. Comparing the number and complexity of passing tests to the number and complexity of pending tests gives a reasonable idea of how the work is progressing. This functional test coverage, as it's called, is part of a Thucydides report.

Figure 3. Functional test coverage report

It can also be useful to see things from a higher level. The Thucydides dashboard report gives an idea of the current state of each feature, both in terms of how many tests passed, failed, or are pending, and in terms of their relative complexity. This can give a good overview of the amount of work involved in implementing different parts of the system, in a relatively objective manner.

Figure 4. Dashboard report

Thucydides reports also keep track of the test results over time, so that you can visualize in concrete terms the amount of work done so far versus the estimated amount of work remaining to be done.

Figure 5. A history report

In conclusion

Thucydides is an open source library that provides supporting tools that make writing Selenium 2/WebDriver tests a little easier. The greater potential of Thucydides, however, is in its ability to turn your automated web tests into automated acceptance criteria, in the true spirit of ATDD. The reporting features introduced in this article provide feedback both at detailed, narrative level for each test executed, and at a much higher level. The higher level feedback uses acceptance test results to document your project's progress and status.

Taken as a set of objective metrics , test results show the relative size and progress of each feature being implemented. In this way, Thucydides reflects the collaborative, multidisciplinary nature of software development projects today. It provides software developers with the detailed information required to test and update code, while giving business managers and architects the more high-level views and reports that suits their needs.

John Ferguson Smart is an experienced consultant specialising in enterprise Java, web development, and open source technologies. Well known in the Java community for his many published articles, and as author of Java Power Tools, John helps organisations to optimize their Java development processes and infrastructures and provides training and mentoring in open source technologies, SDLC tools, and agile development processes. Visit Wakaleo.com to find more of John's articles and presentations related to ATDD, Thucydides, and similar topics.

Learn more about this topic

More from JavaWorld

  • See the JavaWorld Site Map for a complete listing of research centers focused on client-side, enterprise, and core Java development tools and topics.
  • JavaWorld's Java Technology Insider is a podcast series that lets you learn from Java technology experts on your way to work.
Join the discussion
Be the first to comment on this article. Our Commenting Policies