Newsletter sign-up

View all newsletters

Sign up for our technology specific newsletters.

Enterprise Java

Email Address:

Should we be doing more automated testing?

Decide whether automatic testing makes sense for your application

By Ben Teese, JavaWorld.com, 03/07/05

This article is about whether we, meaning professional software developers, should be doing more automated testing. This article is targeted to those who find themselves repeating manual tests over and over again, be it developers, testers, or anyone else.

In this article I ask:

• Are we realistic about how much testing we're going to do?
• When does it become feasible to automate testing?

Note that this article doesn't discuss whether we should be testing (be it automated or manual). Nor is it about any particular type of testing, be it unit testing, system testing, or user-acceptance testing.

Instead, this article is intended to act as a prompt for discussion and contains opinions based upon my own experience.

A real-world example, Part 1

Let's start with the job that I recently worked on. It involved small changes to a moderately complex online Website—nothing special, just some new dynamically-generated text and images.

Because the system had no unit tests and was consequently not designed in a way that facilitated unit testing, isolating and unit-testing my code changes proved to be difficult. Consequently, my unit tests were more like miniature system tests in that they indirectly tested my changes by exercising the new functionality via the Web interface.

I figured automating the tests would prove pointless, as I guessed I would be running them only a couple of times. So I wrote some plain English test plans that described the manual steps for executing the tests.

Coding and testing the first change was easy. Coding and testing the second change was easy too, but then I also had to re-execute the tests for the first change to make sure I hadn't broken anything. Coding and testing the third change was easy, but then I had to re-execute the tests for the first and second changes to make sure I hadn't broken them. Coding and testing the fourth change was easy, but...well, you get the picture.

What a drag

Whenever I had to rerun the tests, I thought: "Gee running these tests is a drag."

I would then run the tests anyway and, on a couple of occasions, found that I had introduced a defect. On such occasions, I thought: "Gee I'm glad I ran those tests."

Since these two thoughts seemed to contradict each other, I started measuring how long it was actually taking me to run the tests.

Once I had obtained a stable development build, I deployed my changes into a system-testing environment where somebody else would test them. However, because the environment differed, I figured I should re-execute the tests just to make sure they worked there.

Somebody then system-tested my changes and found a defect (something that wasn't covered by my tests). So I had to fix the defect in my development environment, rerun the tests to make sure I hadn't introduced a side effect, and then redeploy.

The end result

By the time I'd finished everything, I had executed the full test suite about eight times. My time measurements suggested that each test cycle took about 10 minutes to execute. So that meant I had spent roughly 80 minutes on manual testing. And I was thinking to myself: "Would it have been easier if I'd just automated those tests early on?"

 

About Us | Advertise | Contact Us | Terms of Service/Privacy

Copyright, 2006-2008 Network World, Inc. All rights reserved.