Writing software with the grain

Exploring the humanity of Agile methods

Agile software development has proven its worth from a business perspective: Agile teams produce better software faster and excel at retaining superior talent. Another aspect of Agile methodologies is lesser known, but of equal importance: they align with the grain of our humanity. Tim Berglund's thoughtful essay explores the ways three elements of most agile projects make life better for the people who work on them. If you're not already doing Agile development, this article could convert you. If you're already a convert, you'll gain new insight into Agile's success.

As a thought experiment, try to remember the last functional specification you read. (If you're really brave, think of the last one you wrote.) If you're lucky, it's been a while, and maybe you're having trouble recalling the specifics. Or maybe it was yesterday, but you still can't remember what you read. Sifting through 300 pages of carefully numbered, hierarchically organized, traceable, gatekeeper-approved prose is, after all, only marginally more interesting than hearing an omnibus spending bill being read aloud in a nasal monotone. Reading and writing spec documents aren't the most humanizing activities in our profession. Rare is the software engineer who gains a deep sense of professional satisfaction, or insight into his or her humanity, from such an experience.

Defining Agile

I use the term Agile to refer to any process that broadly follows the principles in the Agile Manifesto, not any specific methodology like Scrum or Extreme Programming (XP). Although Agile processes can be rich and varied in their details, here I focus on the fact that they often capture documentation in the form of user stories, that their documentation is lightweight, and that they usually schedule activities in brief bursts called iterations.

Of course, the hip Agile shop you work in doesn't write functional specs anymore, and that's a good thing. Your shop might have adopted Agile for any of a variety of reasons. Certainly agile methods have been the majority report among thought leaders in the Java community for several years now, as they have in the Ruby, Python, ALT.Net, and other communities. Further, case studies exist to show C-level executives why implementing Agile is a good idea that justifies the training expenses, and perhaps organizational changes and even physical-workspace improvements, too. Agile teams are more productive and can recruit and retain better talent, release features more quickly, and deliver higher quality software. Business managers are responsible for paying attention to and optimizing metrics like these -- but they are never the whole picture.

Agile is also effective for another reason that isn't necessarily on the business managers' radar: its practices align with our humanity. Agile has hit on some key insights and practices that are more consistent with common human characteristics than equivalent practices in the methodologies we call Waterfall. Agile recognizes the nature of the people whose activities it seeks to organize. It wisely capitalizes on several practices that people naturally tend to engage in, and in which we find greater fulfillment.

Going with the grain

Most woods have a grain: the fibers in the cell walls are all aligned in one direction, giving the wood a distinctive pattern and structural characteristics. Woodworkers must always consider the grain. Sometimes, as when they cut a board to length, they must cut across the grain; it takes more work and a specialized tool, but it can't be avoided. But when refining the surface of the wood by planing, sanding, or staining, they must work with the grain. The grain might limit what you can do with the wood, gently nudging the tool's motion along the lines of the wood's fibers. The material of the craft -- the wood -- imposes constraints on how the craft is executed and what kinds of results can be achieved.

Like wood, all of the tools, processes, languages, frameworks, methodologies, and communications media we use as software developers have an orientation to them. We might be able to force them all to produce similar results; all may have their own strong and weak points; and certainly all have their adherents and detractors. But none of them is neutral. When used by a competent practitioner to produce good results, each quietly nudges the development process and the thinking that underlies it in one direction or the other. For example, as Steve Yegge famously explains in his essay "Execution In The Kingdom of Nouns," Java code makes you think of things, not actions. Twitter prevents you from expressing complex ideas. Using your IDE's autocomplete feature makes your identifier names longer. To one degree or another, the medium is the message.

Many of the sensibilities mediated by the tools and techniques we use might be entirely neutral -- long identifiers might not be a problem if the whole team has wide monitors -- but some clearly are not. When we consider matching the grain of our processes with the grain of ourselves -- and the pain that results when we don't follow the grain of the person -- the matter is anything but neutral. If our work requires us to act like someone we are not, that becomes a deeply personal problem that can affect the quality of our work. Denying or suppressing common elements of our humanness doesn't consign a software project to failure, but it certainly doesn't add to its chances of success or contribute to a team's vocational satisfaction.

Agile lets us work with the grain of our humanity. Specifically, each of three elements often found in Agile projects -- user stories, lightweight documentation, and iteration-based scheduling -- goes more with the grain of humanity than its Waterfall counterpart.

User stories

Human beings have told stories for all of recorded history. We tell religious stories, political stories, family stories, epic stories, stories just for fun, stories of bravery, stories of cowardice, stories of love, stories of loss, stories of war, true stories, and false stories. The way in which we narrate the events of our day even mimics the pattern of a story. Stories serve to explain our origins, reinforce our identity, teach ethics, inspire us, and entertain us.

If you wanted to define the metadata of story, you'd have three basic entities:

  • Plot: The sequence of events narrated by the story. The plot reflects changes in the setting and the state of the characters in some meaningful sequence over time.
  • Setting: The context in which the characters are placed in time, location, and circumstance.
  • Characters: The personal agents about whom the story is told. They interact with one another through the events of the plot in the particular settings in which they are placed.

Stories are human and are told by humans everywhere all the time, because they mimic the form of our lives. Your day at work yesterday might not seem particularly worthy of song, but it meets all the requirements of a story: it is a story.

Now think back to that spec document again. When you read -- or are guilty of having written -- text like this: Upon receiving the customer purchase order line item change order non-repudiation request document, the system shall produce a customer purchase order line item change order non-repudiation response document, mapping the customer purchase order line item change order non-repudiation ID to the...

You are left to wonder, to the what? To the ... time to go see what's new on the front page of Digg or in my RSS reader? Text like this creates minds ready to be plundered by Twitter, IM, email, and the ready satisfaction of a new browser tab and our favorite bookmarks. In a wired world in which we're already having trouble establishing deep focus and sustained concentration, we can ill afford documents like this. They do not engage and do not satisfy. But why?

Specifications don't narrate the events happening to a person in a particular place and time. They talk about the features of an abstract thing called a program, and nobody ever made friends with a program. There's nothing especially life-like about specs; that is, there is nothing in them that is like life. Rather, they try to reduce a piece of software to a collection of law-like statements about the software in its essence. Instead of describing what happens in the life of a program and the people who use it, they rush straight for abstraction, trying to describe what the program is one subparagraph at a time, repeated dozens or hundreds or thousands of mind-numbing times.

Sometimes this is the right thing to do. In the Western legal tradition, we don't rely on sayings or parables to define the law; instead, we use spec-like writings that describe in exacting detail what is lawful (or unlawful) in specific circumstances. Likewise, when we describe a file format or a communications protocol, we want to state in exacting detail what shall or must or may or must not happen when such-and-such a state obtains or such-and-such input is processed.

Writing laws doesn't feel natural to most people, although it does to many developers. The act of programming converts the imprecise and untidy world of human activity into law-like machine instructions. Even though the languages and tools application programmers use today operate well above the level of individual op codes, we are still writing programs for what amount to entirely mechanistic Turing machines. Making the translation from mental process to algorithm -- when it is even possible -- is a specialized skill that developers can enjoy and do well, but it is not typically a skill shared by the other members of an Agile team. Customers and product owners define the products we build, but they are usually not good at converting their ideas into laws. We shouldn't ask them to do so.

Enter Agile's insight that requirements should be captured in story form. The Agile community has debated the precise definition and format of user stories, but consensus is universal that story backlogs are not the same thing as spec documents. Rather than describe the essence of a program with law-like statements, stories narrate the interaction of a program with its users. They have:

  • Characters ("As an administrator," "As a truant officer," "As a Wayne Newton fan")
  • Plot ("I want to disable a user account," "I want to update a student's attendance record," "I want to find all covers of the song Danke Schoen produced during the selected date range")
  • Setting (which is implicit in the area of functionality being narrated)

User stories have many advantages, but fundamentally they capitalize on the normal human tendency to retell our lives using story. Unlike specs, stories require us to do little abstraction -- to keep a relatively simple mental concept of the program we are describing in our minds at any given time. We simply follow the characters and their actions, just as we've been doing with all kinds of stories since we learned language. It's the human way to describe software.

Lightweight documentation

The Agile Manifesto famously prioritizes working software over documentation. Traditional software documentation usually goes unread because it is always out-of-date, doesn't compile, and is often among the most obtuse prose produced by the children of men. At worst it misleads those who do soldier through it and is a drain on the time of those who write it. This is not to discount the written word as a means of storing knowledge -- besides being wrongheaded, that would be a self-defeating point to make using written words -- but rather to establish the parameters of how we normally use writing within the human relationships in a typical software development team.

People who grow up free of certain disabilities always learn to speak the language to which they are first exposed. They never need lessons, don't appear to have to work at it, and learn the language more or less perfectly by about age 10. Not so with written language, which far from being a natural capacity is instead a learned technology. Almost no one learns to read and write without specific instruction. It's not hard for most people to pick up, but it is forced when compared to speech. The written word is a step less natural than the spoken word.

That said, human beings have taken to this particular technological form like ink takes to paper. Since the invention of writing, humans have produced mountains of text in a dizzying variety of forms. We write deeply important things, such as national constitutions, histories, and religious texts; and mundane things, like celebrity gossip and bills of lading. We write highly structured poems and highly unstructured stream-of-consciousness essays. Works of all types have survived writing's long history to give us a rich deposit of books to be read and studied, some thousands of years old. Through old books, we have access to millennia of accumulated knowledge and insight. In so many ways, writing is an unqualified good.

But how often do we read old books? For most of us, the answer (given with a tinge of embarrassment) is not often. This isn't because they're worthless, or don't have anything to say to us, or not necessarily even because we're lazy. Rather, it is because our default mode of learning new things relies significantly on people and things near to us. Academic and religious institutions can help with the important work of mediating old texts to contemporary readers. But other than that, most people are more likely to ask a question of someone they trust than they are to look to a Compendium of All World Knowledge for answers -- regardless of whether the people they trust are good sources, or if that Compendium had good answers in it all along. For right or wrong, when we have a question, we'd rather ask a member of the tribe than read the answer in a book.

Agile handles this tension responsibly. Recognizing the power of written language, Agile encourages us to keep written records of stories and even write the occasional spec document when the situation calls for it. However, in pushing for documentation to be lightweight and pragmatic, Agile explicitly recognizes that documentation is at best a reminder to have a conversation with someone we trust: a customer, a product owner, or a peer on the team. A properly functioning Agile team has no one canonical text to which a new team member can be directed to learn everything about the product; instead, the knowledge of the product is stored in the minds of individual team members. This approach has clear advantages and disadvantages, and different observers might come to opposite conclusions after weighing them carefully. Regardless of the outcome of that debate, it is clear that Agile's preference for storing knowledge in the tribe, rather than in long documents, is the more natively human approach.

Iterative development

Waterfall projects required us to be good at predicting the future. Not only did we need to know the requirements of a product that wouldn't come to market for perhaps another year or more. We also had to know how long it would take us to build features we wouldn't become closely acquainted with for months. We were asked to be prophets of the highest caliber, and we know how well this worked out. Humans are notoriously bad at knowing the future.

We're gifted, however, at wanting to know the future. Wanting to know which horse will win, when our stock picks will peak, whether our project plan will be accepted, or whether she'll say "yes" is as human as it gets. We expend significant effort devising means of making good guesses about the future and coming up with the most likely predictions. Depending on the kind of thing we're trying to predict, sometimes we enjoy modest success, but usually our odds are no better than random. This never seems to stop us from trying again. Nothing will keep us from playing soothsayer.

For Agile simply to wallow in the unrewarding human passion to know tomorrow before it happens would not be much of a help. This is, on balance, an unproductive tendency we humans have, and it is one to be minimized if possible. Agile helps us manage it by acknowledging our poor performance as diviners and imposing on us constraints about how far out we may look. It does this through iterations.

Iteration planning still requires us to predict things we can't yet see, but it specifically prevents us from looking out too far. By keeping our would-be prophetic vision on the next couple of weeks, it focuses us on things we know better -- that is, the user stories we've already thought about and that are already functionally related to work we've done recently -- and reduces the future uncertainties about which we must guess. It recognizes the bad results we normally get when we prognosticate, so it keeps our prognostications to a bare minimum. Through the structure of iterations, Agile is circumspect about our ability to know the future. While denying us our desire to predict might seem cruel, this constraint is in fact deeply humanizing, because it respects the limitations inherent in the people who plan and execute software projects. On this account, Agile helps save us from ourselves.

Human, not superhuman

A word of caution is in order, lest I make my point too strongly. Too often, discussions of Agile methods lend themselves to a breathless tone and an air of utopianism. Proponents of Agile sometimes sound as if we just figured out how to write software in early 2001; that all projects undertaken before then were classic Waterfall projects; that all Waterfall projects must have been failures; and that Agile will soon cure the software crisis, world hunger, and Simon Cowell's temper. Prior methods weren't a thorough success -- hence Agile's emergence and enthusiastic uptake -- but the work of the two generations of brilliant programmers and managers who preceded Agile's birth is not to be lightly cast aside.

Agile is a better development methodology than Waterfall, but it's no silver bullet. It is the right paradigm for right now but not the last Big Idea we'll have about how to build software. Paradigms will eventually shift again, and we can trust future thought leaders that the next paradigm will be an improvement over what we are doing now.

We should be modest about Agile's finality, but we shouldn't let that modesty stop us from recognizing what is true about it. By directing us to tell stories about software, preserve knowledge in groups first and documentation second, and keep our predictions to a bare minimum, Agile works with the grain of the human beings who give life to development projects. The human grain is a noble one, and life is better for our teams when we respect it.

Tim Berglund is an independent consultant who loves to help his customers be the best they can be at Web development on the JVM. He enjoys architecting and developing Java- and Groovy-based systems using Agile methods, and training and mentoring teams to do the same. He blogs at www.augusttechgroup.com/tim/blog, and is on Twitter as @tlberglund. In his spare time he pretends to be a liberal arts major.

Learn more about this topic

  • The Manifesto for Agile Software Development states the principles that underlie Agile methods.
  • In "The New Methodology" (Martin Fowler, martinfowler.com, December 2005), Fowler -- one of the signers of the Agile Manifesto -- explores the reasons for Agile methodologies. Among them is "putting people first."
  • Agile's antithesis is the Waterfall model.
  • Waterfall was alive and well at the (fake) Waterfall 2006 conference, which included (fake) sessions like "Work Harder, Not Smarter" and "Introduction to Dogmatic Programming."
  • "Agility meets the waterfall" (ShriKant Vashishtha , JavaWorld, March 2008) offers practical ways to embrace Agile methods, even if you work in a Waterfall shop.
  • The Agile Alliance is a community that support the values and principles of the Agile Manifesto and organizes the (real) series of Agile conferences and other events.
  • Nouns crush verbs in the Java language, according to "Execution In The Kingdom of Nouns" (Steve Yegge, Stevey's Blog Rants, March 2006) -- a great example of how a language's "wood grain" affects the programs we write.
  • The Turing machine, described by mathematician Alan Turing in 1936, is an abstract device that can be adapted to simulate the logic of computer algorithms. The operation of all digital computers can be described using Turing's model.