Scrum co-inventor: Agile can lower risk, but it won't tell you how to code

Ken Schwaber says we need a more evidence-based approach to software development

Ken Schwaber was an originator of the popular Scrum framework for agile software development, along with Jeff Sutherland, and he was a signatory of 2001's Manifesto for Agile Software Development. These days, Schwaber is more interested in evidence-based management, which is intended to base decisions on current, best evidence rather than on circumstantial evidence. He will be speaking on this topic at the ALM Forum conference in Seattle this week.

InfoWorld Editor at Large Paul Krill recently talked with Schwaber about the state of agile, where mobile and Web development fits in, and Scrum and evidence-based management.

InfoWorld: What brought about the need for Scrum and agile development methodologies?

Schwaber: OK, a slight distinction. Methodology tells you what to do very precisely. Scrum is a framework within which you can develop complex products, but you have to figure out how to do it yourself. The old-style methodologies were waterfalls, like from Coopers Lybrand and Ernst & Young, and they very precisely told you task by task by task what to do.

Jeff [Sutherland] and I had a lot of experience with them and found them very inappropriate because those people who had built those methodologies had never been in the same situations, serving the same customers, building the same requirements, using the same technologies that we did. We were both working at different companies in Boston. We built Scrum to build software to suit and serve more customers with, and we were both using fairly complex object-oriented technology to build the software. We were both serving pretty demanding customers, and we felt that we could rapidly handle changing requirements to work with some pretty unstable technology and get [product] to our customers very quickly.

Scrum is very much like a [book on the] game of chess. It simply describes how you would go about building software, but it wouldn't tell you move by move what to do. You have to figure that out for yourself.

InfoWorld: How does agile impact mobile and Web development?

Schwaber: The word "agile" was derived by a bunch of us who were building software for pretty new products and pretty new technologies like you just described. We have iOS, Android, those types of things, or for building software for things like cars or medical products. Almost all those people, none of them use waterfall. They all had these approaches, which required very short cycles of work to find out what was possible and then building on that. Again, think of the word "empirical."

All of those places use something that is like Scrum. They may use different words, they may use different approaches, but Scrum is based on what those people had to do and have to do to survive and compete.

InfoWorld: What makes agile projects go wrong?

Schwaber: It's very hard for an agile project to go wrong. It's very easy for it not to deliver what you expect. If you want to learn the game of chess, I teach you how to play chess, and you go to play someone, and you might not be skilled enough yet to play against that player. It's going to take you years to really gain the skills to do it. Agile processes don't guarantee you success. All they do is set up a short cycle, like every two weeks, [in which] you try to build the most important thing you can.

Let's say you want to build Obamacare. You would try for two weeks to build the most important functionality you could. And you built them and you might try three or four two-week cycles in a row, and at the end of two months you might have learned that this is going to be so complicated it's going to take you six years and $17 billion to do. You might then say, "OK, This is getting too expensive for us." Now at that point, it's not scrum or agile that failed. You discovered that it's going to cost too much. It's a way of controlling risk through short-cycle development. You never commit to doing anything more than one short cycle.

InfoWorld: Would you say that agile is the dominant mechanism for software development these days, or are development shops still relying on waterfall or maybe a hybrid of waterfall and agile?

Schwaber: They don't hybrid well at all. But many organizations -- I think a Forrester survey said 90 percent used agile, so scrum primarily, and there were some 60 percent that still used waterfall in some ways. There you have to think about large banks, large insurance companies, they have systems that are 30 years old. They're not going to change the way they approach or touch those.

InfoWorld: Are they stuck in the waterfall method, or can they move to agile?

Schwaber: They mostly stick with short-term waterfall. They might set up enhancements or things that they have to do to meet legal requirements and do them in a six-month cycle, and they'll use waterfall.

InfoWorld: These days, you're talking a lot about evidence-based management. What does evidence-based management bring to the table as far as business decisions, software development decisions, and the like?

Schwaber: In software development and in IT organizations, you have to establish some baseline trends of the value that they produce. Let's say an organization has a budget of $200 million and it wants to then do something differently. What we're doing is we're saying -- measure the value that you've produced now for the business, then when you change something like adopting a new methodology, a new approach, a new set of tools, measure the change in value you produce to make sure that continues, gets better, and justifies the cost you're spending on doing things differently.

InfoWorld: With evidence-based management are you only applying it to software development, or does it extend beyond that?

Schwaber: It comes out of the medical field, but what I care about is software.

InfoWorld: How important is ALM these days?

Schwaber: It's absolutely critical for building high-quality software rapidly. However, you have to really strongly invest in upgrading the knowledge of most workers who use it. The problem is if you don't, all you've done is bought this really fancy bunch of software that no one knows how to use.

This story, "Scrum co-inventor: Agile can lower risk, but it won't tell you how to code" was originally published by InfoWorld.