Patterns of Software Systems Failure and Success

by Capers Jones

Just when I was beginning to despair that I would have a book worth reviewing this month, the Federal Express delivery truck stopped outside my office, and a breathless young man delivered a copy of Capers Jones' new book to me. I was aware that Capers was working on the book last year, but its arrival was a pleasant surprise indeed.

Anyone familiar with the work of Capers Jones knows that the man is an amazing source of metrics and data about every conceivable aspect of the software industry and the software engineering profession. From personal experience and acquaintance, I know that at least two others, Larry Putnam and Howard Rubin, have access to an equally bountiful database of software statistics; but Jones has been far more prolific when it comes to publishing the numbers for all to see. And with his published works, he focuses on the more important question: what do all these numbers mean?

Of course, numbers and data can be not only used, but misused; all of us have seen instances where statistics have been used to distort the truth, or to introduce confusion into a discussion of a phenomenon that's not well enough understood to be measured at all. From a scientific perspective, it's unfortunate that all three of the major metrics gurues in he field — Jones, Rubin, and Putnam — have built competitive businesses upon their respective metrics databases. The details of almost all of their metrics data (as well as all the other so-called metrics and survey data that we read in the trade magazines and computer newspapers) are unavailable for independent review and verification. This is extremely frustrating at times, but for the moment, I'm prepared to accept the numbers that Jones shares with us in Patterns of Software Systems Failure and Success, and participate in the discussion of what meaning can be derived from them.

The focus of Jones' investigation is extremely relevant and important in today's environment: what are the metrics associated with successful systems development projects in successful companies — i.e., those that are delivered on schedule, relatively close to budget, and with high levels of quality and reliability? As Jones observes:

"What was surprising was that all of the successful projects tend to follow a similar pattern even though they were created by different companies, in different countries, and within different subindustries, and had different business and technical purposes for creation... The pattern is this: there are myriad ways to fail when building large software systems. There are only a very few ways to succeed. All of the paths that lead to successful software have these twelve essential attributes: effective project planning, effective project cost estimating, effective project measurements, effective project milestone tracking, effective project quality control, effective project change management, effective development processes, effective communications, capable project managers, capable technical personnel, significant use of specialists, substantial volumes of reusable material."

And that, in a nutshell, is what the book is all about: voluminous data, and cogent, insightful discussion and analysis of the patterns of success and failure in software projects with regard to these factors dealing with management issues, social issues, and technical issues. Some of the numbers, and some of the discussion, will confirm your "gut feeling" and your common sense about the best way to develop software. And some of it will take you by surprise, and may lead to some controversial arguments; in his discussion of the impact of ISO 9000-9004 standards, for example, Jones notes that "there are only a few years of empirical data. However, to date ISO certification does not appear to affect the failure or success patterns of software projects, or even to benefit software quality in a tangible way."

The emphasis that Jones places on successful projects and successful companies is important, for it complements and supports the "best practices" movement taking place in many organizations. (It is perhaps no coindidence that Jones and I are among the civilian participants in an advisory board known as the Airlie Council that has been developing a formal "best practices" initiative for the U.S. Defense Department software procurement activities.) And this suggests how Patterns of Software Systems Failure and Success might be put to most effective use by software organizations around the world today: as a basis for recommending a set of key "best practices" to ensure success. If the twelve attributes that Jones has identified are indeed "necessary and sufficient," to the use the mathematician's language, then how many of them are being carried out effectively in your organization? How would you know whether they were successful? What kind of metrics do you have — and how do your metrics compare with the metrics documented by Jones?

For example, do your projects exceed a level of 95 percent cumulative defect removal efficiency? If not, why not — and what impact does that less-than-best-practice have upon the success or failure of your projects? If a skeptic in your organization complains that the issue is irrelevant, fine: where is the metrics data that confirms that defect removal efficiency has no impact on project success? And if that issue isn't important, then what are the issues — whether it's a number like 12, according to Jones, or perhaps some smaller number — that determines the success or failure of your projects?

Thus, while there is enormous value to the numbers and the practices identified and discussed by Jones in Patterns of Software Systems Failure and Success, the real value of this book is the introspection and organizational self-analysis that it invites. There's no question that this important new book should be required reading — the sooner, the better! — by all software managers. In my opinion, it's the kind of book that software technicians should read, too, for it will give them a much better insight into the viability of their organization, and the chances of success or failure in their next project.

 

For more information, please visit Ed's companion site here.
You may also visit Ed's blog here.