It is well-known that as software projects grow in size, their probability of failure escalates dramatically. (For our purposes here, “failure” means failing to meet its objective, either by being scrapped, or by the goals being substantially scaled back. A lesser form of failure is when the project drags out far beyond its original time line, by a factor of two or more, and at a cost many times it original estimate.)
So, what is it about large projects that is so difficult? Frederick Brooks pointed out in his seminal work the Mythical Man-Month that small teams are much more effective than large teams. But why? What is it about large teams, and large efforts, that is so intractable?
The “agile” software development movement is a reaction to much of what is perceived as dysfunctional about traditional methods of software development, and the use of large, process-intensive teams creating documents is one of those practices that is rejected by the agile movement. The feeling is that projects are better off if one simplifies how people coordinate their work. The best way to do this is to put the entire team in the same room and make sure that they talk. Documentation is largely dispensed with. The results are impressive. My former company Digital Focus (now Command Information) built its business around this approach.
Not every project can be run this way though; some tasks are large and require large projects and you cannot put 100 or 1,000 people in the same room. This problem must be solved so that large endeavors can be reliably undertaken. I contend that one of the major causes of the failure of large software projects is the use of a document-centric approach throughout the software lifecycle: from the development of requirements through the development of maintenance materials.
Not Knowledge
Documenting is a great aid for the documenter. As the author of four lengthy books, I can tell you the process of writing compels the author to organize one’s thoughts. In a software project, the creation of documentation should never be seen as a milestone of progress. This is because documents are not actionable. Documentation of requirements or business processes does not convey knowledge. At best, it is only a starting point. Disbanding a team that assembles requirements as documents squanders the knowledge that team accumulated.
Only knowledge is actionable. Knowledge is the result of humans interpreting information – from documents, from the explanations of others, and from their experiences – and building understanding in their minds over time. The phrase “people, process, and technology” that one often hears is therefore misleading and dangerous. The flaw is in the concept of process, which is often taken to mean process documents.
I have seen too many projects planned around the concept that you just have to have someone create documents, and then bring in people to execute based on those documents, using some provided technology. It does not work that way, and that type of thinking leads to failure. You need to focus on the knowledge that is created as a result of creating documents, systems, and the other durable elements of your organization. The act of creating information creates knowledge, and that knowledge is the core of your business process; and that knowledge resides inextricably in those individuals.