Everything is fragmented—Ingenuity and co-evolution

When I was at school, I created an all-time record, namely by avoiding participation in the annual cross-country run for all seven years from the age of 11 to 18. That took a fair degree of ingenuity, and I gather toward the end, bets were being laid in the staffroom to the annoyance of the sports master. Now, I climbed most weekends, sailed a trapeze boat twice a week and played rugby. So, it wasn’t laziness; I just couldn’t see the point of running 12 miles in the freezing cold through muddy lanes.

As I started to develop a career in the IT industry, I needed to exercise a similar level of ingenuity on projects to avoid the imposition of waterfall design methods and excessively formal project management. I was designing and developing advanced decision support systems, and it was simply impossible to specify the system in advance of writing the code and experimenting. No one had ever had a decision support system before; they were still using journals and reports (spreadsheets were novel in those days). And few if any of the users had any idea of the capabilities of software. As a result, if you asked them what they wanted, they told you what they currently did, or asked for automation of existing processes. To use an adage of that time, "Users say they know what they want until they get it, and then they want something different."

Now I was in a good position of having been development accountant of a company for three years, so I understood the application area. In addition, we had the first high-level languages (anyone else remember FCS, more powerful than Excel?) that allowed rapid development and amendment of systems. I also had the good sense to build parameter-driven sub-routines for all common functions including screen formatting. As a result, I could talk with the users in their own language; go away and develop a module with real data; and create reports, monitoring screens and other processes based on a synthesis of my knowledge, the stated needs of the client and my knowledge of the technology. The application would work in novel ways, users would find new ways of working, and modifications would be agreed upon. Over the course of a year, a powerful application emerged that was very different from anything that either the user or I could have defined.

Of course, such an approach requires trust, and it was hairy at first. When I got the job, I thought I had competed with others, but it turned out I was the first person in a year to turn up with the right qualifications, and they had a contractual commitment. I joined on Monday, went on a three-day crash course in FCS (which I never used) and was sold to the client as an "expert" on day four. I learned as I worked, just about staying ahead of the client. But we ended up with the most profitable account in the business (which got me promoted) and with a very happy client, because we had co-evolved a solution to a problem they knew they had (shorted reporting timescales). But they also had a set of solutions to problems they had not been aware of, or they had assumed no solution existed.

That experience, a growing interest in object orientation and a frustration at linear design led to the formation of the DSDM Consortium (dsdm.org), which created a set of standards for RAD/JAD approaches (that’s Rapid Application Development and Joint Application Design, for my younger readers).

This is a good example of the theme of this series, namely the application of complex adaptive systems theory to the whole field of system development. In a complex system, we have light constraints on agent behavior and agents that modify the system with which they interact. Overconstrain the system (waterfall design processes) and you end up delivering something that matches the spec, but doesn’t match the aspiration of the users. Remove all constraint (the users get what I think they should have), and you have chaos.

By creating a co-evolutionary process, with a fair number of safe-fail experiments, we avoid the pitfalls of fail-safe design. In more recent years, I have formalized software design within the Cynefin Framework—an approach that uses the power of waterfall process for ordered or structure environments, but starts to use methods such as social network stimulation, the subject of my previous column. It also leads us to the need to capture requirements with the necessary ambiguity of narrative. How to do that will be the subject of my next column.