"every man lives only this present time, which is an indivisible point, and that all the rest of his life is either past or it is uncertain". - Marcus Aurelius
I often say that process is an emergent property of people working together. I’d like to explain this a little bit more now.
A problem and a solution always mirror each other. When a problem is novel, i.e. when we've never come across it before, there is no solution. In this situation, what people tend to do is make a solution up as they go along. They improvise. In hindsight, we may be able to say what our process was, but that doesn't mean that before we started to solve the problem we had a process defined. (Looking back like this to create a story means you are falling into the narrative fallacy. Humble et al. say "This is arguably how methodologies are born".) This is why I say process is an emergent property.
Right now, at Container Solutions, we are coming up with new uses for new technologies, like Docker and Mesos. At the same time, we are trying to help our customers makes sense of these technologies. There are no best practices we can fall back to. This makes sense when you think about what a best practice is.
A best practice is a rule of thumb that emerged from a time when a problem was novel. The best practices in hospitals are the results of doctors and nurses, in times past, ‘making it up as they went along’. So what was an emergent practice years ago is a best practice today. The problem with technology, and software development in particular, is that each problem is novel, or at least parts of it are novel, and therefore a set of best practices cannot form the whole solution. This is of course true for medicine today; doctors and nurses rely heavily on best practice but also improvise when they come across something new.
We can rely on best practice only when a problem is familiar. I can give an example: at Container Solutions we have an accountant and a lawyer. We did not need to experiment for a year to discover that we needed these. Knowing that we need an accountant was not an emergent property of running the business.
Why Does It Go Wrong?
In big organisations, people tend to rely on process so they see any failures in software development as a failure in process. Their reaction is to apply more discipline to whatever scrum-fragile-waterfall-kanban process they are following. In such organisations, many problems are well understood and best practice often works, it's just that they don't work for developing software.
Other people, the experimenters, say nothing can be known. Their reaction to any hiccups in the software development process is to do more experiments.
What’s the Truth?
Where does the truth lie? As always, the answer is, ‘it depends’. Many of our clients would benefit from improved process, but usually in trimming their process down, not adding to them. At the same time, in the more complex parts of their work, for example in a bank where they are playing with new algorithms, it can be wise to experiment a bit more.
Complexity Must be Met With Action
Over at Container Solutions, we are innovating in a number of areas. This means we don't exactly know what’s going to happen. The best strategy here is to act. When a problem is familiar, you can afford to think, make a plan, and then act. When you are innovating, the opposite is true. You act, then you reflect/have a think about what you just did, then you make a plan for what to do next. This is a nice way of saying "we make it up as we go along". During times of innovation, the past won't help, the future is unknown, and so all the entrepreneur or innovator can do is react accordingly to the present moment. When done properly, this looks more like surfing a wave then fighting a fire. But just like real surfing, you sometimes fall off.