After seven years of trying, we at Container Solutions have figured out that the best way to start a Cloud Native Transformation is with a little bit of upfront thinking followed by a little bit of building. In these early stages, we are building to learn, to remove risk, to form new habits, and to establish the right culture.
We do our upfront thinking with our Cloud Native Assessment: the “Think” phase in our “Think Design Build Run” process. We then do our building in the five month design sprint which forms our “Design” phase. Together, the Think and Design phases make up our “fuzzy front end”, which is the first part of the innovation process. It is during the fuzzy front end where opportunities are identified, concepts developed, and key constraints identified. The front end of the innovation process is where we apply creativity to create value. We use all manner of tools here, from the whiteboard to surveys, from Edward de Bono’s 6 hat thinking to rapid prototyping.
The funny thing about the fuzzy front end is that it isn't really fuzzy at all: it is systematic and rigorous, following a defined and repeatable process. We call it fuzzy because it deals with the incoherent, chaotic, ambiguous and contradictory thoughts and ideas we all have at the beginning of the creative process. In other words, it is our ideas that are fuzzy, not the process itself.
To cut through the chaos, during the fuzzy front end we build to learn. This is common in other disciplines too. Architects use mock ups, moving models on a table or dragging and dropping fixtures and fittings on their CAD tools. Car designers create full size, clay models. They want to feel what it’s like to stand next to the car.
As a child, Charlie and the Chocolate Factory author Roald Dahl went to school next to a real chocolate factory. In the book, Willy Wonka is described as “the most amazing, the most fantastic, the most extraordinary chocolate maker the world has ever seen!'", and depicted as inventing all the chocolate bars and sweets entirely himself. In reality though food scientists from the factory would regularly visit the school with new sweets to test on the kids. They too were building to learn.
What did those food scientists have in common with car designers and architects? Why do they all work like this? They work like this because “this” is how creativity works. Creativity is not about first fixing a problem and then searching for a solution. Instead, we formulate the problem, the initial solution, and then through a constant iteration of analysis and design, refine both at the same time. That’s why after testing the sweets on the kids, those scientists would dash back to the lab and integrate any new insights into the next version of their recipes.
I said that the fuzzy front end was rigorous and followed a well defined process. But I never said it was linear. We bounce between the solution and problem like those food scientists bounced between the school and the lab. However, at some point, both the problem and the solution start to emerge and solidify.
The fuzzy front end is about using time wisely. Our large customers might have a lot of money but they cannot magic up more time. Every year they waste is a year in which their competitors get ahead of them. The fuzzy front end is where we use the right amount of time and creativity to work out what we’re going to do for the next few years. It’s the place where we sow the seeds for future success, and decide what we’re not going to do. It is the moment when order is drawn from chaos. From our customers’ perspective, this is the most vital service we provide.
Why doesn’t everybody do this?
Honouring the innovation process and the fuzzy front end is the best way to succeed with Cloud Native transformation. So why doesn’t everybody do it?
If you’re an engineer, as I once was, the fuzzy front end can be an uncomfortable place. When we are younger, we learn to solve closed problems as I did when I was a student learning to program. These problems were about calculating insurance claims or tabulating football results or solving famous equations, like Euler’s.
Later, in the world of work, things were not much different. My first job was building out the web application for a large car manufacturer, which was cutting edge in 2000. As a back-ender, I had to work out the nearest dealership to a customer. Or I had to provide quotes for a car based on the particular model and range of accessories. Just like the problems I solved as a student, these too were bounded and relatively easy to solve. I was comfortable solving them.
As my career progressed, I was eventually drawn into the much murkier world of requirements, where we had to figure out what our software should do. This is an open-ended problem. Open-ended problems have a range of solutions, some of which are better than others, some of which are awful. I was plunged into the ambiguous, chaotic and confusing world of product design: the fuzzy front end.
Back then, software development (mostly) did not honour the innovation process. Instead, we first fixed the problem and then solved it sequentially, progressing from the problem to requirements to design to implementation to operations. This didn’t work. The remarkably high failure rate of software projects was proof of that. The failures on my project with the car manufacturer were further proof, as if we needed it.
We all know that creativity sees us bounce between the problem and solution “spaces” as we gradually understand both, so why did we stick to our linear, “waterfall”, method of development? Perhaps because it was emotionally untaxing. It did not force us to live with uncertainty, did not ask us to be wrong, did not ask us to fail, did not ask us to tackle that notorious open-ended question, “what should this software do?”. Instead, the waterfall method changed product development into a closed problem. Sequential. Calming. Logical. Utterly unthreatening to the egos of those using it. And wrong.
It led to products that never shipped or worse, bloated garbage that never should have shipped. The way we developed was the software equivalent of those food scientists coming up with a new sweet and producing a million of them without testing them on a single child.
Feedback loops
This state of affairs led some people, including Barry Boehm in 1988, to seriously consider feedback loops in software development. Boehm’s Spiral Model explicitly asked us to improve prototypes into what he called operational prototypes, similar to the Minimal Viable Product (MVP) from Lean Startup. By doing this, Boehm had figured out how to develop software whilst discovering requirements. This allowed the problem and solution to co-evolve, integrating parts of the fuzzy front end into an iterative method for software development: a vast improvement on earlier sequential or “waterfall” methods.
Extreme Programming
In 1999 we had another breakthrough. Kent Beck created Extreme Programming (XP). Beck famously said that “Optimism is an occupational hazard of programming; feedback is the treatment”.
Like Boehm’s Spiral Model, XP also tackled feedback explicitly, capturing the different loops in the XP “planning onion”, shown below.
XP brought many innovations but the two most important ones, when it came to building software as you discovered requirements, were continuous integration and a team that included a customer.
Continuous Integration
Continuous integration (CI) was, for most teams, a mind blowing innovation. Before CI, I used to plan my work, get through it, then pleasantly integrate my changes into the repository, sorting out merge issues as they popped up. The merge process included walking over to other people’s desks and discussing our changes. This was a social part of our work (much more social than reviewing pull requests). We were not slower back then but our work tended to come out much more fully formed. But, the way in which we integrated was not as effective and we of course had more merge conflicts to resolve.
I used to teach continuous integration by asking my colleagues to imagine a robot sitting behind a computer in the corner of the room. This robot’s job, I said, was to poll the repository for changes - and yes, the repository was Microsoft’s SourceSafe - build your code and run your tests. Whether or not the build failed, the robot would send you a message, thus automating a manual process which we all already followed. But, by checking in regularly, we could keep that robot busy and in doing so remove a mountain of build errors and importantly, integration bugs.
There were, however, three blockers to any team adopting continuous integration:
- The first was psychological; people were not used to checking in half-baked source code and some were embarrassed others would see their work in progress.
- The second was habitual; people had their way of working that was for the most part effective. Why should they check in after every major change? A lot of people liked to work all week and integrate their changes on Friday afternoon before having a beer.
- Finally, and most fatally for the adoption of continuous integration, almost nobody wrote automated unit and integration tests. Continuous integration makes no sense if there are no automated checks, but back then people used manual tests, the debugger and common sense. Manual testing, i.e. the good judgement of a human being, was one thing that our robot in the corner could not do.
Embedded customers
Once you had continuous integration it was possible to always have a working product. Not a complete one, of course, but by starting with the most important features you’d have something usable as soon as the first few features were in. This meant that an embedded customer could do continuous user testing, giving feedback in real-time. If the customer had made a mistake in stating what they wanted built, they could correct it. If a mistake was made in what was built, they’d correct that, too. A customer in the team made sure we built the right thing, as the engineers would make sure we built the thing right.
This combination of build automation and better team configuration made dealing with working out what software should do a lot more pleasant, especially if you were a developer (which I was). No more requirements gathering and documentation. No more workshops. XP really did let us “collaboratively develop software while discovering requirements”. The results were often staggering. Building software was now as fun in a team setting as it was in your bedroom on your own. XP, like Boehm’s Spiral model, baked the fuzzy front end into the software development process.
Enter the Cloud
Despite its power, XP could not alter the constraints of the universe. Businesses still needed to budget. Teams still had to be assembled. Starting quickly and iterating forward, as XP asked, was perfect for application development but made less sense in other areas. Many years later we would discover that one such area was Cloud adoption.
Adopting Cloud technologies was not a local change but something all-encompassing. It changed the way finance teams recorded costs and how businesses would deliver to their customers. It altered operations and application development, asking both groups to harmonise on tooling. Cloud did and still does require a large upfront investment, which the FT recently reported should be 5 to 10% of total infrastructure costs. This meant companies could not iterate to the Cloud in the way they had got used to iterating towards new software products.
After AWS’s release in 2006, a pattern started to emerge. Companies who tried to ramp up too quickly would find themselves ramping down just as quickly. This was embarrassing and costly. Large companies could not code themselves onto the Cloud like the XPers had coded their way to requirements. The difference was that the XPers, because of their way of working, honoured the fuzzy front end. Those early movers to the Cloud did not. They were skipping that vital, non-linear, iterative and chaotic bit at the beginning where they worked out what they needed to do. Their impulsive jumps to the Cloud were like our food scientists ramping up full scale production for the first chocolate bar that popped into their heads.
This gave the team at Container Solutions a problem to solve. We knew full well that problems and solutions co-evolve, and that flying into a Cloud Native transformation without any upfront work was a recipe for disaster. The failure rate of those moving to the Cloud was showing us that as early as 2010. How could we stay true to our agile roots and acknowledge that Cloud Native transformation was more like taking a new chocolate bar to market than it was building a new application? We needed to do what Boehm and Beck had done before us. We needed a method that included the fuzzy front end of innovation.
After a few years of trying, iterating between the solution and problem space, we adapted the innovation process for Cloud Native transformation. In our “Think” stage we use a range of tools to refine both the problem and the solution. This process leads to a lot of hypotheses, many of which obviously won’t work and so are discarded, and quite a few that might work. We test the ones that might work experimentally in our “Design” phase. Together, these two phases take 15 weeks. Our Think stage is the only time we don’t build. That takes three weeks. In Design, we start building to learn.
What type of things do we shake out in our fuzzy front end? We work out which cloud provider to use. Or we discard the public cloud altogether. We figure out the right tools, the developer experience and the processes that link the two. We also start to inventory the applications that might be refactored or re-engineered, those that will go to the public cloud and those that will not. These days we also work out which applications from failed Cloud migrations need to be repatriated. In essence, we map out the Cloud Native transformation, iterating on the design of our customer’s chocolate bar until we get it right.
Pini Reznik and I started Container Solutions because of the experiences we both had as young engineers. We - and thousands of programmers around the world - knew that you do not fix a problem and then search for its solution. We didn’t want to contribute to what can only be described as collective insanity. Our ideas were formed by the work of Boehm and later Beck and the other pioneers of the agile software methods, many of whom we looked up to and many of whom we’d later meet. But we were confronted with a problem that at first we could not solve; how do you transform a whole organisation to Cloud Native? You can not get away from thinking. Many organisations’ bias for action was causing them problems, because their actions were undirected, and since the Cloud made clear winners and losers in all organisations, they were also hugely political.
We found ourselves drawing not on recent ideas but going back in time. Back to strategy, and especially Mintzberg’s emergent strategy, and back to the innovation process. Fearful that we might become what we had previously beheld, fearful that we would create a linear method for Cloud Native transformation - for it undoubtedly goes at least a little bit in a straight line - we came up with a short thinking phase followed by an intense build-to-learn phase. We thus honoured our bias for action whilst still honouring the fuzzy front end. Like Boehm and Beck before us, we had worked out how to collaboratively build both software and programmable infrastructure whilst flushing out constraints and requirements.
We built elements of the fuzzy front end into our method. It took us half a decade to figure out, with the design phase only really coming to life when we assembled a team of strategists and behavioural psychologists. But we got there in the end. Our customers love our think and design tools. It’s not just that they save a ton of money. They save a ton of headaches, too.
If you’ve made it to the end of this post, I highly suggest you download our book on Cloud Native Transformation. It further tackles obstacles that organisations face on a Cloud Native journey, it’s got patterns to work through people problems, and it touches on a little bit of the tech that supports it all. The stories are showered with mistakes, lessons learned and new ways of looking at old problems. This is why it’s been (unpromptedly and externally) dubbed the ‘go-to guide’ for Cloud Native Transformations. It’s currently free to download.
Previous article
