Under the heading ‘The Future of Cloud’, Gartner ran a symposium for CIOs and IT executives in 2022 including much discussion about strategies relating to Cloud and Cloud Native trends. At least two of the main talks (one, two) were centred around a five-year horizon, discussing where cloud adoption will be in 2027 compared to where it is now.
As part of those talks, Gartner referenced a useful pyramid-shaped visualisation of different stages of cloud adoption. It could be viewed as a more schematic version of our Maturity Matrix, which we use as part of our Cloud Native Assessments with clients.
Previously, we wrote about the pyramid in “The Biggest Cloud Native Strategy Mistake” blog post. Building on that, the timelines suggested by Gartner in their talks prompt us to consider the topic of Cloud Native Transformation timelines in general based on our experiences at Container Solutions.
The pyramid
Gartner’s pyramid depicts four stages of cloud adoption from the perspective of business scope. These stages are shown as a hierarchy where the bottom layer represents the lowest dependency (“Technology Disruptor”) and the top layer represents the highest level business goal (“Business Disruptor”).
The four stages can be briefly summarised as:
- Cloud As Technology Disruptor
- The new base technology is adopted. For example, containerised applications, or a move to using a cloud service provider instead of a data centre.
- Cloud As Capability Enabler
- Now you have new technology in place, you can more easily build capabilities that may have been more difficult to achieve before, such as automated testing, or CI/CD.
- Cloud As Innovation Facilitator
- With new capabilities, you have the right environment to foster innovation in your business. This means you might, for example, leverage your cloud platform to deliver features more quickly, or conduct A/B testing of new features to maximise your return on investment.
- Cloud As Business Disruptor
- The most advanced stage of cloud adoption, where you can use the previous three stages’ outputs to change your business model by, for example, migrating to a SaaS model, or scaling your client base significantly, or introducing an entirely new product line.
Whilst it is somewhat higher level, this pyramid is similar to our Maturity Matrix in that it helps give you a common visual reference point for a comprehensible and tangible view of both where you are, and where you are trying to get to, in your Cloud Native program. For example, it can help in discussions with technologists to ask them how the changes they are planning relate to stage four. Similarly, when talking to senior leaders about stage four, it can help to clarify whether they and their organisation have thought about the various dependencies below their goal and how they relate to each other.
2027
Where did Gartner get the 2027 date from? One cynical interpretation would be that it’s five years into the future, and that’s long enough for anyone to make a prediction that they’re not likely to be held to. However, it’s not completely arbitrary, as it’s likely related to a question they asked industry leaders.
Gartner asked 367 technology leaders how far their companies had progressed on their Cloud Native programs. Asked whether cloud was ‘indispensable or heavily impactful’ to their business, 50% agreed in 2022, but 90% said they expected progress to that point by 2027.
This five year horizon caught our eye at Container Solutions, as a sizable chunk of our potential customers come to us with plans to complete their cloud transformation programs within two years rather than five.
The realism of cloud transformation plans was another theme of Gartner’s symposium, as they asserted that 40% of enterprises’ multicloud expectations will be unachievable, with large challenges specifically around high availability requirements especially prevalent in highly regulated industries.
Why are there these disparities between stated goals and realistic expectations? This in turn leads us to reflect on the realism of cloud native transformation timelines in general.
Why We Have Timelines
Timelines exist for a number of reasons, mostly around alignment. Putting a timeline on a project helps with coordination, gives clarity to priorities and goals, and helps define what success looks like for staff who want to be rewarded. And most transformation programs are so large, any date put on them is effectively an arbitrary one set by someone who doesn’t know the detail, and also most likely the date is set for political reasons.
Timelines are of course also a kind of estimation. By putting a timeline on a goal you are implicitly saying it can and should be done in that time. Unfortunately, this impression is often not backed up by the reality of most projects.
The trouble is that even at a small scale estimation is difficult. This article about estimation times for design projects shows that, even for simple and well-known multi-step processes, the base estimate median estimate diverges very quickly from the reality. Even with 2-3 steps to complete a project, 50% of estimates were too low. Each step introduced to a process imposes a ‘task tax’ that makes meeting the deadline less likely.
Fifty percent might sound pretty good, but remember that this is for the very simplest of projects with the least uncertainty. It’s simple because there are only 2-3 well-defined steps, and the uncertainty is low because it’s a repeated business process where quality can be sacrificed to meet the deadline.
If you are drafting a new logo for a design client and a deadline is looming, you can present something that is ‘good enough’ rather than spend more time polishing it. This is not so true for technical projects where there may be a more binary outcome to the task as set. For example, if you are tasked with authenticating with an API, it will either succeed or fail. You might succeed in five minutes if you have the correct details to connect with and the default library works with your code, or it might take weeks if there is a bug on the API side.
We can quickly see that our cloud transformation timelines are a perfect storm of worrying indicators: Both a high number of steps (a cloud transformation might take thousands or even millions of steps when broken down), and a high degree of uncertainty as to the estimation of those tasks.
This is a recipe for massive project overrun against the original timelines set by the business. And indeed, that is what we have observed at Container Solutions: Arbitrary dates set for transformations bear little resemblance to the real-world timelines of delivery.
The Effect of Unrealistic Timelines
Does this matter though? Many believe that a target date helps motivate teams to deliver quickly, and counteracts tendencies to waste resources in perfecting solutions that are already good enough. Again, our experience is that this is true if the domain is relatively well known and the uncertainty is low, but not if the uncertainty is high and the domain less well known.
An analogy might help here. Imagine we are tasked with doing our weekly supermarket shop by our spouse, who is a harsh taskmaster (any resemblance to an actual spouse is entirely coincidental). Our spouse believes we can do the shopping and return home within one hour and hands us a list of items to get. We need to be back within the hour because our daughter needs to be taken to a doctor’s appointment that can’t be missed, and our partner will be very angry if we are late.
Now imagine two scenarios. In the first scenario, we know where the supermarket is (it’s 15 minutes’ walk away). In the second scenario, we have just moved home and don’t know where the shops are, or what they are likely to stock.
In the first scenario, we walk to the supermarket (15 minutes) and shop (25 minutes), collecting most of the items. Some are missing, so we make various ad hoc decisions about either not buying the item or finding an alternative. These decisions get increasingly rushed as we run out of time. The queue at the checkout is busy today, but we have enough time, so we leave the shop and take 20 minutes to get home, with 5 minutes to spare. The result was not perfect (a couple of items are missing and substitutes may not be ideal), but in general it was a good outcome as we now have enough food for the week.
In technical terms, we might say that for scenario one the project was successfully delivered on time, and some technical debt was incurred.
In the second scenario, we try to google the nearest local supermarket. It’s 20 minutes away, so you walk there, but we discover that Google had the wrong opening times, and it’s closed on Thursday afternoons. So we hastily find another shop 5 minutes’ walk away from the first shop. This shop is small and has very few of the key items we want. We only have ten minutes to get out of the store, so we rush around an unfamiliar shop trying to prioritise the most important items. This takes 20 minutes, and the checkout takes another 5 minutes. We run home as fast we can (15 minutes), and get back 5 minutes late.
In technical terms, we might say that for scenario two the project was an abject failure. We have exhausted ourselves trying to deliver on time, the delivery failed to get key items, and we were late.
The negative outcome for scenario two was far more likely because there was so much uncertainty about how it could be delivered (or whether it could be delivered at all), and the domain was unknown.
| Scenario One | Scenario Two |
| Known domain (done this here before) | Unknown domain (never done this here before) |
| High degree of certainty about tasks (little variance in outcome in previous attempts here) | Low degree of certainty about tasks (no data on previous attempts here) |
| Goal achieved with ‘good enough’ success | Goal failed, work will need to be re-done |
| Delivered on time, no effect on dependent projects | Delivered late, knock on effects to other dependencies. |
| Spouse satisfied, daughter satisfied, you are healthy | Spouse unhappy, daughter unhappy, you are exhausted |
In the first scenario we saw how the deadline acted as a spur to action, encouraging you to make decisions that struck a balance between meeting the deadline and fulfilling the overall goal of the project (get food for the week).
In the second scenario the deadline was actually detrimental to good decision making. The deadline forced decisions which meant that In every dimension the project failed. In fact, a more agile approach to the deadline could have resulted in a better outcome. For example, a decision could have been taken to be late for the appointment, or to defer the shopping until a later date.
Scenarios such as the second one can result in a significant demotivation of staff tasked with delivery against deadlines set by those far away from the project. This can lead to further negative outcomes such as a significant increase in staff attrition, a feeling of drift and failure in the project, or defensive, non-risk taking behaviours designed to protect jobs rather than result in any positive outcomes for the endeavour.
What’s the Alternative?
If arbitrary deadlines are not appropriate to the task of transformation, then what is to be done? First, identify what kind of project this is. Hopefully it’s clear by now that delivery of large-scale transformations are in the category of endeavours not helped by arbitrarily set deadlines.
Once you have identified what class of project this is, you should identify with as much clarity as possible what your overall goal is. We use our ‘measure what matters’ Cloud Native pattern to articulate this approach. This pattern emphasises the need to have the right metrics for success for your project. Where uncertainty is high, optimising for cost or deadlines can lead to perverse outcomes, as we have seen.
Does this mean that cost and timelines don’t matter? Of course not! A business that doesn’t minimise cost and never delivers is in trouble very quickly. Here we use other Cloud Native patterns to ensure the best outcome for the least cost. To minimise cost when uncertainty is high, we need to reduce uncertainty by ‘research through action’ and ‘reducing the cost of experimentation’.
‘Action’ (as opposed to extensive ‘analysis paralysis’) is the best way to reduce uncertainty, by doing targeted proofs of concept or individual spikes of work to flush out what is well understood and what is not. Since you are not trying to do ‘everything’ in one project, you can focus more on reducing uncertainty for future deliverables. Each action, ‘small bet’ or experiment should be done as quickly and cheaply as possible, with the goal of building the learning functions of your business.
Focussing on your rate of learning allows you to move with more certainty towards your goal, reducing waste in the process. You will even meet your goals earlier.
Applying This To Our Shopping Scenario
How can we apply this kind of thinking to our shopping analogy above? Scenario one is an example of a ‘mature’ operation, so research and small experimental bets were not required. However, in scenario two, we tried to apply our ‘mature’ scenario one approach to a novel situation which required innovation, and this resulted in failure.
The first thing we could have done differently in scenario two was spend a small amount of the hour we had confirming some details with the shop. For example, we could have phoned the shop to check it’s open and check what they had in stock. If our researches had not confirmed these things were safely known, then we could have renegotiated the shopping task with the requester in the light of this learned uncertainty. This could have resulted in a more informed decision about how to proceed, eg:
- Abandon the shopping task as too risky
- Increase the spend on the task to help ensure safe delivery (eg by getting an Uber to the shop and home, or ordering a more expensive home delivery)
- Renegotiate the doctor’s appointment
- Accept the risk of failure and proceed as before
Maybe you can think of other ways to reduce risk and get a more favourable outcome. Innovation should, after all, be a creative process!
Do We Need Timelines At All?
In an ideal world, we wouldn’t talk about timelines for transformations, as we should all by now understand that such timelines bring little value. Rather, we should be talking about reducing risk by spending smaller amounts of money in tighter cycles in a more agile way, in order to reach our goal as fast as possible.
Not all of us live in an ideal world though, so the real challenge here for most businesses is to move the cultural needle away from simply thinking in terms of dates and plans and towards a more iterative, experimental way of working. A ‘cloud native’ way, in short. We outline that way of working - and how to get there - in our latest book, The Cloud Native Attitude, which is available as a free download from the CS website.
Further reading:
Driving engineers to an arbitrary date is a value destroying mistake
Software Estimation in the Fractal Dimension
https://apenwarr.ca/log/20171213#slide8 - just throwing out a number doesn’t work
Previous article
