I recently read Mark Schwartz’s book War and Peace and IT, an excellent treatise on why companies struggle so much with delivering IT capabilities. What Schwartz writes about organisational dynamics of large enterprises really hit me because it put my personal experiences with Cloud Native transformation projects into a context I didn’t see before.
Let me explain...
In many companies around the world, tensions simmer between IT and business leaders, and their complaints illustrate the problem we are facing:
‘If only they would have given us the correct requirements we wouldn’t have to rebuild this now’, say the engineers and IT managers.
‘We only have two weeks to get this service out, but the engineers want to gold-plate it again and it’s now going to take two months’, say the business leaders.
Neither of these statements are wrong—but also, neither of them is the right way to look at the challenges modern enterprises face. So what’s really happening here?
Much of the antipathy between ‘The Business’ and ‘IT’, comes from both sides’ view of uncertainty, risk and complexity, according to Schwartz. Whilst business leaders are dealing with the uncertainties of changing markets, unreliable suppliers, and keeping a good reputation, IT is dealing with the complexity of technology—mounting technical debt, service outages, the struggle to retain knowledgeable engineers. Most importantly, neither side wants to own or sometimes even acknowledge the risk they are unfamiliar with.
Typically, the business’s strategy to deal with technological complexity has been to manage IT as a service provider—to agree on deliverables and deadlines. This allows the business to ‘manage away’ the underlying complexity, and force IT to deal with the associated risks.
From a purely technical point of view this approach works, but it backfires terribly as we progress deeper into a digital future where IT is essential to everything the company does. It’s not possible to move fast without IT and business working as one team (or rather, as many cross-functional teams; more on that later).
IT’s reaction to the supplier-contractor model invented by the business was to accept it and work on the basis of requirements. Whenever the project didn’t meet its business goals, it was never IT’s fault, it was the business leaders who submitted the wrong requirements.
The supplier-contractor model of IT management has become so entrenched that, even today, people consider it ‘just the way things are’. As a result, agile transformations bump up against the business/IT divide; many organisations wind up adopting the trappings of Agile, such as holding a few Scrum ceremonies, without truly getting the benefits of working in an Agile way.
The pushback against the broken supplier-contractor model has been ongoing since the publishing of the Agile Manifesto. At its heart, Agile development is about close collaboration in small teams and fast iteration, rather than the slavish following of requirements documents.
Around 2009, the DevOps movement started, which aimed to heal another rift—one between software development teams and the operations teams who are tasked with keeping the servers (and thus the software) working. The way software gets deployed and managed in production has a huge effect on the company’s overall agility. Many Agile development teams found it overly difficult to respond to users when they could only deploy to production once a month.
Back in 2009, implementing technical practices for DevOps wasn’t easy. There was a reason behind the separation of development and operations. They were very different worlds; one was focused on the codebase whilst the other was focused on managing the hardware resources and operating system configurations.
With the arrival of the first public cloud provider—AWS—this part of the equation changed. All the hardware could be provisioned using APIs. With virtual machines and Docker, the operating system became much less important. The line between software development and operations blurred—or even disappeared entirely. This, in turn, made DevOps practices much easier and cheaper to implement. And DevOps, of course, made real Agile development much more feasible for organisations with many teams deploying software to users multiple times a day.
At Container Solutions, we use the term ‘Cloud Native’ for software development and operations that fully leverage the cloud. With the latest annual State of DevOps report, we now have proof that to be a high-performing organisation in today’s fast digitising world, you have to become Cloud Native.
This means embracing Cloud Native technology with the aim of fast delivery of software features. Fast delivery will in turn unlock improved collaboration between business and IT. To make this happen, the supplier-contractor relationship has to be replaced.
What should it be replaced with? With tight-knit, cross-functional teams. Ones where technologists and others work closely together to deliver better products and services. We need tech-savvy domain experts and business-savvy software developers discussing how to deliver market-winning solutions cost-effectively.
We associate startup companies with innovation and creativity. The secret sauce startups have that makes them so fast and agile is communication; they are a small, tight-knit group of people who have to cover every function in the company whilst closely collaborating.
Communication lines are short and people need to be flexible in what they do and how they solve problems. There are no unnecessary silos, and everyone knows that IT is powering everything. Finally, everyone works under direct market pressure—if the product doesn’t sell, the whole company goes under.
Startups are great, but there is no reason an established company couldn’t replicate this way of working (and of course, the high-performing ones do).
As an example, imagine a large re-insurance company that wants to accelerate the creation of new life insurance companies in a particular market. The firm has everybody it needs—software engineers, analysts, actuaries, UX designers, and so on—but all of those people are spread across many offices, and the IT organisation has somewhat antiquated practices.
It isn’t that the re-insurer’s IT people don’t know what modern best practice looks like. It’s that the organisation has, over time, evolved into a siloed entity with over-specialised departments.
To work around the problem, the company creates and funds an independent startup. The start-up has its own office, brand, and budget. Everyone is physically co-located. If a developer needs to speak to an actuary, she can walk across the room and do so. The actuary, in turn, understands how the IT system works; it’s not somebody else’s problem in another building. IT is incentivised to move fast, and slow bureaucratic processes aren’t an excuse.
This is where Cloud Native comes into play. The IT team will need compute to run their code. Instead of waiting for new hardware to arrive, they’ll run their first application using an on-demand cloud service like Fargate or App Engine. They will create CI/CD pipelines using managed services and move to Kubernetes once the complexity of the solution grows. Getting a Kubernetes cluster will be a few clicks away.
Of course, this cloud infrastructure will quickly grow complex when adding more and more services. Managing that complexity will be its own challenge and techniques like Infrastructure-as-Code, comprehensive monitoring, and DevOps will have to be adopted— otherwise the project will slow to a crawl. Many of our clients have experienced this explosion in complexity and needed our help to get it under control.
Is it that simple? Just treat projects like startups, staff them will all the right people, and bury the trenches that have been dug between ‘IT’ and ‘the Business’? During my time at Container Solutions I have seen many companies where this approach would have made a huge difference, but in truth it rarely gets properly adopted. Cloud technology adoption is often treated as solely an IT concern.
This is because it is very difficult to go from a siloed organization to an agile, Cloud Native model. All the people you need are dispersed in different departments, working under different managers. To successfully execute such a project, you need understanding and buy-in on all levels of the organisation. Achieving this can be a multi-year process, depending on the company’s size.
Difficult it may be, the cloud technology revolution can only be fully leveraged by adopting the right organisational structure and ways of working. If you don’t do it, you’re giving the competition that does get it right a massive head start.
Small, quick actions that require little investment of time and money but increase knowledge,reduce risk, and benefit the entire organisation—inside or outside of a transformation scenario.
In an uncertain environment, slowly increase investment in learning and information gathering; eventually you uncover enough information to reduce risk and make better informed decisions.
When someone has an idea that requires validation, the costs of doing experiments around it needs to be as low as possible.