Structure is Not Enough; Or, Why People Matter

This is part three in a series of blogs about strategy and Cloud Native. The second part is here. The fourth part is here.

From Basic to Build Automation

Like many others my age, when I was a child I had access to a computer called the BBC. The BBC landed in our laps through the most tragic of circumstances. A man called Christopher Evans, a psychologist who developed an interest in computer science, died of cancer in 1979.  In his last years he wrote a book called The Mighty Micro: The Impact of the Computer Revolution. In the introduction, Evans makes clear how disruptive he thought the coming decades would be:

We of the closing years of the 1970s have the gift of foresight, the ability to contemplate - if not for long - the amazing change that man is about to force upon himself. This book is an attempt to predict the nature of the change, and to signal its imminence and inevitability. There may be very little time left for us to prepare ourselves for the future - but any time is better than none at all.

The book was optioned by ITV and Evans presented the show himself, but by the time it was broadcast he was already dead. Even so, the show had a profound effect on the public.

As a reaction to the ITV documentary, the BBC created the BBC Computer Literacy Project, which included documentaries and the commissioning of an affordable micro-computer complete with a programming language, BASIC.

And this is how, aged 9, I started to program.

A couple of summers later, my Mum got us a Commodore 64, which like the BBC had BASIC built into it. One summer, maybe in 1987, I walked to the library where I took out a book on programming. Once I was home, with the house empty, I sat on the floor in our bedroom and started to type. I had to be patient, exacting, had to be fully conscious of my actions. Once the program was in, I typed ‘run’. Two squares appeared on the screen, when they were the same colour, I had to hit the spacebar and was awarded points. I tested the different controls, which was not straightforward because the monitor was black-and-white, and with a growing sense of satisfaction, I turned it off, the program dying as the electricity emptied from the chips. I had taken my first steps into a different world (a world, I was soon to discover, where knights rode chickens).

What was it about that day that left such a strong mark on me? It gave me a glimpse into the creative process. I learnt that problems and solutions co-evolve, affecting each other. This continues until a solution you can’t imagine to a problem you don’t understand reveals itself. When this happens, you are left marveling, did I do that?

The other thing about computer programming, especially with an interpreted language like BASIC, is that the creator and the creation are close together. Like sculpting, the thoughts of the creator are expressed in the sculpture in almost real time. (This is the opposite of the feedback cycle for an architect, who has to wait years to see the final result). And of course, once the hand stroke is complete the creation changes. This, in turn, changes the creator’s view of it, which is just another way of saying the creation changes the creator. This is the most serene conversation you can have with yourself. Non-programmers don’t get this. Artists do.

Meanwhile, In the Real World...

Things changed when I got my first job. I was only twenty-two, knew nothing about business, nothing at all, and so I couldn’t see the struggles my older colleagues were dealing with. I thought everything was a bit rubbish. What I found in industry was that software was not developed with any process at all. It was either driven by people with loud voices or through massively bureaucratic systems that, to me at least, made no sense at all. For example, if we wanted access to lifelike machines to run performance tests, we’d have to submit a form 12 weeks in advance to get a slot. By the time you’d got the slot, the test you wanted to run didn’t make sense anymore. What this meant was that our work disintegrated to project management, lining tasks up, chasing people, and hoping things would go off according to plan. They rarely did.

To alleviate the management burden, we started to heavily configure our systems and our build scripts until everything was about configuration and automation. The call of ‘who broke the build?’ was a social tool we used to call out someone who a. screwed up the config or b. merged their changes without integrating the latest version of the repo before running their tests*. These social tools, of course, made up for the lack of actual tools. In this sense, we at least changed our frustrations into something constructive.

* Most teams didn’t have automated tests so this in itself was an innovation we rediscovered.

I didn’t know it at the time, but I was angry as hell because I felt that the workplace drove a wedge between me and the thing I wanted to create. (Clearly, a tolerance for frustration was something I’d not yet developed). The wedge seemed to make no sense at all, to my younger ears it was simply excuse layered upon excuse and, ultimately, seemed like a load of bullshit. There were clearly more logical ways to build and deploy software. The teams I worked with started to take matters into their own hands.

We weren’t the only ones. In hindsight, it seems obvious, but at the time, we didn’t know this; many programmers had begun to converge on similar ideas at a similar time. Heavily automated and scripted builds would evolve to become continuous integration. Formal code reviews, which were actually useful, would give way to the more nimbler pair-programming and later things like rubber ducking. Long range planning, in the face of changing requirements and unknown technologies, gave way to strategic goals and iterative planning and replanning. We became obsessed with feedback loops, which are of course at the centre of the creative process, the thing so many of us were missing.

Failure to Scale

Across the industry, it became clear that these new or rediscovered methods for organising teams of people who developed and deployed software worked. By reconnecting the creator to the created, methods like XP had managed to help bring joy back into the workplace. Many of us were as giddy as we had been as children tapping away at our C64s. These bright and lively methods of working soon began to attract interest from bigger businesses. This begat the dreaded two day certifications and agile transformations. Workplaces subsequently became covered in sticky notes, whiteboards, and backlogs. Long range plans were broken into two week segments bookended by a planning meeting at the beginning and a retrospective at the end. But the sticky notes didn’t make it through the board, work didn’t get done. The retrospectives were neither effective or blameless. Software delivery and deployment was as bad as before, only now they were full of Newspeak, and the rich conversations and productivity that had accompanied effective teams ceased to exist. The product backlog became a todo list instead of a constraint to be optimised; the storyboard became a tool of accounting and task tracking. Teams who had previously lived under the tyranny of the never-ending plan (what we called the death march) now lived under the tyranny of the never-shrinking backlog, staring down upon them from the board.

Why did the adoption of the agile software methods fail so epically? Mainly because we misdiagnosed why they succeeded in the first place.

Agile methods, such as TDD and continuous integration, were the result of good teams looking to optimise their processes. From this desire to optimise, to improve, many teams stumbled upon similar things, such as build automation. Later it got a name, continuous integration, and its first open source tools, such as CruiseControl. The methods and tools were an effect of good, open minded people working together. They were an outcome of the teams’ work -- not an input to it.

Getting cause and effect mixed up was at the heart of things going wrong when the agile methods were more widely adopted. The reversal of cause and effect is also at the heart of what’s going wrong when companies attempt to adopt the Netflix stack and management methods. People shouldn’t copy Netflix’s outcomes (the stack and management methods) but the thought processes that created them.

Why Do Some Organisations Learn and Others Don’t?

There are two types of organisations that really matter to society, bureaucratic and generative.

Bureaucracies are about running things at scale. Examples include the government, our banks, and post-offices, manufacturers and food producers. Bureaucracies focus on repeatability, predictability, longevity and driving costs down. Costs are driven down by firstly simplifying work, so a less qualified person or machine can do it, and then by interconnecting the work into flows so that as information moves around it increases in value. For example, raw data might come into the Passport Office as a birth certificate, a completed form for a new passport and a cheque for £72.50. This will move through the security department, where the application is reviewed, before been shifted along to the next stage. All along the process, the original information is augmented and pops out the other end as a passport, which is more valuable than the sum of its parts.

The simplification of work is essential for the bureaucracy’s size and longevity. Any workplace whose tasks aren’t simplified will require more qualified people, which means scaling it and getting repeatable results is difficult or impossible.

We are each of us huge beneficiaries of bureaucracies. In the UK we have free healthcare and a school system that for all its faults gets millions of people educated for free. Within our homes we have electricity, running water, cheap products from around the world, good food and awesome TV - most of which is now also free, or at least very cheap.

We are each of us also at the mercy of bureaucracies. When we moved house, we couldn’t change our electricity bill because we’d contacted the electricity board at the wrong time. My poor Dutch wife was dropped into a hellish, infinite loop of being transferred from one department to another. It took literally weeks to work out.

Bureaucracies are a modern invention, their rise going hand in hand with industrialisation and the widespread deployment of capitalism. We know these things were massively beneficial for society. We also know they’ve brought us to the brink of ecological collapse. This is why we are ambivalent about bureaucracies. When they are there for us and working well, they are invisible and we take them for granted. This means when they slightly malfunction like if a train is ten minutes late in The Netherlands, we go crazy.

We love the law -- which is itself a bureaucracy, though most of the time mainly invisible to us -- as it allows us to function as a society. If however we are ever caught up in the system, for example during a divorce procedure, we can’t stand it because bureaucracies don’t care about our personal circumstances; they are blind by design, treating everyone equally.

It’s folly to judge bureaucracies as good or evil. They are a vital part of our society. It doesn’t matter if this is a popular thing to say or not. It’s nothing more or less than a fact.

The other type of vital organisation is the generative one. A generative organisation’s job is to solve a particular type of problem, known as a generative problem. A generative problem is one that requires new knowledge to solve it. For example, Container Solutions are a generative company. We are trying to solve a problem that cannot be fully solved with today’s knowledge, namely the widespread adoption of the Cloud Native computing paradigm.

Generative organisations sometimes solve a how problem. In the Netherlands, a now famous company, Buurtzorg, are solving an old how problem - how to organise nurses for home visits. The generative problem Buurtzorg are having a crack at is not simply how to get nurses to their patients - that’s easy - but how to do that in a way that is effective, cost-effective and doesn’t treat the nurses and patients as bureaucrats tending to customers, which neither patently are.

Generative organisations, as you read this, may sound cool. And in my opinion, they are. But they are also dependent on people acting autonomously, working in teams with their egos nicely in check, and who are happy to constantly carry out failure analysis, mining what went wrong so they can learn. Such people are rare, not because they are special, but because almost all of us are trained from a very young age to get ready to work in bureaucracies. The first thing we have to do at Container Solutions is hire for and then train people to think and act autonomously. This is so much easier said than done.

Image 1 - Westrum’s taxonomy of organisations.

So Why Then Is Structure Not Enough?

In my last blog, I promised to explain why understanding the structure of emergent strategy was not enough to succeed with it. In this blog, we saw that teams who pioneered the agile software methods did so because they realised that solutions emerged from an iterative dance where problems and their solutions co-evolved. The creative dance needed scaling for teams. With the urge to constantly improve, many teams discovered similar methods, such as build automation, and temporal structures, such as the iteration. These methods and structures were later diagnosed as the source of such a team’s success. They were copied all across the industry -- where they failed to have an impact on developer productivity or software quality. The wrong thing was copied. It wasn’t the structure that needed copying but the thought process that led to the creation of the structure in the first place.

Constant improvements are outcomes of a healthy team and their process. The outcomes can be fed back into the team to compound their effects. This is the basis of a generative team culture. From that culture, group norms, processes, and structures develop. Only then does a strategic execution method like Hermes or OKRs begin to make sense.

Image 2 - methods for strategic execution are layered around an existing culture -- not the other way around.

If you are in a bureaucratic team or organisation, knowing the structure of emergent strategy won’t be enough to succeed with it. This was evidenced by the failure of organisations to succeed with the agile software methods just by copying them.

There were of course teams and companies who did succeed with agile. They did so, however, not because of the methods they used -- but by transforming themselves from bureaucratic to generative teams / organisations. It was the people, not the process, who made it happen.

Conclusion

Strategic execution loses almost all of its power when deployed bureaucratically. This is why it’s not surprising to see massive failures with the adoption of OKRs now in the same way we saw massive failures with the adoption of the agile methods in the last ten years. To understand this failure is to understand why Cloud Native transformations also regularly fail. It is to this subject that we must next turn.

Want to know more about Cloud Native strategy? Check out our toolkit below.