Agile turned 20 this year. Yet how many organisations are actually ‘agile’? What is agile, anyway, and does agility even make sense for all stages and ages of business? Now that IT has gone from being a department within a business to most of the time being the business, it’s a good moment to reflect on the methodologies of the past to see what works for building the future.
First, let’s review the four grounding principles of the Agile Manifesto, which recognises value in both sides, but gives priority to the left:
- Individuals and interactions over processes and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Responding to change over following a plan.
Some would say that a methodology that was created two decades ago by a group of white men doesn’t stand up anymore. After all, so much has changed. And if just about everyone is a user of technology, we are still abhorrently poor at including everyone in the creation of it. We are also far from hitting the widespread customer collaboration mark, especially in B2C.
People are more intolerant than ever of less-than-perfect tools, so just “working” software won’t be delighting users. Developers, who’ve grown over the last two decades to become their own user base, consistently complain documentation isn’t good enough. In the widespread adoption of open source, docs are essential for onboarding, and as code gets more and more complicated, docs are a sign of code stewardship, a living history of why different choices were made.
After years of developer autonomy and the microservices and containerised technology that’s driven it, the pendulum is swinging back toward more cohesive team-based plans, because there’s simply more to do than can be done—choices have to be made. There’s also less widespread experimentation—it has its place, but it’s not throughout the software lifecycle: guardrails are essential for stability and security. As the adoption of low-code and no-code accelerates, it democratises the future, giving everyone an ability to participate in creating technology, without necessarily even having knowledge of any project management principles.
Still, some of the manifesto, especially putting people before code, is more appealing and laudable than ever. And frankly the first clause of the Agile Manifesto rings very true with the growing concentration on psychological safety in the workplace.
Most notably, we know that not every organisation has adopted agile. And a lot of others are “agile in name only”. Yet, business agility is still the gold standard for many. So what can you do?
For this piece, we spoke to Simon Wardley, creator of Wardley mapping, about when—and when not—to adopt agile or its seeming competitors.
Wardley says he has spent 15 years hearing different versions of the same refrain: Agile is the only solution; Waterfall is the only solution; Six Sigma is the only solution; Scrum is the only solution; Extreme programming (XP) is the only solution; Kanban; SAFe: Lean. The list of “perfect fits” goes on. For him, the solution is found in not putting all your eggs in the proverbial basket. As organisations grow, goals diverge, code changes, and how you build software evolves.
Any journey should start with a map
So, if nothing suits every stage and every team, how do you decide which approach applies to which part of the business and stage in your software lifecycle?
Back in 2001, Wardley was CIO of the Fotango online photo service, when they adopted organisation-wide extreme programming (XP), which he uses interchangeably with agile.
“By 2003, we found that it didn’t work everywhere,” he said, realising firsthand that one size does not fit all.
It was around then that Wardley began his mapping. A map is a way to better understand your landscape and identify patterns, so that you can then share information in a common language to literally get everybody on the same page. A map has three essential components:
- The anchor or magnetic north
- Position of pieces (ie: North, South, East, West and distance)
- Consistency of movement
“People like to talk about maps all the time because maps invoke this idea of strategy. But actually what we mostly have are graphs,” Wardley said, offering the example below of first graphs—no due north, so they are actually all the same—and then of maps below.
There is no one process to rule them all
In 2012, James Finley, who was then the CIO of HS2, wanted to build the high-speed rail in a virtual world. Wardley, who was advising him, warned that with government projects like this one, it always comes down to three questions:
- Which bits do we outsource?
- Which bits do we build in house?
- Which bits are off-the-shelf products?
So the HS2 team built a systems map—which, since it lacks a due north or any sort of anchor, Wardley pointed out, is really a graph. Graphs in business are often mislabeled maps.
“In that simple graph, those three questions will give you about 387 million permutations. So how do you know which one to pick? What tends to happen is we don’t. We just go, ‘Let’s outsource the whole lot.’ And then we tend to break into contract structures or lots which sound familiar,” Wardley said. They are often grouped up into outsourceable bits like:
- Engineering - 3D visualisation, project information management system [PIMS]
- Back office - finance, risk ware
- User experience - UX support, CRM collateral
“So we break things into these lots. And then we hand them out as contracts for other people to do. And, generally, it doesn’t work,” he said.
Wardley said to imagine you outsourced not just the commodities and utilities on the right, but also the elements on the left that are labeled custom-built.
“I can tell you this is going to fail before we’ve even signed the contract,” Wardley said. You will incur a massive cost of change, and the vendor will say that’s because your team keeps changing its mind—you didn’t actually know what you wanted.
Wardley continued that these outsourced custom actions on the left “will include excessive change control costs because, if we try to specify them in the contract, you’re going to end up with a fight with the vendor because your project will spiral out of control.”
Now, on the right, outsourcing makes for efficient, on-time delivery because things like the compute power and the website are known entities.
In order to figure out how to better outsource, he tasked Finley with drawing the map below, who then in turn asked Wardley how to manage it.
Wardley said, “Well, that’s pretty simple, from what we learned, because you learn patterns through maps and you reapply them. There are about 30 different common economic patterns, about 40 different principles, and about 110 different forms of game play.”
One clear theme is that there’s no such thing as one-size-fits-all.
That means agile experimentation—what Wardley calls lightweight extreme programming, or what Container Solutions’ CEO Jamie Dobson refers to as the “fuzzy front end”—works great on the left side where you are just working out what you want to do. “It’s all about the Genesis, the creation of the novel and new and deviation is going to be the norm. It’s going to be constantly changing.” By keeping it in-house you shorten the feedback loop and reduce the cost of change.
Then, on the right side, it’s best to use Six Sigma-style outsourcing to utility suppliers, as they are usually best at reducing deviation.
Finally, in the middle, once you know what you want, you start to create a more regular cadence of experimentation, working to create a minimal viable product. This is when lean, scrum and kanban become appropriate processes, and you can often leverage off-the-shelf products.
When following this method, “HS2 was praised for being ahead of schedule and under budget".
“And this is why I say use appropriate methods. So understand your landscape, apply appropriate methods,” Wardley said.
Agility can’t be a religious rulebook
As time goes on, things will evolve. Once you’ve grown from a fledgling startup, not everything should be about constant deviation and change. As what you’re building is actually becoming a product, it’s time to create an MVP, with a roadmap that can be executed with scrum or kanban, user stories and sprints.
Things that started out as an experiment, may be something you release to the public or decide to even outsource. Then you start experimenting with something else. As technology evolves, so will the processes to build it. Cloud storage is becoming a commodity, but was once something you tested on a department or two before replicating across an organisation.
At Agile Tour London, senior business analyst Snigdha Satti talked about how the very agile, scrum and kanban News UK needed to change some agile processes to meet the unique needs of data science teams. Agile, she defined, in the barest sense, as a “time-boxed, iterative approach” to creating a software project.
Data science is, by nature, also very iterative, however it’s not as tangible as software development. Data science involves a lot of continuous investigation and exploration of data, followed by testing and tuning of an algorithm, which can then lead to more investigation and exploration. On the other hand, without clear agile boundaries like scrum and without any transparency into overall business objectives, the distributed team of News UK data scientists— based in both London and Bangalore—were just saying yes and prioritising tasks in the order received. This meant they weren’t able to properly prioritise work or meet deadlines.
“Overall it was chaos, which led to a lot of demotivation and there was no sense of achievement in the team,” Satti explained, which is why she was tasked with persuading the data scientists to go agile. She began by meeting with them one on one to explain the value and understand their hurdles.
Next, Satti worked with stakeholders to define the business needs and project objectives and created a backlog that helped the team understand the deliverables and hypotheses. The addition of daily standups and retrospectives enabled better teamwork and collaboration. She said the greatest impact came from implementing planning sessions at the start of sprints, allowing each person to focus for two to three week sprints without interruptions.
Finally they tweaked the concept of product demos and retrospectives to better fit this work that doesn’t really have a tangible output. This helped them decide much sooner whether they should go ahead with an experiment or not.
“Because sometimes, before these demos, what would happen is that they would carry on doing it for three or six months, going down the rabbit hole, and then realise that ‘Oh, it’s not going to work.’ And it was such a waste of time and effort,” Satti said.
However, as she said, it’s impossible to implement all aspects of agile in the world of data science because building models is very distinct to feature building. She found they couldn’t take on the agile rituals of:
- Estimations - When a model is first being built, scientists aren’t sure if datasets are clean or if they even have all the datasets needed. Satti said they can create rough timelines of a month or two, but it would be impossible to allot story points.
- Tangible solutions - “There won’t be a button on a website, but you can see that a model is working or that a model has progressed 70% with the existing dataset.”
- Scope and requirements - These can change quite drastically within the sprint, which has to be accepted in data science, while it would be deemed wholly unacceptable in a traditional software sprint.
- Deliverables - You can follow sprints, but you cannot expect a deliverable for each sprint.
In the end, by picking and choosing some agile aspects, without doing it all by the book, Satti saw a significant increase in team productivity and motivation, which in turn increased the stakeholders’ trust in the team.
“I strongly believe that we should use processes for our advantage, instead of being process-led. There are no hard and fast rules here. There was a lot of trial and error that took place, and I feel we should apply whatever works for our teams. We shouldn’t let agile stop our teams from doing any valuable work,” Satti said.
Embrace the agile pick’n’mix
Wardley recognises his is not the most popular idea. Pitching a one-size-fits-all panacea is seductive, which perhaps explains why so many Fortune-500 firms have adopted Scaled Agile Framework (SAFe) across the organisation. But since departments and projects vary, a catch-all approach always ends up being “a method that doesn’t do anything,” as Wardley found with wide-spread enforcement of XP back in 2003.
About 35% of businesses are using SAFe but, as Wardley puts it, “67% of generals bomb hills therefore, what should we do? Bomb a hill. I make this joke because too much is simply copying what other people do.” As we’ve explored before on WTF, magical thinking and Cargo Culting are still surprisingly common in industry.
The importance of mapping is the collective process of making the map, not the artifact itself. By mapping properly, you allow everyone to create a common language, bringing you closer to what Eric Evans refers to as “Ubiquitous Language” in Domain Driven Design. You also allow them to challenge the assumptions you made, since the map depersonalises the challenge; it’s the map that is wrong, not the individual. In the end that map supports the first goal of prioritising people over processes and tools, truly embracing psychological safety. And you find out what works for your team in that moment.
Most importantly, project management is less about the project and more about the people. So talk to your people before you make any widespread declarations or commitments.
Previous article
