“If we flashback circa 2016, Spotify was in hyper growth mode and hiring engineers at a breakneck pace,” Platform Developer Experience Tribe lead Pia Nilsson told WTF, “But when we looked at our developer productivity metrics, they weren’t getting better despite all the new hires. In fact, our onboarding metrics kept ballooning upward which meant new hires weren’t able to meaningfully contribute for quite a while”.
Spotify has a highly autonomous engineering culture, and engineers were already well along the path of migrating from monolith to Microservices. In addition, ”the Platform organisation was inventing a whole set of new tools for our engineers to enjoy”, Nilsson said. But there was a problem— tools were “scattered over a multitude of different sites, and many were hard to find without prior knowledge. The result was a lot of internal fragmentation and chaos”.
In such a situation, you need some sort of centralised solution, but how can you make that work in a highly autonomous culture like Spotify’s? To solve the quandary, Nilsson explained:
“We took the humans-first product development lens we’d apply to any consumer product at Spotify, conducted user research with our engineering teams and asked what was blocking them. Two things came up consistently:
- Constant interruptions and context-switching as a byproduct of growth (people moving teams, filling gaps, new joiners asking questions, etc.)
- People couldn’t find stuff and they didn’t know who to talk to in order to find stuff.
We knew we had to address this problem with a centralised solution, but we also knew that a top-down centralised approach wouldn’t fit within our culture. We also wanted to know more about services than simply where they were deployed. Things like: Is it running in production? Where is the documentation and is it up-to-date? Who owns this service? And who do we contact if something goes wrong?
So at the most basic level, we needed a company-wide repository where someone could find everything they needed but nobody had to use anything. And when we looked, there was no off-the-shelf solution that would work at our size. We had to build it ourselves and we had to make it sticky, user-friendly, and able to scale with our already staggering code base”.
Enter Backstage
Spotify’s solution was Backstage, a developer portal that unifies tooling, services, apps, data, and docs with a single, consistent UI layer. The firm open sourced it in March 2020, and it is now an incubating project under the auspices of the Cloud Native Computing Foundation.
“Spotify built Backstage to tame our software chaos,” Nilsson told us “And that changed everything—aligning a distributed, autonomous culture, and bringing together hundreds of squads, thousands of engineers and tens of thousands of software components. Backstage at Spotify now enables better collaboration, unlocks collective potential, and empowers both individuals and teams to do what they do best”.
“Backstage addresses the kind of everyday complexity that can really bog engineers and their teams down”, Nilsson said, “which then slows your whole organisation down.” She continued:
“Developers have access to more technologies than ever before, which comes with more responsibilities than ever before. Whether working at small or large companies, engineers interface with systems for alerting, monitoring, security, compliance, documentation, Kubernetes, access control, cloud provider controls, data and ML pipelines, mobile apps/components, services and libraries owned by other teams, and more. All of these things come with their own interfaces and ways of doing things—and navigating within those different environments, searching for stuff, and tapping people on the shoulder for help all ultimately result in less time spent actually writing code.
A developer portal built off of the Backstage platform puts everything in one place, with a single interface that is customised to a company’s software product catalogue and a developer’s team context. So instead of switching between all these different tools and dashboards and systems and UIs, there’s just one frontend for all of it—a single pane of glass. Building a central hub for your engineers around your production software and a consistent, cohesive UX for your tech infrastructure, unlocks many different things, including speed, improved collaboration, or even just a smoother day.
Backstage as a product and as a platform is really about creating a better experience for engineers—streamlining their workflows, making it easier to share knowledge, and getting the messy parts of infrastructure out of their way. It enables them to better focus on building business value: innovative products and features”.
Respect the culture
We’ve noted elsewhere in this issue of WTF is Cloud Native that engineering culture is an essential part of DevEx, and in a highly autonomous culture like Spotify’s (or indeed that of the Financial Times) you don’t want to have to enforce standardisation. In Spotify’s case “Instead of working against our culture, we worked with it by creating a marketplace of inner source contribution within Spotify”, Nilsson said:
“This marketplace where developers can contribute solutions to common problems they have, through Backstage’s 180+ (and growing) plugins, and others can adopt them. This is also what makes Backstage so valuable and customizable in an open source context, plugins that extend Backstage’s functionality can come from the community or be entirely in-house/bespoke to your organisation.
Backstage works by helping people to find the right solutions, without mandating them. And this contributes greatly to overall developer effectiveness and happiness—which we take great pains to measure qualitatively and quantitatively”.
The Spotify team uses a number of metrics to assess the value that Backstage brings. “At Spotify, our north star metric was reducing that onboarding time for new joiners”, said Nilsson. “Specifically, we measured as time-to-merge your 10th pull request, which went from 60 days before Backstage to less than 20 days after. We also focus on a few other important proxy metrics for our internal version of Backstage, ranging from feedback from our quarterly developer surveys to more traditional metrics, like daily active users”.
The power of open source
Spotify have also got tremendous advantages from their decision to release it as open source, according to Nilsson:
“Let’s imagine that two years ago we didn’t open source Backstage, but instead we poured the same amount of internal resources into it as we have gotten from the external community. (And, btw, based on the tremendous community engagement so far, that would have been a huge investment and tricky to fund.) It still would not be as good a product as it is today. A diversity of viewpoints and use cases has improved the product, making it more robust and enabling the platform to keep up with the pace of change going on both inside and outside a particular company. Plus, if we had not open sourced, we’d be the only ones using and depending on Backstage. If eventually, a different open source solution emerged, we would have had to migrate to that solution as the community-fed innovation eclipsed our ability to keep pace.
So, we are very much incentivized to make the best platform, not just for us, but for other companies. An example is Backstage Search: We had search features within Spotify’s Backstage but rather than just open sourcing our internal version, the project maintainers asked the community what they needed and collaborated on a version that allows adopters to design and build a search tool that best suits their org’s needs. This approach not only created a better search tool for the community but for us as well. In that sense, the more companies find value in Backstage, the more value Backstage brings back to us at Spotify”.
Alongside daily use by development teams Backstage is gaining increasing adoption amongst teams in data science, machine learning, and design within Spotify. They are using it to create new components, write documentation, discover which squad owns which service, and align to golden paths.
Outside of Spotify, Backstage is also having considerable impact. Amongst the nearly 200 public adopting companies you’ll find UK online challenger bank Monzo, who we spoke to elsewhere on WTF, American Airlines, and FinTech start-up Brex, as well as firms like Expedia Group, HelloFresh, Netflix, Unity, Peloton, Splunk, Wayfair, Zalando, HBO Max, IKEA, and HP.
VMWare is using Backstage to build Tanzu Application Platform’s GUI; third-party software providers like FireHydrant and Snyk are building out their own plugins and adding them to the Backstage open source plugin marketplace, and Roadie is offering hosted/managed service for Backstage.
In all, Backstage now has 1500+ contributors across the globe and more than 60 open source plugins have been contributed. There are more than 6,000 members in the Discord channel, where maintainers are helping contributors and contributors are helping newcomers.
The core maintainers are currently managing 100+ pull requests per week, roughly 80% of them are coming from non-maintainer contributors, and half of that 80% are coming from external adopter companies. “At this point, it's fair to say that the core maintainers are largely focused on driving the open source project’s direction as well as stability and maturity of the core components, while big and small innovations are mostly coming from the community”, Nilsson said.
Previous article
