Enabling Developer Self-Service with Internal Developer Platforms

A concise overview

Even the most advanced organisations struggle to scale their development output when faced with a growing talent shortage. As a result, enterprises and development teams start looking at their infrastructure in terms of Internal Developer Platforms to master those challenges and to accelerate delivery at scale. Leading analyst firms such as Gartner, thought leaders like Martin Fowler, vendors like Humanitec and Upbound, and technology giants such as Apple, Netflix, and Twitter have all picked up on the conversation around Internal Developer Platforms. This review article summarises the different voices, providing a complete lowdown together with practical advice and relevant references. We will also highlight a few recommended focal points for the planning, implementation, and operation of IDP on an enterprise level.

The Internal Developer Platform is an important new concept as it addresses three key challenges in the software delivery life-cycle. It

• Provides environment management, from development to production with integrated CI/CD
• Integrates repeatable patterns that development teams can utilise for their applications
• Reduces the transactional load between development and operations teams

Often, just hiring more engineers is not enough either to solve the problems faced or to support the growth of the business organisation. We will unpack this in a little more detail later.

Cloud computing, containerisation, and microservice architecture provide organisations with a broad choice of internal tools and systems to complete the request for new environments. Internal Developer Platforms are expected to provide development teams with an adequate selection of tools and resources while abstracting away the operational burden. Intuitively or intentionally many development teams notice these benefits, and infrastructure responsibility gravitates towards a small handful of disparate individuals providing such abstraction. Often organisations are unaware they are in fact building an Internal Development Platform or at least variations thereof. Scott Carey from InfoWorld captures this notion describing Internal Developer Platforms as "snowflakes" stating that "no two are the same. Each platform varies from organisation to organisation depending on their stack, culture, codebase, and toolset.

We consider the observed adaptability of an Internal Developer Platform a core element of the concept. It is part of what makes the Internal Developer Platform concept so powerful and attractive for software development organisations. Hence we will avoid academic definitions but highlight the different functions and implementation modes that we see in the literature to facilitate further adaptation.

What is an Internal Developer Platform

The community-driven website internaldeveloperplatform.org offers a great starting point for understanding what an Internal Developer Platform is:

Internal Developer Platforms (IDPs) are configured by Ops teams and used by developers. Ops teams specify what resources startup with what environment or at what request. They also set baseline templates for application configurations and govern permissions. This helps them to automate recurring tasks such as spinning up environments and resources and makes their setup easier to maintain by enforcing standards.

Brendan Kamp and his team at Container Solutions see an IDP "as a system of tools that integrate with each other into a central platform that supports internal development processes. Developers experience it as an 'Internal Platform as a Service' ".

The aforementioned InfoWord article from Scott Carey also highlights the service character of the IDP with regards a development team’s specific preferences:

A good internal developer platform should abstract away infrastructure decisions, enable self-service environment builds, integrate with existing continuous integration and delivery (CI/CD) and deployment processes, and assign role-based access controls, all without a developer ever having to learn YAML.

Value functions of the Internal Developer Platform

The iconic quote from Watts S. Humphrey that "Every business is a software business" is almost two decades old, and has been reiterated and varied upon by many CEOs and strategic leaders including Microsoft’s Satya Nadella. This quote emphasises that any organisation - regardless of their actual industry vertical - needs to provide their software development teams with an environment suitable to solve the key challenges in modern application development on a competitive scale.

The classic supply vs demand problem

Many organisations have reached an unprecedented level of growth with cloud adoption projects, digital-first initiatives, and digital transformations. These initiatives have created a large burden on the operational teams that are traditionally responsible for building and maintaining platforms for the development teams.

Simply hiring more engineers or developers to reduce the overall backlog and distribute the workload more evenly does not solve the issue. Without clear processes and a shared understanding, this leads to a situation where "More Hands" may not be immediately effective as every organisation’s internal technology stack differs; new engineers need time to "fill the gaps” of their skillset before they are able to start adding value to the team.

Reducing cognitive load on teams

Teams involved in the delivery of software often have a lot of tasks and responsibilities to juggle during their daily work. An Internal Developer Platform elevates some of the cognitive load associated with a demanding variety of daily tasks. Cognitive load from context switching is frequently identified as an issue in loss of developer productivity. Developer productivity experts Matthew Skelton and Manuel Pais identify cognitive load from context switching as the main reason why teams act overwhelmed. In their bestselling book Team Topologies, the authors explain this issue exceptionally well: "When the cognitive load isn’t considered, teams are spread thin trying to cover an excessive amount of responsibilities and domains. Such a team lacks the bandwidth to pursue mastery of their trade and struggles with the costs of switching contexts." Likewise the 2020 state of DevOps Report from Puppet states that "Most software developers and operations engineers understand that switching back and forth between two contexts is a huge cognitive drain. There’s a good reason for this, apart from the normal human challenge of context-switching: the details and mindset of each realm are so different."

However cognitive load is rarely considered during organisational or project planning. This leaves the development teams at a massive disadvantage and prevents organisations from productive wins. Internal Developer Platforms provide collections of reusable patterns for teams to consume in a singular point of interaction for their roles. This reduces the amount of context switching and thus reduces the cognitive load placed on the teams.

Autonomy and the tool diversification challenge

Writing on Martin Fowler's blog, Evan Bottcher does an excellent job explaining the challenges faced by development and operational teams in organisations and how an Internal Developer Platform can aid them:

...clearly a characteristic of a good platform must be that it reduces the amount of backlog coupling. The platform must provide services that do not require tickets to be raised and work to be assigned. Self-service is a key defining characteristic for a good platform… The platform should provide a team with self-service access. Specifically, it should allow for self-service provisioning, self-service configuration, and self-service management and operation of the platform capabilities and assets...to find the right balance between autonomous diversification and mandated consolidation, and that is even more difficult to predict upfront.

Allowing teams to deliver faster

Delivering software with higher quality at a higher pace has been a key focus for development teams and the business they support for many years, This has been mostly down to a consumer demand for new features, and competition within the marketplace. A few very well-known examples of the pressure that organisations face within their respective markets are AirBnB upsetting the hotel industry, or Uber and the Taxi Industry. This shows that companies need to be able to react to remain relative in the ever-changing marketplace, this has given rise to Agile and DevOps, companies as early as 2015 started "Kicking the tires on DevOps". 

Since then many companies, both built in the cloud and the more traditional, have become known as "High Performing DevOps Organisations” as coined by Puppet in their annual State of DevOps report. This term takes into account many industry leaders and analysts' opinions as to what these companies do to deliver software faster and outperform their composition in their respective lines of business. Some of the key metrics that are used to evaluate if an organisation can be considered a high-performing DevOps company are: Deployment Frequency/Cadence, Lead Time, Waiting Time, and Mean Time To Repair (MTTR).

In the 2020 version of the State of DevOps report, there was a noticeable trend between these high-performing companies and the adoption of an Internal Developer Platform. Puppet found "a strong relationship between DevOps evolution and the use of internal platforms. Highly evolved firms are almost twice as likely as mid‑level organisations to report high usage of internal platforms and are six times more likely to report high usage than low-level organisations".

Teams involved with an Internal Developer Platform

Internal Developer Platforms are traditionally built and managed by a dedicated team of individuals, who we'll refer to as the platform team. Alongside the platform team, the operations team is responsible for the organisation's technical needs, and the development teams are responsible for delivering software. We will break down each of the teams, their needs, and challenges in more detail below.

Platform team

The formation of the platform team is a key element to the success of the Internal Developer Platform within an organisation, and it is important that the team's mission is well defined and set in stone. By defining the platform team as a separate entity within the organisation, their focus becomes dedicated, and this should reduce some of the transactional load that may come from their prior roles within the organisation. Evan Bottcher again:

"Platform teams build, deploy, monitor, and are on call for the platform components and underlying platform infrastructure... The platform team ideally doesn’t even know what applications are running on the platform, they are only responsible for the availability of the platform services themselves."

The platform needs to adopt a Product mindset in the building and maintenance of the platform. We will delve further into this later in the Blog. 

Operational teams

Traditionally operations teams are responsible for a number of key elements within an organisation's IT infrastructure. Unlike development teams which may have a small number of concurrent projects, operational teams' scope of work is normally far larger, managing multiple environments, databases, networks, cloud infrastructure, and many other elements that make up a software solution. 

Alongside this operations teams are responsible for the organisation’s live environment, where consumers and internal users normally interact with the company on a daily basis. This calls for an interesting balance between enabling development teams and "Keeping the Light on” to all for the business to function. These teams often work on a FIFO (First In - First Out) principle managed by a ticketing system, with outages taking preference over other tasks.

The 2020 state of DevOps report highlights the need for high-performing DevOps organisations to invest in an Internal Developer Platform, to enable developer self-service to reduce project and transactional load on the operations teams. Many organisations start the process of providing an Internal Developer Platform to the teams without knowing it. Organisations will start to standardise on a specific cloud or deploy onto a dedicated container platform like Kubernetes or Openshift. Going down the path of building an Internal Developer Platform, organisations play a game of unblocking dependencies one after the other, in an attempt to improve the delivery of software throughout the life-cycle, "If you need to release faster, you play this game of unblocking impediments and removing bottlenecks and create a strategy to solve this root cause," writes Jan Loeffler former head of platform at Zalando

Gene Kim, the author behind the books "The Phoenix Project” and "The Unicorn Project” provided some useful metrics whilst presenting at AmbassadorFest which Ops team can use to see the effectiveness of their internal platforms. Quoted by Daniel Bryant on Twitter, Kim said:

"A great metric for an ops team is how many 'thank yous' you are receiving from app delivery teams on a regular basis. The ultimate goal is to fix recurring blockers, reduce deployment friction, and make it easy to deliver value to customers" @RealGeneKim at #AmbassadorFest pic.twitter.com/bQfpAPvvPA

— Daniel Bryant (@danielbryantuk) November 17, 2020

Development teams

The core team to benefit from an Internal Developer Platform is the Organisation's development team. Development teams often have the massive task of them bringing new features to market or building internal systems that are core to the organisation's day-to-day business. 

Development teams often need to wait for the operation or DevOps teams to create environments for them to deploy and test their applications on. Without a centralised platform responsible for the CI/CD for a specific application, a development team will often choose what suits them best. This leads to "limited control or audit capabilities across the whole organisation,” explains Jan Loeffler. This generally leads to a massive amount of tool sprawl across the organisation as individual teams adopt and build their own workflows upon the tooling that they either understand best or happen to prefer. Alongside the tool sprawl that this can create it can also lead to massive problems in organisations that have strict regional or PCI requirements. 

At KubeCon North America 2020, there was a large emphasis from many attendees on the need for developer self-service for infrastructure, Dave Sudia said:

In the context of building platforms, application developers are the key customer and usability is a big part of the developer experience. Unless developers have the ability to speedily and safely configure how the platform runs their services — and also configure how applications are released to end-users — then the value of a cloud-native platform will not be fully realised.

How to get started with an Internal Developer Platform

Often teams will have started by formalising and standardising their approach to CI/CD, Infrastructure as Code, and Kubernetes. These technologies form the building blocks of most Internal Development Platforms. Establishing the Platform as a product is a key component of building a successful Internal Developer Platform. 

Evan Bottcher highlights three key prerequisites for building a successful IDP:

• The platform is a product and needs a long-lived and stable product team tasked with both build and run.
• You must be willing to shift some or all of the run responsibility for applications into the application teams and away from centralised operations and support
• Thirdly you must be willing to trade off strict consistency of implementation against the freedom and responsibilities that you’re handing to the autonomous application teams.

Alongside this by establishing the Internal Developer Platform as a product, Micheal Coté in his report on "The Business Bottleneck” highlights why a product approach is a key to success:

A product approach focuses on the full life of the software: is the software useful and does it help the customers and users and thus the business? Everything is oriented around gathering customer and market feedback and changing the software accordingly, ongoing.

Although this has been written for companies whose business relies on delivering software, it is applicable to an Internal product as well, it gives the team a mandate to manage their deliverables as if they were interacting with external customers.

In her Medium article "Product for Internal Platforms” Camille Fournier writes

Whether you are a platform engineer, engineering manager, or PM, it pays to remember that you still need to be customer-focused and strategic about your platform offerings. Without a clear strategy for showing impact and value, you end up overlooked and understaffed, and no amount of cool new technology will solve that problem.

Key components of an Internal Developer Platform

An Internal Development Platform (IDP) is a collection of tools and systems that are closely integrated to provide a consistent experience for development teams. 

These platforms are generally unique as are the teams that construct them, but there are a few components and patterns seen across many different enterprises. The team over at Internaldeveloperplatform.org has curated these into a concise list (Application Configuration Management, Infrastructure Orchestration, Environment Management, Deployment Management, Role-Based Access Control). We will dive into each of these and highlight some of the key areas within them.

Application configuration management

Managing application configuration across multiple different environments can easily become complex, especially when the application is composed of a number of different services and platforms. Internal Developer Platforms typically manage the application configuration the same way as code: by using a source code management platform for maintaining the application configuration a couple of key capabilities are allowed such as Versioning and Change Management. 

As the complexity of the applications grows so does the complexity of the configuration required to run, deploy and manage the application. By expanding the scope of the configuration stored within the SCM it allows for external dependencies to be configured from a central location and versioned/managed alongside the application that it relates to. 

Infrastructure orchestration

Internal Developer platforms need to be compatible with existing and future infrastructure within an organisation. This is done by building upon an extensible platform. An Internal Developer Platform will often have a wide range of integrations within an organisation's toolchain from Pipelines, Clusters/Servers, Networking (DNS, Firewalls, Routing), Registries, Artifact Repositories and more. 

Ideally, the Internal Developer Platform becomes a driver for Infrastructure as Code, allowing for components within a solution to be defined and managed programmatically. Often individual teams, or the DevOps team, have built and deployed their infrastructure through means of Terraform or similar. But integrating the existing scripts into the Internal Developer Platform allows for a catalog of known architectures to be consumed by the teams. 

Environment management

Often the creation of a new environment is a mix of challenges between platform capacity, team bottlenecks, or sometimes even cost constraints. Teams often need to wait days or weeks for an environment even just to test something small. Internal developer platforms approach new application environments in a holistic manner, grouping together all the dependencies for the application to run within a singular environment. This gives the development teams access to self-service environments as they are needed. This massively reduces the bottlenecks and allows the development team to deliver faster.

By utilising this capability of an Internal Developer Platform, The benefits are far greater than development teams that need a new environment for testing. It allows for teams to bring up whole application stacks with all their dependencies as they are needed, and destroy them when they are no longer needed. This is often exceptionally beneficial cloud environments where the usage of the services is based on a consumption model, and having large environments such as performance testing environments idling is often expensive. 

Deployment management

Delivering software to consumers is a key role of any development team within an organisation, the demand for faster delivery is a well-known challenge. The adoption of Continuous Integration and Continuous deployment has given the development to deploy quicker and more frequently, which often comes with its own challenges and problems. 

By introducing an Internal Developer platform into the development life-cycle, it becomes the central point for all automated deployments into the target environments. Internal Developer Platforms become the Automater of automation, responsible for the versioning of deployment, and the associated items surrounding the deployment. This integration is normally accomplished through the usage of webhooks. 

Along with the deployment of the applications and associated elements, Internal Developer platforms often provide a centralised point for all the debugging of deployments, acting as the centralised point for the version of an application allowing rollbacks to be executed from the same point as the deployments.

Role-Based Access Control

Role-Based Access Control (RBAC) is an element that cannot be overlooked or excluded for any platform where multiple teams will utilise a shared platform. An effective RBAC policy will limit the scope the individual teams can interact with the platform, members can also be segregated by their roles within the organisation. 

Alongside managing which teams are allowed to interact with the platform, it also limits the blast radius that unauthorised individuals may gain access to. In their talk at Kubecon EU in 2021 Ellen Körbes and Tabitha Sable, dove into some of the risks associated with having a weakness in an Organisation's Kubernetes RBAC policies. A key quote from the session - "We are made of stars, But your RBAC shouldn’t be”, implying that there shouldn’t be open access to systems or resources within an environment.

Conclusion 

In closing, an Internal Developer Platform is a key element to an organisation's digital transformation and growth. By allowing development teams to focus on building solutions that delight their customers and grow the company in their target market. But as with any internal project within an organisation needs to be measured to understand the impact that it has on the teams using and building it. Internal Developer Platforms are no different in this aspect. 

The largest driver for a successful Internal Developer Platform is adopting a Product Mindset in the creation and management of the Platform, by gathering the feedback from the end-users (The Developers), and incorporating this feedback into the platform as it grows. 

At each step of the journey, it is key to find the next task or problem that is removing the development and operations team’s focus from their deliverables. For operations teams, this gives them the capacity to deal with real business-critical issues, and for development teams it lets them build the features or products that will ultimately grow the business.