The Docker blog has stated that the Open Container Project (OCP) will enable the container ecosystem to ‘focus on innovation at the layers that matter’. We agree, and we also think the same is true of Mesosphere’s Datacenter Operating System (DCOS) in respects to innovating around large-scale mixed deployments of microservice applications and data processing workloads. Continue reading to learn why you probably shouldn't be 'yak shaving' within your deployment infrastructure when creating your next software system...
Container Solutions are both Docker partners and also big supporters of open standards, and so we were doubly excited by the announcement of the Open Containers Project (OCP) at DockerCon last week. For those of you that haven’t had time to catch up on the news about the OCP, it is a new open standard for container technology, and is supported by Docker, CoreOS and a broad coalition of container technology luminaries and vendors - you can find out more in my recent InfoQ OCP news article. Firstly, we have to say kudos is due to Docker for their technical excellence in producing what has become the de facto container technology, and also kudos to CoreOS for highlighting the benefits of an open standards and for driving the conversation forward.
Secondly, there was so much goodness talked about at DockerCon, that it could be easy to ignore the impact that the announcement of the OCP will have on our industry. The obvious benefit to an open standard for container technology is the preservation of the much talked about values of containers, such as composability and simplicity - both core tenets of software development and delivery.
"We like simple, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators. A simple runtime that is well specified that can call and manipulate a well documented, standard format speaks to this view."
However, an even more important point mentioned in the Docker blog - and it’s easy to initially underestimate the impact of this - is that this will enable container technology vendors to focus on the things that will add the most value:
"[T]his effort will allow the ecosystem as a whole to focus on innovation at the layers that matter, rather than wasting time fighting a low level standards war.
The Docker Blog (emphasis mine)
Focusing innovation where it matters is key in any organisation in order to create value, and this is especially true within a technology-based organisation, where we are literally spoilt for choice for layers in which to apply innovation. Dan North has recently been talking about how the goal of software development is ‘to sustainably minimise lead time to business impact’, and for anyone familiar with lean principles this will resonate deeply. I’ve also been (re-)reading the fantastic ‘Lean Enterprise’ by Jez Humble, Joanne Molesky and Barry O’Reilly, and a large part of this book is dedicated to providing guidance on how enable your organisation to rapidly innovate and respond to changing market conditions, customer needs and emerging technologies when building software-based products.
On related topic, anyone who has worked within the software development industry for a few years will have experienced ‘yak shaving’ when working with new technology, or simply when glueing things together. Don’t get me wrong, some of this is inevitable, particularly if you are an innovator or early adopter (check out my notes from Bryan Cantrill’s O’Reilly Software Architecture conference talk: “The Peril and Promise of the Early Adoption of Container Technology”). However, this kind of exercise can take time away from the activities that add value to the organisation, or you as an individual.
A lot of the work we do at Container Solutions is firmly based within the sphere of innovation and early adoption, and we’ve become highly efficient at minimising the need to shave any of our favourite bovine friends. Inspired by lean and agile principles, we recognise the importance of enabling rapid build-measure-learn cycles and minimising waste. Adrian Cockcroft has talked a lot about this in his recent presentations, such as "when developers operate, and operators develop", and he and several other industry luminaries are suggesting that enabling continuous delivery with containerised microservices can really help with both fast-feedback and reducing wasted effort.
Moving back to the initial topic of the OCP, the same decision of where to focus innovation applies also to the container technology vendors themselves, and here the OCP will allow innovation to occur at a layer that benefits us, the users of this technology, the most. There are a lot of bright minds working at Docker and CoreOS, and imagine the amount of effort that could have been wasted if we had been forced into a low-level container standards war. My gut feeling is that if this did happen, then we wouldn’t be short of yak fur…
Changing subject slightly, at Container Solutions we also work closely with Mesosphere, creators of the Datacenter Operating System (DCOS), which is based upon the Apache Mesos cluster manager. Apache Mesos originated out of work conducted at Twitter and UC Berkeley, and famously powers the datacenters behind Twitter, AirBnB and Apple’s Siri. Mesos is built using the same principles as the Linux kernel, only at the level of abstraction of the datacenter. The Mesos kernel runs on every machine and provides applications (or scheduler ‘frameworks’ in Mesos parlance) with API’s for resource management and scheduling across entire datacenter and cloud environments.
We’ve been using Mesos for quite some time at Container Solutions (and within our sister company, OpenCredo), and we have several clients running this very successfully in production. The ability to treat your datacenter as a pool of compute resource where it is easy to run a mix of application/microservice workloads (via Marathon or Kubernetes, and using Docker), batch jobs (Chronos) and data crunching (Hadoop or Spark) is very liberating. By using a technology like Mesos, which has many framework/deployment options, you don't get the vendor lock-in that can (sometimes inadvertently) happen when leveraging a Platform-as-a-Service (PaaS). You are also working with a technology that provides primitives at a higher-level of abstraction than is typically found within Infrastructure-as-a-Service (IaaS), and so this limits the need for bespoke component assembly and configuration. However, using Mesos as it comes ‘out-of-the-box’ can be quite intimidating, particularly for a traditional operations team. This, in a nutshell, is the pitch of Mesosphere’s DCOS - the DCOS enables the wide array of deployment options, scalability and high availability provided by Mesos to be used by anyone, (almost) regardless of experience or operational skills.
Mesosphere’s DCOS provides a nice web UI, and also a CLI, and installing applications within a DCOS-powered datacenter should be as simple as installing applications locally with a package manager (think apt-get or brew). Once a Mesos framework that manages long-running tasks has been installed (Marathon, Aurora, Kubernetes etc), it is then easy to deploy microservice applications packaged as Docker containers. Scaling applications and dealing with underlying infrastructure failures will be handled by Mesos and the associated frameworks. For anyone who has worked with deploying and managing large-scale applications this sounds revolutionary - and it is (check out this Mesosphere case study with Yelp!). It is relatively early days for the mainstream adoption of this technology, and there are still a few rough edges to be worked around, but we and (more importantly) our clients are already seeing the benefits of Mesos. One of the repeated observations of developers using Mesos is that it minimises time spent yak shaving when provisioning infrastructure, deploying applications and scaling, and ultimately frees up time to focus their skills with innovating on more interesting things...
For many organisations innovation is the key to success, both to the bottom line and also in terms of staff engagement. However, innovation must be focused like a laser beam to truly see the benefits. Just look at Apple’s explosion within the mobile hardware space, Facebook’s rise to success in the social network domain, or Cisco’s dominance of the networking hardware industry - all of these companies have laser-like innovative powers. The challenge with software development and delivery is often in knowing where to focus innovation in order to get the highest return on investment (and have the most fun!).
I believe the Open Container Project and Mesosphere’s DCOS have made this decision easier for a lot of us who are creating software. No one can argue of the disruptive force of container technology, cloud platforms and lean/DevOps methodologies, but integrating all of this into an organisation can be challenging. We think innovating in the container and datacenter space is a lot of fun, but it’s not for everybody. Many organisations and people are still struggling to fully embrace agile, lean and DevOps, and ultimately they just want to build cool applications that solve interesting business or social problems. These people don’t want to focus their energy innovating within the application deployment fabric - the layers that matter the most for them sit on top of container and datacenter technology - and this is where the power of the OCP and the DCOS shine.
So put aware that yak shaving kit - thanks to the OCP and DCOS, innovating for fun and profit with software applications that will be delivered via large-scale container deployments just got a whole lot easier...