I am just back from ContainerSummit where I was working with Casey Bisson from Joyent. We got talking about how things can work on one machine but not on another, say, in production. This is known as the ‘it worked on my machine’ problem. However, with a bit more thought you can see that it should really be known as the ‘it orchestrated on my machine’ problem. As of today, this has a meme:
In this blog I’d like to do the following:
-
Define what we mean by orchestration.
-
Explain what minimesos is and how it deals with the ‘it orchestrated on my machine’ problem.
-
Explain what ContainerPiliot is and how it deals with the same problem.
Orchestration
When we think of software orchestration all we really mean is that we have automated some operations tasks. Thus, rather than manually deploying an application to a production machine, we get an orchestrator to do it.
When we develop our applications locally we either orchestrate manually or with tools that we don’t find in production. This is why an application orchestrates locally but not on a different machine. The way to fix this is to use the same orchestration tools all the way through the pipeline.
Minimesos
Last summer when we started to ramp up our work with Mantl, we started to run headfirst into the ‘it orchestrated on my machine problem’. As well as developing applications that were composed of microservices we were also building custom Apache Mesos frameworks. We didn’t want to get all the way to a machine in the cloud before we started finding problems. Rather, we wanted to bring the cloud to our local machines. We looked for tools to help. We found none. This is how minimesos was invented.
Minimesos started life as a few scripts that would replicate an Apache Mesos cluster on our local machines. We checked our code in and the continuous integration server would recreate the cluster for testing before deploying it to the cloud. We essentially solved the ‘it orchestrated on my machine’ problem by creating a portable orchestrator that travelled with the code.
Later, as minimesos matured, we added support for JUNit testing. An API for different languages. Finally, we added Weave Scope for visualisation and exploratory testing. Minimesos is now used by a number of companies who are (of course) very happy with it.
ContainerPilot
ContainerPilot, from Joyent, also solves the ‘it orchestrated on my machine’ problem. Unlike minimesos, however, which is an outside-in orchestrator, applications written with ContainerPilot are orchestrated inside-out.
ContainerPilot overview.
Applications running on ContainerPilot have three crucial differences to the sorts of applications that you’d build with minimesos. These differences are: an agent embedded in the container; an adaptation layer; a global store. Let’s go through the diagram one piece at a time.
-
Inside the cloud you can find a number of microservices that collaborate to form an application.
-
Around the microservices you have a light blue ‘glue’ that represents a connection to a global store.
-
Inside the container you find your microservice process.
-
Wrapped around the microservice is an adaptation layer. The adaptation layer is where a number of lifecycle rules live. These rules are defined by a programmer and are responsible for orchestrating the application once it is deployed.
-
Finally, you have the agent, which is responsible for mediating between the adaptation layer and the global store.
ContainerPilot, which is based on the AutoPilot pattern, uses smart agents. Microservices that are created in this way are not passive, they don’t just sit around waiting for an external orchestrator to take care of them. Rather, they react both to each other and their environment. This means that applications using ContainerPilot can, for example, automatically recover from failure, scale up and down, and execute master-slave handshakes.
ContainerPilot requires platform support. This is currently not really a problem, however, as Mantl, Triton and Swarm all do support it. (Minimesos, by comparison, only needs Docker for it to work.)
Conclusion
It simultaneously became clear to many people in the industry that we are not really solving the ‘it worked on my machine’ problem but rather the ‘it orchestrated on my machine’ problem. One pattern that is emerging from this idea is portable orchestration. Minimesos achieves this by travelling with the application. ContainerPilot does it by embedding orchestration instructions within the application. Minimesos’ outside-in orchestration is ‘old-school’, is based on how we orchestrate applications now. ContainerPilot is new-school, is based on the how we might orchestrate applications in the future.
Previous article
