In the last two months, I've worked together with Chef to evaluate Habitat from a cloud native developer perspective. This is the last blog in a series of three where I'll share my experiences. The first blog is about the build system. The second blog describes how to run supervisors on top of Kubernetes. In this last post I’m explaining how to run a HA Redis with Habitat on top of Kubernetes.
Before we jump to the part that is about Habitat, I’d like to quickly introduce the relevant parts of Kubernetes. Then I’ll discuss some limitations of the build-in concepts of Kubernetes with relation to Self Organisation and show how Habitat compares to that.
Kubernetes is an orchestration engine to run (micro)services. It makes decisions on where to run which workload. Developers write manifests to declaratively specify how their (micro)services should be deployed, which in turn is interpreted by Kubernetes.
The basis in Kubernetes is the Pod, the minimal unit that can be deployed. Pods do not have any complex behavior on their own; if a Pod dies, it’s gone and nothing happens.
However, there are higher level objects like ReplicaSets, which has a template for the Pods it should manage and some configuration on how many replica’s of the same Pod it should run. It makes sure that the correct number of replica’s is up by starting or killing pods.
The ReplicaSet knows which Pods it’s responsible for by selecting Pods by label.
Every object in Kubernetes can have multiple labels, which are arbitrary key/value pairs (with some constraints on the format of keys).
Another example of how labels are used are Services. Services are used to provide service discovery: binding a name to a set of Pods that implement this service. Services add Pods by selecting on labels as well.
For stateless services, one should use a Deployment, which owns a ReplicaSet. When you want to deploy a new version of your service, you’d update the Deployment, which would create a new ReplicaSet and do a rolling update by scaling down the original ReplicaSet, and scaling up the ReplicaSet of the new version.
Distributed services like Redis that manage state are harder. Especially when you’d have heterogeneous nodes. For example, Redis supports having one leader and multiple follower nodes, where the leader only accepts writes, which are then replicated to followers. ReplicaSets can only create identical Pods, and therefore using them directly with Redis will not work; we’d either have only followers or leaders.
Kubernetes’ answer to this problem is Operators. They allow you to create a custom resource (like Deployment or ReplicaSet), which is backed by a central management container that will take care of your stateful service. It contains all the logic in one central place to create/monitor the Pods in the cluster and make sure that they are set up correctly.
Habitat attacks this problem from the other side: each Supervisor instance gossips with each other to reach consensus about which pod should have which role. Habitat supports several topologies, but for Redis we’ll be using the Leader/Follower topology, which is a perfect fit for this purpose. Selecting this topology makes the Supervisor take care of the actual leader election process. It will trigger hooks and re-generate the configuration of the Redis service if the topology changes.
Even better, the Habitat package core/redis which is managed by Chef, actually supports setting up a HA cluster by default! This allows us to simply create a ReplicaSet, where the Habitat Supervisors will take care of assigning the leader and follower role to the pod.
However, this information should still be exposed to Kubernetes; we want to apply a label role=follower or role=leader to the Pod to make use of the Service abstraction in Kubernetes.
The idiomatic way to implement this is to create a side-kick container that will run in the same Pod as the redis container. This side-kick container inspects which role the Supervisor has chosen and then applies the correct Kubernetes label.
During the implementation, I encountered some issues with the provided hook not firing as I would expect it to. Given the constraints on time, I opted to create a simple side-kick container that polls the HTTP API endpoint provided by the supervisor and use that to compute the role of the pod. The code for the side-kick container can be found on GitHub.
During this project I encountered several issues, which Chef is picking up to solve. Habitat is positioning itself to be a an easy way to run stateful services on bare metal, cloud VM’s or orchestrators, as I have demonstrated in this blog post series.
We at Container Solutions strongly believe that lowering (operational) knowledge from one central place to the services themselves is a trend that is only getting started.
Habitat’s Supervisors, just like the Autopilot Pattern are steps in this direction.