One of the main drivers behind Kubernetes’ success is the ability to manage containerised workloads and services by specifying a declarative configuration and letting the system make it happen. This effectively automates and abstracts away all those tedious and error-prone step-by-step deployment tasks usually known as ‘operations’.
In the past, developers would define workloads and services in software that, once built and packaged, required specific knowledge and careful execution in order to be deployed in production by specialised human operators. Cloud Native techniques and platforms like Kubernetes changed this dynamic forever: while the platform takes care of the nasty bits, developers can focus on building resilient, Cloud Native code, greatly reducing the need for operations.
But is this always true? At Container Solutions we often help clients build, run, and support their business' digital infrastructure. Whether it is a large bank or a cloud provider, chances are that Kubernetes is part of that infrastructure.
While things are significantly easier if your Kubernetes workloads are run with the help of cloud providers like GKE or AKS, some of our clients have special needs. Some in fact are so large that they run their own Kubernetes clusters, or even manage commercial automated deployments of Kubernetes clusters as a service.
In these cases, the reality is that operations are not going away. On the contrary, suddenly, someone inside these organisations need to think—a lot!—about Kubernetes operations and upgrades. Luckily, Kubernetes adheres to strict development principles and new releases follow a well-established process that makes operations easier.
In this blog post, we will explore some aspects of Kubernetes versioning that every Kubernetes administrator may be wondering about but never dared to google.
Every system administrator knows that an important part of day-to-day operations is keeping systems up-to-date. It might seem obvious, but why bother with upgrades in the first place?
Upgrading is important for several reasons. New versions of Kubernetes often introduce new functionality or improvements in performances and scalability. Additionally, bug-fix releases often cover security-related vulnerabilities that might compromise the security of the entire Kubernetes cluster if left unpatched.
The Kubernetes release scheme uses the Semantic Versioning Specification, which prescribes that, given a version number Major.Minor.Patch, you should increment the:
A new version of a software is backwards-compatible, if the changes it introduces don’t break external components that rely on interfaces defined in older versions.
Staying on top of Kubernetes releases can be difficult:
If you are hosting Kubernetes clusters, you should support the latest three Minor releases and, for each, the last two or three stable Patch releases.
Each Minor version release might receive Patch version upgrades. Patch releases address critical bug fixes to the latest Minor version. These include security vulnerabilities, problems affecting a large number of users with no workarounds, or blockers for products based on Kubernetes.
No incompatibilities should be introduced between Patch versions of the same Minor version.
With such an aggressive release cycle and if you manage several Kubernetes clusters, it might be very beneficial to automate the upgrade process as much as possible. Thankfully, Kubernetes is designed to be upgraded in a rolling release fashion:
For example, a v1.3 master should work with v1.1, v1.2, and v1.3 nodes, and should work with v1.2, v1.3, and v1.4 clients.
Minor version upgrades are generally safe. In other words, upgrading from one Minor version to the next one should work without downtime or inconsistent states. However, the Kubernetes project does not recommend upgrading more than two Minor releases at a time. And you should run only the latest Patch release of any given Minor release.
GKE supports at least two Minor versions at any given time. New GKE clusters are created with a default version, a stable release of a recent Kubernetes Minor version or Patch release. Versions newer than the default are also generally available on a weekly basis.
Google upgrades cluster masters and nodes automatically by default. This includes: OS base image, the Docker daemon, kubelet and kube-proxy.
Google ensures that master and nodes versions are in sync. Google Cloud documentation has detailed information about automated upgrades and specific versions release notes and timelines.
AKS supports four Minor versions of Kubernetes: the current Minor version and three previous Minor versions. Each Minor version also supports two stable Patches.
When a new Minor version is introduced, the oldest Minor version and Patch releases are retired.
AKS aims to certify and release new Kubernetes versions within 30 days of an upstream release, subject to the stability of the release.
The Charmed Kubernetes distribution published by Canonical uses snap channels to manage version of Kubernetes. Upgrades can then be performed by selecting the right channel and running the upgrade action using juju.
I couldn’t find a specific upgrade policy published by Canonical. However, release notes seem to follow Kubernetes releases in a timely fashion.
The upgrade does not include some of the components of the Kubernetes clusters used by the Charmed Kubernetes distribution, such as: Docker, Etcd, Flannel and easyrsa.
See Canonical upgrading documentation for additional details.
According to its documentation, Digital Ocean Kubernetes supports Patch-level only upgrades that can be triggered manually or automatically during a custom maintenance window by enabling automatic upgrades for each cluster.
Minor version upgrades are not supported. The recommendation is to migrate your workload to another updated cluster.
The upgrade process takes care of the master nodes first, which entails a few minutes downtime for the Kubernetes API and then the nodes in a rolling fashion, one worker pool at a time.
For security-related issues, Patch level upgrades may be forced (even if automatic upgrades is not enabled) during specified maintenance windows and with advance notifications.
Amazon EKS supports only specific versions of Kubernetes. At the time of this writing, for newly created clusters these are: 1.13.8, 1.12.10, 1.11.10.
Amazon EKS is committed to running at least three production-ready versions of Kubernetes at any given time, with a fourth version in deprecation (currently 1.10).
When a Minor version gets deprecated, Amazon EKS clusters running the version targeted for deprecation will begin updating to the next Amazon EKS-supported version of Kubernetes.
EKS has an additional Amazon EKS platform version of the form ‘eks.n’ that specifies details of the cluster’s capabilities. When a new Kubernetes Minor version is available on Amazon EKS, the initial Amazon EKS platform version for that Kubernetes minor version starts at eks.1.
Automatic upgrades of existing Amazon EKS platform versions are rolled out incrementally, and might take some time.
Clusters are always created with the latest available Amazon EKS platform version for the specified Kubernetes version.
Downgrading is not recommended in general. For example, GKE allows only Patch downgrades, but not Minor downgrades. In short: just don’t.
Hope this proved helpful. It's good to know when K8s upgrades happen, and what to do about them when new versions arrive.
Photograph by Ross Findon/Unsplash
Kubernetes is a powerful new open source operations tool for enterprises to speed up deployment, improve resilience, and handle more scale. Our training allows you to take full advantage of it and the range of open-source tools that surround and support it. Click below to see more: