Once your shiny new Kubernetes cluster is up-and-running, one of the first things you'll want to add is a local registry for storing private images. This is typically achieved using the official Kubernetes registry addon. Unfortunately, the official addon has a few shortcomings, especially with regards to security. In this post, I'll describe these shortcomings, how they can be addressed, and point to a tool we've built that can help when setting up a registry.
Registry Addon Shortcomings
So what are the problems with the registry addon? The main one is that exploits the "localhost registry logic" in the Docker registry to avoid the need to configure a certificate for the registry. It does this by adding a proxy service on each node which forwards "localhost:5000" to the actual registry service. This tricks Docker into thinking that all communications are internal to the host and there is no need to use TLS for transfers - without this step Docker would refuse to push images to the registry. Whilst this trick is effective, it does mean that transfers to and from the registry are taking place unencrypted. This isn't quite as bad as it sounds at first - remember that the communications are limited to the cluster and shouldn't be going out over the internet at large - but it is still a compromise, and as such it requires some consideration and will be unacceptable in some situations.
Further, the proxy works by installing an instance of haproxy on each node (the kube-registry-proxy image), which seems an unnecessarily heavy solution, when a similar effect could be achieved using the Kubernetes NodePort functionality.
Finally, accessing the registry from a remote client (i.e. a developer's laptop) requires having kubectl installed and setting up a proxy. This process works, but requires the installation of kubectl and precludes the developer from running a registry on localhost:5000.
A Better Way
So how can we do things better without too much work? If you control a domain name that can be forwarded to the registry, your best bet is to obtain a public certificate for the registry that will allow secure access from any location. This can be done for free using a service such as let's encrypt and the Docker docs have plenty of info on how to configure it. You'll then need to set up a Kubernetes ingress service to forward traffic to the registry. However, in some cases you may find you can't, or don't want to use a public domain for your registry, due to networking, infrastructure or political issues. In this case, the solution is a bit more complex:
1) Create your own self-signed certificate, for instance using openssl. You can follow the instructions on the Docker website.
2) Distribute copies of the public part of the certificate to all clients that need a copy. This should include all cluster nodes to allow them to pull images. Using Kubernetes secrets to store the cert can be helpful here.
3) Set up network routing so that traffic is forwarded correctly to your registry. A simple and effective (but somewhat hacky) solution is to edit /etc/hosts to add the appropriate route.
An Even Better Way?
As this is a bit complex, we've written a tool that will perform all the above steps for you. Assuming you have kubectl installed and pointing to your Kubernetes cluster, running reg-tool will take care of creating a certificate, installing an instance of the registry and routing via /etc/hosts. It can also be used to easily fetch and install a copy of the certificate on developer's laptops, allowing quick and easy access to the registry without the need for a proxy.
Previous article
