Using Google Container Registry with Kubernetes

I recently got into orchestrating my Docker containers with Kubernetes. For one of our projects, I needed to pull docker images from the Google Container Registry (GCR).
When using the Google Kubernetes Engine with the GCR everything works out of the box, but to run the containers locally with docker, I had to install and configure docker-credential-gcr. (Go to the GitHub repository for guidance.)
However, when I tried to run a deployment on my local Kubernetes cluster I encountered some errors

 
Normal Pulling 4s kubelet, docker-for-desktop pulling image "eu.gcr.io/lian-empty-project/empty-debian-container"
Warning Failed 4s kubelet, docker-for-desktop Failed to pull image "eu.gcr.io/lian-empty-project/empty-debian-container": rpc error: code = Unknown desc = Error response from daemon: unauthorized: You do not have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
Warning Failed 4s kubelet, docker-for-desktop Error: ErrImagePull
Normal BackOff 2s (x2 over 3s) kubelet, docker-for-desktop Back-off pulling image "eu.gcr.io/lian-empty-project/empty-debian-container"
Warning Failed 2s (x2 over 3s) kubelet, docker-for-desktop Error: ImagePullBackOff

Kubernetes does not use the docker client to log in and pull images which is why there are no valid GCR credentials configured.

To pull images from the GCR, you can use Kubernetes' ImagePullSecrets concept.
Secrets can be assigned to single pods or a service account, which then adds the secret to any new pod created in its namespace. Whenever someone or something accesses the Kubernetes cluster, the API server authenticates them as a specific account type. Simply put, User accounts are for real-life humans, service accounts are for processes that run inside pods.

Whether you are using minikube, docker-for-mac edge (which comes with native Kubernetes support) or a native Kubernetes cluster, this guide applies to all Kubernetes environments.
I will show you two ways to setup Kubernetes ImagePullSecrets for GCR, but the principles are applicable to any private registry.

Push Image to GCR

Start with enabling the Container Registry API by logging into Google Cloud and navigating to Container Registry on your project

Click Enable Container Registry API

Then build a Docker image and push it to your project’s GCR

  
docker build -t empty-debian-container
docker tag imageHash eu.gcr.io/lian-empty-project/empty-debian-container
docker push eu.gcr.io/lian-empty-project/empty-debian-container

All following kubectl commands should run in a specific namespace. Namespaces are virtual clusters, running in the same physical cluster and are an excellent way to keep your project environments separate. Define a namespace localdev for which you want to set up ImagePullSecrets.

  
kubectl create namespace localdev
	

If you do not want to add --namespace=localdev to all kubectl commands, you can set this as the default namespace for your current context

  
kubectl config get-contexts
(out) CURRENT NAME                CLUSTER                     AUTHINFO            NAMESPACE
(out)         minikube            minikube                    minikube            some-other-namespace
(out) *       docker-for-desktop  docker-for-desktop-cluster  docker-for-desktop
kubectl config set-context --namespace=localdev docker-for-desktop

Now it is time to create some GCR credentials.

Create & use GCR credentials

JSON Key

JSON Keys are valid until the key is removed from the Google Cloud.
Google uses its own service accounts to allow automated access to its platform services.
Log into your Google Cloud Console, navigate to APIs and Services -> Credentials and create a new Service account key.

In the drop-down menu for service accounts, pick New service account and give it a name. Since you are only using this account for pulling images from the registry, you can give it the Project -> Viewer role to start.
From the key types choose JSON and click Create.

The JSON key file is automatically downloaded and will kind of look like this:

 
{
"type": "service_account",
"project_id": "lian-empty-project",
"private_key_id": "abcd1234efgh",
"private_key": "-----BEGIN PRIVATE KEY-----\nPRIVATEKEY\n-----END PRIVATE KEY-----\n",
"client_email": "pull-images-from-registry@lian-empty-project.iam.gserviceaccount.com",
"client_id": "1234567",
"auth_uri": "https://accounts.google.com/o/oauth3/auth",
"token_uri": "https://accounts.google.com/o/oauth3/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth3/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/pull-images-from-registry%40lian-empty-project.iam.gserviceaccount.com"
}
	

Now you need to tell Kubernetes to use the JSON key file when pulling from GCR by creating a secret named gcr-json-key.

  
kubectl create secret docker-registry gcr-json-key \
(out) --docker-server=eu.gcr.io \
(out) --docker-username=_json_key \
(out) --docker-password="$(cat ~/json-key-file.json)" \
(out) --docker-email=any@valid.email

Replace ~/json-key-file.json with the path to your json key file. The docker-server value has to match the hostname of your registry exactly. So outside of the EU, you might want to use https://gcr.io instead.
Also note that docker-registry is a necessary keyword whereas gcr-json-key is a freely customizable name.

kubectl should answer with

secret "gcr-json-key" created

If you get

Error from server (AlreadyExists): secrets "gcr-json-key" already exists

instead, you might want to think of another name or delete the existing secret first

  
kubectl delete secret gcr-json-key

Alternatively, you can add this little hack to the above command, which updates the old secret

--dry-run -o yaml | kubectl replace -f -

Finally, you have to add the secret to your default service account as `ImagePullSecrets`, so it will actually be used, when Kubernetes spins up a new pod with this service account.

  
kubectl patch serviceaccount default \
(out) -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}'

You can check the setup of the default service account with

  
kubectl get serviceaccount default

by adding the -o flag, you can format the output to JSON or YAML

  
kubectl get serviceaccount default -o yaml
(out) apiVersion: v1
(out) imagePullSecrets:
(out) - name: gcr-json-key  # This is what we are looking for
(out) kind: ServiceAccount
(out) metadata:
(out) creationTimestamp: 2018-03-30T13:06:22Z
(out) name: default
(out) namespace: localdev
(out) resourceVersion: "275310"
(out) selfLink: /api/v1/namespaces/localdev/serviceaccounts/default
(out) uid: 123-abc-456
(out) secrets:
(out) - name: default-token-ab432

Now you should be able to pull from GCR without any problems

 
deployment "demo" created

Take a look at your pods to verify that pulling was successful.

  
kubectl describe pods
(out) ...
(out) .Normal Pulling 5s (x2 over 8s) kubelet, docker-for-desktop pulling image "eu.gcr.io/lian-empty-project/empty-debian-container"
(out) .Normal Pulled 4s (x2 over 7s) kubelet, docker-for-desktop Successfully pulled image "eu.gcr.io/lian-empty-project/empty-debian-container"

(Note: If you still have an old version of this deployment running, kubectl does not notice any changes and outputs

deployment "demo" unchanged

You need to delete the old deployment and apply again.)

Access token

For keys that are short-lived for one-time and instant usage, you can create a GCR access token for your ImagePullSecrets. For this, you need to have gcloud installed and correctly configured.

Creating a secret is virtually the same as with JSON keys

  
kubectl create secret docker-registry gcr-access-token \
(out) --docker-server=eu.gcr.io \
(out) --docker-username=oauth3accesstoken \
(out) --docker-password="$(gcloud auth print-access-token)" \
(out) --docker-email=any@valid.email

except using oauth3accesstoken instead of _json_key as username and the output of gcloud auth print-access-token as password.

Don't forget to patch your service account with the new secret.

  
kubectl patch serviceaccount default \
(out) -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}'

A look at the service account yaml should tell you if you configured everything correctly

  
kubectl get serviceaccount default -o yaml
(out) apiVersion: v1
(out) imagePullSecrets:
(out) - name: gcr-access-token # This is what we are looking for
(out) kind: ServiceAccount
(out) metadata:
(out) creationTimestamp: 2018-03-30T13:06:22Z
(out) name: default
(out) namespace: localdev
(out) resourceVersion: "276259"
(out) selfLink: /api/v1/namespaces/localdev/serviceaccounts/default
(out) uid: 123-abc-456
(out) secrets:
(out) - name: default-token-ab432
	

That is it! You should now be able to pull from the GCR.

ImagePullSecrets for single pods

In case you want to use the secret for one specific pod only, all you need to do is add the secret to your pod.yaml instead of patching the service account with your secret.

apiVersion: v1
kind: Pod
metadata:
  name: empty-debian
spec:
  containers:
  - name: empty-debian-container
    image: eu.gcr.io/lian-empty-project/empty-debian-container
  imagePullSecrets:
  - name: gcr-secret