I’ve been playing around with the Kubernetes Service Catalog and in this blog post, I’d like to share some of the fun. First things first: what is this Service Catalog thing? Here’s what the documentation has to say about it:
Service Catalog is an extension API that enables applications running in Kubernetes clusters to easily use externally managed software offerings, such as a datastore service offered by a cloud provider.
Quite, obvious, no? Well, not really. Actually, we need to grasp several concepts in order to understand how Service Catalog makes Kubernetes ever more powerful. Let’s go briefly through them.
A large portion of the code powering most modern cloud native applications is written by someone else. Even in enterprise environments, where trust in the origin of your code is typically of crucial importance, adoption of third party software is an established practice.
Some of this software is so popular that several companies developed a business model that revolves around productizing, managing and offering it as commoditized products. These products are often called Services.
Through public APIs, companies like Amazon, Google and Microsoft offer message queues-as-a-service, databases-as-a-service, storage-as-a-service, logging-as-a-service, and the list goes on. All of these APIs make adding advanced capabilities to any application a breeze.
The explosion of third party service offerings and the simplification given by the APIs to manage them is certainly making our application more interesting and reliable. In fact, managing several services for several applications rapidly becomes an operational nightmare: how do you handle credentials to the third party service providers? How do you track used and unused services? How do you inject endpoints and credentials for the services in your application?
The answer to these questions is the Service Broker. A Service Broker is a piece of software that implements the Open Service Broker API that manages the lifecycle of Services. Its job is to create new Service Instances and handle their associations to one or more applications by generating Service Bindings. Typically, a Service Binding will also provide credentials to the application that makes use of the Service.
Each Service Broker exposes a definition of the Service it offers and the relative Plans associated with it. For example, a Service Broker could offer a memory cache service with different memory sizes, each one associated with a subscription plan with different costs involved. Here is an example:
(out) service plans description
(out) rediscloud 100mb*, 250mb*, 500mb*, 5gb*, 10gb*, ... Enterprise-Class Redis for Developers
The collection of these service definitions exposed by the Service Brokers it is effectively a Service Catalog, available on your cloud platform for your application to use.
All of these concepts are generic enough to be encapsulated in a well defined API and that is exactly what the Open Service Broker API is. It was launched in 2016 by the Cloud Foundry Foundation with the goal of creating a standardized, structured way to connect any service to any application on any cloud platform.
We can map the concepts introduced by Open Service Broker API to the API Resources provided by the servicecatalog.k8s.io API in Kubernetes:
(out) Open Service Broker API Kubernetes servicecatalog.k8s.io Kinds
(out) Platform Kubernetes
(out) Service Broker ClusterServiceBroker
(out) Service ClusterServiceClass
(out) Plan ClusterServicePlan
(out) Service Instance ServiceInstance
(out) Service Binding ServiceBinding
Platform Kubernetes
Service Broker ClusterServiceBroker
Service ClusterServiceClass
Plan ClusterServicePlan
Service Instance ServiceInstance
Service Binding ServiceBinding
I wanted to see the Kubernetes Service Catalog in action, so I decided to give it a shot. At one of our clients, we set up a CI/CD pipeline in which each feature branch was deployed in a separate namespace and a bunch of smoke tests were run against it. Each feature deploys the entire platform, including backend services like databases.
This seemed like the perfect scenario to test some of these ideas. I wanted to be able to provision a database for each feature branch without having to worry about credentials and their lifecycle. To make things a little spicier, I am going to host all the Services, Service Provider and Service Broker on the same Kubernetes cluster!
Here’s a diagram of what we want to achieve:
To keep things simple and reproducible, I used minikube 0.25.2 with RBAC and CSR siging enabled.
minikube start --container-runtime docker \
(out) --memory 8192 \
(out) --cpus 2 \
(out) --extra-config=apiserver.Authorization.Mode=RBAC \
(out) --extra-config=controller-manager.ClusterSigningCertFile="/var/lib/localkube/certs/ca.crt" \
(out) --extra-config=controller-manager.ClusterSigningKeyFile="/var/lib/localkube/certs/ca.key"
So we want to provide our Applications with SQL Databases services. To do this I decided to use one of my favorite open source project, CockroachDB. Luckily, there is a Helm chart that makes deploying it very easy:
helm init
helm upgrade -i --wait cockroachdb01 --set Service.type=NodePort --set Secure.Enabled=true stable/cockroachdb
In order for CockroachDB to generate users with credentials, it needs to run in secure mode. This introduces a bit of complexity in the installation as documented in the chart installation step.
We can verify that everything is working:
kubectl run -it --rm cockroach-client \
(out) --image=cockroachdb/cockroach \
(out) --restart=Never \
(out) --command -- ./cockroach sql --insecure --host cockroachdb-public
(out) If you don't see a command prompt, try pressing enter.
(out) root@cockroachdb01-cockroachdb-public.cockroachdb:26257/> SHOW DATABASES;
(out) +----------+
(out) | Database |
(out) +----------+
(out) | system |
(out) +----------+
(out) (1 row)
(out) Time: 5.889874ms
Nice, we are ready to provision databases in the CockroachDB cluster.
We can install the Service Catalog using helm as well using the chart hosted in the svc-cat repository:
helm repo add svc-cat https://svc-catalog-charts.storage.googleapis.com
helm install svc-cat/catalog \
(out) --name catalog --namespace catalog
Once Helm is done, we get all the goodies:
kubectl get clusterserviceclasses
(out) No resources found.
kubectl get clusterservicebroker
(out) No resources found. Of course, there’s not much to be seen… yet!
Since the Service Catalog implements the Open Service Broker API, we can reuse any Service Broker that respects the same API. For this hands-on example, I am going to use a CockroachDB Service Broker I found on github.
To make it run on Kubernetes all I needed to do is fork it, dockerize it and write a few manifests (and fix a couple of bugs :)). The Service Broker runs as any other application. So it has a deployment and a service. Typically, a third party provider will expose a Service Broker on their own infrastructure, but as I mentioned above, I decided to run it on my own same Kubernetes cluster.
The Service Broker reads its configuration from environment variables which are defined in the env section of the deployment manifest I created:
env:
- name: SERVICES
value: '[
{
"id": "e2e250b5-73f8-45fd-9a7f-93c8dddc5f00",
"name": "cockroachdb",
"description": "CockroachDB service broker for creating cloud-native SQL databases.",
"bindable": true,
"plan_updateable": false,
"metadata": {
"displayName": "CockroachDB",
"longDescription": "CockroachDB service broker for creating cloud-native SQL databases.",
"documentationUrl": "https://www.cockroachlabs.com/docs/",
"supportUrl": "https://www.cockroachlabs.com/community/",
"imageUrl": "https://www.cockroachlabs.com/images/CockroachLabs_Logo_Mark-lightbackground.svg"
},
"tags": ["cockroachdb", "database", "SQL", "cloud"]
}
]'
- name: PRECONFIGURED_PLANS
value: '[
{
"name": "default",
"description": "Default",
"metadata": {
"displayName": "Default"
},
"serviceID": "e2e250b5-73f8-45fd-9a7f-93c8dddc5f00",
"crdbHost": "cockroachdb-public.default.svc.cluster.local",
"crdbPort": "26257"
}
]'
- name: SECURITY_USER_NAME
value: user
- name: SECURITY_USER_PASSWORD
value: pass
The Service Catalog will extract this information and will make it available in the Kubernetes cluster. To make the Service Broker known to the Service Catalog we need to create a ClusterServiceBroker object:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ClusterServiceBroker
metadata:
name: cockroachdb-service-broker
spec:
url: http://user:pass@crdb-service-broker.default
Let’s install the Service Broker and check the resulting entity created in the Service Catalog:
kubectl apply -f k8s/servicebroker.yaml
(out) service "crdb-service-broker" created
(out) deployment "crdb-service-broker" created
(out) clusterservicebroker "cockroachdb-service-broker" configured
kubectl get clusterserviceclasses -o=custom-columns=SERVICE\ NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName
(out) SERVICE NAME EXTERNAL NAME
(out) e2e250b5-73f8-45fd-9a7f-93c8dddc5f00 cockroachdb
kubectl get clusterserviceplans -o=custom-columns=PLAN\ NAME:.metadata.name,EXTERNAL\ NAME:.spec.externalName
(out) PLAN NAME EXTERNAL NAME
(out) d9a19cc6-fbae-597e-af9d-2c3fc640e42c default
Cool, the CockroachDB Service Broker is in place.
We can finally provision a Service Instance of CockroachDB and bind it to an Application using a Service Binding. Here is how:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
name: mydb01
spec:
clusterServiceClassExternalName: cockroachdb
clusterServicePlanExternalName: default
The manifest above defines a ServiceInstance of service class CockroachDB and plan default, that we defined in the Service Broker and registered with the Service Catalog.
The next step is creating a ServiceBinding as follows:
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
name: mydb01-binding
spec:
instanceRef:
name: mydb01
As you can see, we are referencing mydb01, the CockroachDB ServiceInstance defined above.
To keep things simple, we are going to use a Pod to interact with the ServiceInstance. This Pod will act as our Application in the diagram at the beginning of this section:
apiVersion: v1
kind: Pod
metadata:
labels:
app: client-app01
name: client-app01
spec:
containers:
- command:
- sleep
- "2147483648"
image: cockroachdb/cockroach:v2.0.3
imagePullPolicy: IfNotPresent
name: client-app01
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/secrets/mydb01-binding
name: mydb01-binding
env:
- name: "COCKROACH_URL"
valueFrom:
secretKeyRef:
name: mydb01-binding
key: uri
volumes:
- name: mydb01-binding
secret:
secretName: mydb01-binding
Let's deploy it:
kubectl apply -f k8s/clientapp.yaml
(out) pod "client-app01" configured
(out) servicebinding "mydb01-binding" configured
(out) serviceinstance "mydb01" configured
The ServiceBinding exposes the credentials generated by the ServiceBroker as a secret with the same name as the ServiceBinding:
% kubectl get secret mydb01-binding
NAME TYPE DATA AGE
mydb01-binding Opaque 7 4h
In the Pod above, we are using this secret to populate the environment variable COCKROACH_URL. Consequently, the SQL shell will use this variable at runtime.
Finally, we can launch an sql shell and see the content of the database:
kubectl exec -ti client-app01 -- /cockroach/cockroach sql
(out) # Welcome to the cockroach SQL interface.
(out) # All statements must be terminated by a semicolon.
(out) # To exit: CTRL + D.
(out) #
(out) # Server version: CockroachDB CCL v2.0.3 (x86_64-unknown-linux-gnu, built 2018/06/18 16:11:33, go1.10) (same version as client)
(out) # Cluster ID: 444a3d8c-4d64-44cb-83e6-99cf8396400a
(out) #
(out) # Enter \? for a brief introduction.
(out) #
(out) pnembfjkbagknfdfbkpekmihebkllnkf@cockroachdb-public.default.svc.cluster.local:26257/cf_nnmpifmgiloklfaofkpdjlmjbaieacke> SHOW TABLES;
(out) +---------+
(out) | Table |
(out) +---------+
(out) +---------+
(out) (0 rows)
(out) Time: 7.206238ms
(out) .
(out) pnembfjkbagknfdfbkpekmihebkllnkf@cockroachdb-public.default.svc.cluster.local:26257/cf_nnmpifmgiloklfaofkpdjlmjbaieacke> CREATE TABLE mytable ("mycolumn" VARCHAR(255));
(out) CREATE TABLE
(out) .
(out) Time: 147.35522ms
(out) .
(out) pnembfjkbagknfdfbkpekmihebkllnkf@cockroachdb-public.default.svc.cluster.local:26257/cf_nnmpifmgiloklfaofkpdjlmjbaieacke> SHOW TABLES;
(out) +---------+
(out) | Table |
(out) +---------+
(out) | mytable |
(out) +---------+
(out) (1 row)
(out) .
(out) Time: 14.004174ms
When we create a Service Instance the Service Broker creates a new database (as in CREATE DATABASE) in the CockroachDB cluster with a new user which own that database and an associated password. As a result, we can see this by connecting as root to the CockroachDB cluster:
% kubectl exec -it cockroachdb-client-secure -- ./cockroach sql --certs-dir=/cockroach-certs --host=cockroachdb-public
(out) # Welcome to the cockroach SQL interface.
(out) # All statements must be terminated by a semicolon.
(out) # To exit: CTRL + D.
(out) #
(out) # Server version: CockroachDB CCL v2.0.3 (x86_64-unknown-linux-gnu, built 2018/06/18 16:11:33, go1.10) (same version as client)
(out) # Cluster ID: 444a3d8c-4d64-44cb-83e6-99cf8396400a
(out) #
(out) # Enter \? for a brief introduction.
(out) #
(out) warning: no current database set. Use SET database = to change, CREATE DATABASE to make a new database.
(out) root@cockroachdb-public:26257/> show databases;
(out) +-------------------------------------+
(out) | Database |
(out) +-------------------------------------+
(out) | cf_nnmpifmgiloklfaofkpdjlmjbaieacke |
(out) | system |
(out) +-------------------------------------+
(out) (2 rows)
(out) .
(out) Time: 9.212959ms
When unbinding this service (i.e. kubectl delete serviceinstance) the database is removed from the CockroachDB cluster.
The Service Catalog offers powerful abstractions to make services available in a Kubernetes cluster. These services are typically third-party managed cloud offerings but there is nothing preventing us from offering self-hosted services as demonstrated. On the one hand, this allows Kubernetes operators to integrate third party services and offer fine tuned services hosted on the cluster itself. On the other hand, developers can focus on the applications without the need for managing complex services deployment.
Looking for a new challenge? We're hiring!