Tutorial: How to Set External-Secrets with Hashicorp Vault

External Secretsis an Open Source Kubernetes operator that integrates with external secret management systems such as AWS Secrets Manager, HashiCorp Vault, Google Secret Manager, and Azure Key Vault, and is designed to enable the synchronization of secrets from external APIs into Kubernetes. The project is managed as part of the recently announced CS Labs.

In the first two articles in this series we’ve looked at setting External-Secrets with bothAWSandGCP. This time we’re going to use Hashicorp Vault as the secret provider. In this tutorial, we will bring up our own Hashicorp Vault instance, configure authentication for our workload using Kubernetes Authentication, and then use External-Secrets to fetch information from our Vault instance.

Getting started

To properly follow this tutorial, make sure you have installed the following tools:

• Helm version 3
• minikube - or any other kubernetes cluster
• external-secrets deployed
• vault cli installed

Kubernetes Authentication

External-secrets allows configuration of several authentication methods for the Hashicorp Vault provider. In order to run this example, we need to perform the following steps:

1. Set up Hashicorp Vault

2. Configure Hashicorp Vault Authentication:

a. Create a policy;
b. Configure Kubernetes authentication endpoint;
c. Create a role;
d. Bind the role to kubernetes authentication endpoint;

3. Configure External-Secrets

Set up Hashicorp Vault

To deploy Hashicorp Vault, we are going to use their Helm chart. If you already have a Vault deployed, or if you are using Hashicorp Cloud Platform, you can skip this step and go straight to the Configure Hashicorp Vault Authentication section.

We first add helm repository and then install Vault with the following command:

helm repo add hashicorp https://helm.releases.hashicorp.com

helm install vault hashicorp/vault -n vault --create-namespace

After vault is deployed, the next step is to expose its service so we can configure it using vault cli. Then, we are going to use vault cli to initialize vault and unseal it

export VAULT_ADDR=http://127.0.0.1:8200
kubectl port-forward svc/vault -n vault 8200:8200 &
vault operator init

The previous command outputs 5 unseal keys and one root token. We will use 3 out of the 5 unseal keys to initialize our Vault. It doesn’t matter which keys you pick, as long as there are three different keys.

vault operator unseal <key 1>
vault operator unseal <key 2>
vault operator unseal <key 3>

Next, we can login with the provided root token to our vault:

vault login <root token>

Configure Hashicorp Vault Authentication

After we have initialized our own vault, the next steps are to configure secure ways to allow External Secrets to fetch it.

The first step is to create a demonstration policy. Policies are a way to enforce specific access within Vault. Here we are creating a policy that only allowsGETsecrets fromkv/path/to/my/secretsecret. Change that policy to your needs. You can use wildcards to configure full path access.

vault policy write demo-policy -<<EOF     
path "*"                                                  
{  capabilities = ["read"]                
}                         
EOF

After our policy is created, we then enable Kubernetes authentication and add our Kubernetes cluster information as a source of it.

vault auth enable kubernetes

k8s_host="$(kubectl exec vault-0 -n vault -- printenv | grep KUBERNETES_PORT_443_TCP_ADDR | cut -f 2- -d "=" | tr -d " ")"
k8s_port="443"            
k8s_cacert="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)"
secret_name="$(kubectl get serviceaccount vault -o go-template='')"
tr_account_token="$(kubectl get secret ${secret_name} -o go-template='' | base64 --decode)"

vault write auth/kubernetes/config token_reviewer_jwt="${tr_account_token}" kubernetes_host="https://${k8s_host}:${k8s_port}" kubernetes_ca_cert="${k8s_cacert}" 
disable_issuer_verification=true

Note: on kubernetes v1.21 and higher, the vaultissuerfield is no longer supported. That is why we addeddisable_issuer_verification=trueto our configuration. You can find more details in this pull request.

After our Kubernetes endpoint is configured, the next step is to create a vault role to access secrets.

demo_secret_name="$(kubectl get serviceaccount external-secrets -n es -o go-template='')"
demo_account_token="$(kubectl get secret ${demo_secret_name} -n es -o go-template='' | base64 --decode)"                   


vault write auth/kubernetes/role/demo-role \
    bound_service_account_names=external-secrets \
    bound_service_account_namespaces=es \
    policies=demo-policy \
    ttl=24h

vault write auth/kubernetes/login role=demo-role jwt=$demo_account_token iss=https://kubernetes.default.svc.cluster.local

There are two things happening here. First, we are defining a role that only our external-secrets service account can access, and binding the role to the demonstration policy we’ve just created. Second we are creating a Kubernetes authentication method providing the Service Account Token that will authenticate to it and the role we want it to assume.

Whilst Hashicorp Vault supports a variety of secret engines, such as KeyValue, SSH, LDAP, PKI, etc., at the time of this writing external-secrets only supports the KeyValue engine.

Let’s enable a KeyValue engine endpoint and then add a secret to it:

vault secrets enable -version=2 kv
vault kv put kv/path/to/my/secret password=secretpassword

Configuring External Secrets

Once we have finished configuring Vault, the next step is to create aSecretStorebinding to thedemo-rolewe have just created, and then create anExternalSecretto fetch information from our secret stored inpath/to/my/secret

cat << EOF | kubectl apply -f -
 apiVersion: external-secrets.io/v1alpha1
 kind: SecretStore
 metadata:
   name: vault-backend
 spec:
   provider:
     vault:
       server: "http://vault.vault:8200"
       path: "kv"
       version: "v2"
       auth:
         kubernetes:
           mountPath: "kubernetes"
           role: "demo-role"
EOF

Note: If you’re using the Hashicorp Cloud Platform solution, you also need to addnamespace: "admin"in the vault session.

cat <<EOF | kubectl apply -f -
 apiVersion: external-secrets.io/v1alpha1
 kind: ExternalSecret
 metadata:
   name: vault-example
 spec:
   secretStoreRef:
     name: vault-backend
     kind: SecretStore
   target:
     name: example-sync
   data:
   - secretKey: foobar
     remoteRef:
       key: path/to/my/secret
       property: password
EOF

And that’s it! Now, we check if our secret was really created with the proper value

kubectl get secrets example-sync -o jsonpath='{.data.foobar}' | base64 -d
secretpassword%

We can also checkExternalSecretstatus:

kubectl get externalsecrets.external-secrets.io vault-example
NAME            STORE           REFRESH INTERVAL   STATUS
vault-example   vault-backend   1h                 SecretSynced

A tip to deploy External Secrets on production environments

In production environments where we might have hundreds (or even thousands) ofExternalSecretsobjects deployed, the operator may get overloaded, causing performance degradation on any secret syncing operation, often manifested in secrets taking longer to get the new value, which can in turn possibly cause some disruption to the application.

In order to increase availability it is possible to install the operator with multiple replicas, and enable leader election to have it distribute its workload. The Leader Election feature is available in operators created with the Kube Builder Toolkit which enables load-balancing. When enabled in a scenario with multiple replicas, a leader amongst all replicas is elected. If leader election is disabled, every replica will try to reconcile every resource, which is generally something we want to avoid. More information on this is available in the official Kube Builder docs. With this feature, multiple External Secrets replicas can be run without having them duplicate efforts onExternalSecretsobjects.

In order to deploy the operator with leader election enabled, we simply need to pass two extra arguments during Helm chart installation:

elm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets external-secrets/external-secrets --set replicaCount=3 --set leaderElect=true 

And that’s it! Now all three replicas will share the workload if one of them fails.

Note: Although not part of this tutorial, it is also possible another way to distribute workload in External Secrets is through the use of controller tags. In this setup, we deploy multiple operators with a set of unique controller tags (one per operator), and then bind theSecretStoresto one controller (using the fieldspec.controller). Then, allExternalSecretsbound to thatSecretStorewill be handled by that operator only. You can read more about specifying controller labels in the documentation.

Wrapping things up

In this post we’ve seen how to configureexternal-secretsto allow using Hashicorp Vault as a provider. There are other supported authentication methods as well, such as token authentication and AppRole authentication. You can check that in the official external-secrets documentation.

Wrapping up what we have done:

• Set up local Hashicorp Vault instance
• Configure Kubernetes Authentication
• Configure External Secrets
• Configuring Leader Election

In the next post, we’ll see how to configure Azure Key Vaultas aSecretStore! Stay tuned!