Tutorial: How to Set External-Secrets with AWS

As you may already have seen, the team at Container Solutions have recently announced The birth of the external secrets community, where multiple people and organizations are joining efforts to create a single External Secrets solution based on existing projects. The new Kubernetes operator integrates external secret management systems like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault and many more, and is designed to enable the synchronization of secrets from external APIs into Kubernetes. The project has extensive documentation, and, of course, it’s Open Source!

However, it’s not always easy to know what we need to do for every possible case when it comes to configuring both providers and permissions. This series of posts is going to focus on how to set up external-secrets with different providers. This article we’ll explore how to set up external-secrets with AWS SecretsManager, Subsequent posts will cover how to set external-secrets with GCP Secret Manager, Azure KeyVault and Hashicorp Vault.

To properly follow this tutorial, make sure you have installed the following tools:

In addition, make sure you already set up your cli environment by running aws configure before continuing through this guide.

Programmatic Authentication

External-secrets allows configuration of several authentication methods for the AWS Secrets Manager provider. This guide will focus on programmatic Authentication mainly because there is no need to spin up any AWS resources other than Secrets Manager.

In order to do so, we need to do the following steps:

1. Configure AWS:

a. create an IAM policy;
b. create an IAM group and bind the policy;
c. create an IAM user and attach it to a group; and
d. create credentials for that user;

2. Configure External-Secrets

Configuring AWS

We first starting by creating a policy named secrets-reader:

POLICY_ARN=$(aws iam create-policy --policy-name secrets-reader --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}' | jq -r .Policy.Arn)

We now create a group that will use this policy:

aws iam create-group --group-name secret-readers
aws iam attach-group-policy --policy-arn $POLICY_ARN --group-name secret-readers

Next, we create an username and attach it to the recently created-group:

aws iam create-user --user-name external-secrets
aws iam add-user-to-group --group-name secret-readers --user-name external-secrets

Finally, we create a set of credentials for that user, and add it as a secret in kubernetes:

aws iam create-access-key --user-name external-secrets > creds.json
ACCESS_KEY=$(cat creds.json | jq -r .AccessKey.AccessKeyId)
SECRET_KEY=$(cat creds.json | jq -r .AccessKey.SecretAccessKey)
kubectl create secret generic aws-secret --from-literal=access-key=$ACCESS_KEY --from-literal=secret=$SECRET_KEY

Now, let’s add some secrets in our secret Store!

aws secretsmanager create-secret \
     --name super-secret \
     --secret-string my-custom-secret \
     --region us-east-2

Configuring External Secrets

The first thing we need to do is define a SecretStore . This resource is where all backend-related configuration is going to be stored, to allow external-secrets to reach out to Secrets-Manager.In the example we create a SecretStore called my-secret-store, using the resources we created from previous steps:

cat <<EOF | kubectl apply -f - 
apiVersion: external-secrets.io/v1alpha1
kind: SecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    aws:  # set secretStore provider to AWS.
      service: SecretsManager # Configure service to be Secrets Manager
      region: us-east-2   # Region where the secret is.
      auth:
        secretRef:
          accessKeyIDSecretRef: 
            name: aws-secret # References the secret we created
            key: access-key  
          secretAccessKeySecretRef:
            name: aws-secret
            key: secret
EOF

The next thing is to create an ExternalSecret definition. In this definition, we are going to indicate which secrets from a SecretStore we want to synchronize into a Kubernetes Secrets object. Here is an example of an ExternalSecret called my-external-secret which gets the secret we created in AWS Secrets Manager:

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: my-secret-store #The secret store name we have just created.
    kind: SecretStore
  target:
    name: my-kubernetes-secret # Secret name in k8s
  data:
  - secretKey: password # which key it's going to be stored
    remoteRef:
      key: super-secret # Our secret-name goes here
EOF

And that’s it! You can see that a secret called `my-kubernetes-secret` is created in kubernetes by the external-secrets operator. We can evaluate the secret value by typing:

kubectl get secrets my-kubernetes-secret -o json | jq -r .data.password | base64 -d

We can also check ExternalSecret status with the following command line:

kubectl get externalsecrets.external-secrets.io my-external-secret


NAME                 STORE             REFRESH INTERVAL   STATUS
my-external-secret   my-secret-store   1m                 SecretSynced

Adding better policies and proper Roles

The example above is a good, simple demo, but it isn’t exactly production ready. For one thing the policy secrets-reader we have created is not enforcing. It allows external-secrets to get any information from the AWS Secrets Manager. Although this might be valid for small deployments, production-ready environments need to be more strict on which resource a given account can access. One way to enforce this restriction is to follow the least privilege principle. In this session we will show how to configure AWS Policies to achieve this goal.

But first, let’s add some more secrets so we can test the policy effect:

aws secretsmanager create-secret \
    --name another-secret \
    --secret-string this_is_another_secret \
    --region us-east-2
aws secretsmanager create-secret \
     --name forbidden-secret \
     --secret-string "We won't see this" \
     --region us-east-2

The first thing we need to do is to create a more strict policy. In order to do so, the only thing we need to do is to add specific Resources instead of a general ‘*’ policy. This example below creates a policy that allows access to super-secret and another-secret, but no access to forbidden-secret.

STRICT_POLICY_ARN=$(aws iam create-policy --policy-name strict-policy --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-2:<YOUR_ACCOUNT_ID>:secret:super-secret-<foobar>",
                "arn:aws:secretsmanager:us-east-2:<YOUR_ACCOUNT_ID>:secret:another-secret-<foobar>"
            ]
        }
    ]
}' | jq -r .Policy.Arn)

Note: Please remember to replace the secrets ARN with the proper value for your scenario! You can check secrets ARN with the following command:

aws secretmanager list-secrets --region us-east-2

Now, let’s detach the old policy and attach this new one:

aws iam detach-group-policy --policy-arn $POLICY_ARN --group-name secret-readers
aws iam attach-group-policy --policy-arn $STRICT_POLICY_ARN --group-name secret-readers

Finally, let’s create two more ExternalSecret objects: one for another-external-secret and one for forbidden-secret:

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: another-external-secret
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: my-secret-store
    kind: SecretStore
  target:
    name: another-kubernetes-secret
  data:
  - secretKey: key-in-kubernetes
    remoteRef:
      key: another-secret
EOF/pre>

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: invalid-permissions
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: my-secret-store 
    kind: SecretStore
  target:
    name: should-not-be-created
  data:
  - secretKey: wont-be-created
    remoteRef:
      key: forbidden-secret
EOF

And that’s it! If we look to the ExternalSecret status we will see:

kubectl get external-secrets                                                               
NAME                      STORE             REFRESH INTERVAL   STATUS
another-external-secret   my-secret-store   1m                 SecretSynced
invalid-permissions       my-secret-store   1m                 SecretSyncedError
my-external-secret        my-secret-store   1m                 SecretSynced

which shows us that our invalid-permissions resource indeed cannot access forbidden-secret!

Wrapping things up

In this post we’ve seen how to configure AWS and external-secrets to allow using Secrets Manager as a provider. There are other supported authentication methods, if you are running on an EKS cluster. You can check that in the official external-secrets documentation.

Wrapping up what we have done:

• Create IAM resources in AWS
• Create secrets in Secrets Manager
• Create SecretStore and ExternalSecrets
• Improving AWS IAM Policies

In the next post, we’ll see how to configure GCP Secret Manager as a SecretStore! Stay tuned!