AWS Secrets Manger in Kubernetes

Prerequisite

• eksctl. details

• AWS CLI. details

• Helm. details

Create EKS Cluster

This will create an EKS cluster in us-east-1 region with 2 nodes in the node-group.

export CLUSTER_NAME=demo-cluster
export AWS_REGION=us-east-1
eksctl create cluster --name ${CLUSTER_NAME} --version 1.26 --node-type t3.medium --nodes 2 --managed --region ${AWS_REGION}
.....
.....
2023-05-28 18:47:47 [✔]  EKS cluster "demo-cluster" in "us-east-1" region is ready.

verify

When running the following command kubectl get nodes you should see the nodes in a ready state.

➜ kubectl get nodes   
NAME                             STATUS   ROLES    AGE    VERSION
ip-192-168-17-231.ec2.internal   Ready    <none>   114m   v1.26.4-eks-0a21954
ip-192-168-61-195.ec2.internal   Ready    <none>   114m   v1.26.4-eks-0a21954

Deploy External Secrets Operator

Use below helm commands to:

• add/update the repo

• install the operator

helm repo add external-secrets https://charts.external-secrets.io
helm repo update

helm install external-secrets \
  external-secrets/external-secrets \
    --namespace external-secrets \
    --create-namespace \
    --set installCRDs=true \
    --wait
.....
external-secrets has been deployed successfully!
....

verify

When you run k get all -n external-secrets you should be able to see all the pods pod/external-secrets-* , pod/external-secrets-cert-controller-*, pod/external-secrets-webhook-* running successfully.

➜ k get all -n external-secrets
NAME                                                    READY   STATUS    RESTARTS   AGE
pod/external-secrets-5997dc48b4-drncs                   1/1     Running   0          2m33s
pod/external-secrets-cert-controller-587977f796-mdk9p   1/1     Running   0          2m33s
pod/external-secrets-webhook-674f79cb47-jz7ds           1/1     Running   0          2m33s

NAME                               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/external-secrets-webhook   ClusterIP   10.100.84.243   <none>        443/TCP   2m33s

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/external-secrets                   1/1     1            1           2m34s
deployment.apps/external-secrets-cert-controller   1/1     1            1           2m34s
deployment.apps/external-secrets-webhook           1/1     1            1           2m34s

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/external-secrets-5997dc48b4                   1         1         1       2m34s
replicaset.apps/external-secrets-cert-controller-587977f796   1         1         1       2m34s
replicaset.apps/external-secrets-webhook-674f79cb47

Configuring IAM roles for service accounts (IRSA) and Secrets Manager

Add OIDC Provider

➜ eksctl utils associate-iam-oidc-provider --cluster=${CLUSTER_NAME} --approve

2023-05-28 13:25:07 [✔]  created IAM Open ID Connect provider for cluster "demo-cluster" in "us-east-1"

Create Secret in Secrets Manager

SECRET_ARN=$(aws secretsmanager create-secret --name user-creds \
    --secret-string "{\"user\":\"admin\",\"password\":\"topsecret\"}" \
    --region ${AWS_REGION} | jq -r .ARN)

Create IAM Policy

IAM Policy to Read/Describe the user-creds secret in the secrets manager

➜ IAM_POLICY_ARN=$(aws iam create-policy --policy-name eso-sm-reader-policy --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": ["'${SECRET_ARN}'"]
    }
  ]
}' | jq -r .Policy.Arn)

Create a Service Account and Association

➜ eksctl create iamserviceaccount \
    --name eso-secret-irsa \
    --namespace default \
    --cluster ${CLUSTER_NAME} \
    --role-name "eso-sm-reader-role" \
    --attach-policy-arn $IAM_POLICY_ARN \
    --approve

verify

k describe sa eso-secret-irsa                      
Name:                eso-secret-irsa
Namespace:           default
Labels:              app.kubernetes.io/managed-by=eksctl
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::808053714438:role/eso-sm-reader-role
Image pull secrets:  <none>
Mountable secrets:   <none>
Tokens:              <none>
Events:              <none>

Deploy External Secret Store/Secret

Create SecretStore

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-secret-irsa
EOF

Verify

kubectl get SecretStore                
NAME           AGE   STATUS   CAPABILITIES   READY
secret-store   58s   Valid    ReadWrite      True

Create ExternalSecret

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: user-creds
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: secret-store
    kind: SecretStore
  target:
    name: user-creds
  data:
  - secretKey: user
    remoteRef:
      key: user-creds
      property: user
  - secretKey: password
    remoteRef:
      key: user-creds
      property: password
EOF

Verify

➜ kubectl get ExternalSecret           
NAME         STORE          REFRESH INTERVAL   STATUS         READY
user-creds   secret-store   5m                 SecretSynced   True
➜ kubectl get secret
NAME         TYPE     DATA   AGE
user-creds   Opaque   2      30s

Deploy Application

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: creds-app
spec:
  containers:
    - name: app
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "env" ]
      envFrom:
      - secretRef:
          name: user-creds
EOF

Verify

➜ kubectl logs pod/creds-app     
....   
user=admin
password=topsecret
....