Kubernetes Secrets and etcd encryption: risks and better options

What a Secret really is

A Kubernetes Secret is an object for small sensitive values such as passwords, tokens and TLS keys. The values are base64-encoded, which is just a way to represent bytes as text, not protection. Anyone who can read the object can decode it instantly; try it with the Base64 tool.

kubectl create secret generic db-cred -n shop --from-literal=password='<your-password>'
kubectl get secret db-cred -n shop -o yaml

Secrets are stored in etcd, the cluster database. Unless you configure encryption at rest, they sit there in plain form. So a Secret is exposed to anyone with RBAC read access to it, anyone who can create pods that mount it, anyone with access to etcd or its disk, and anyone who obtains an etcd backup.

Restricting access

• Grant get, list and watch on secrets only to the identities that need them, and scope by namespace. Note that list returns secret values too.
• Remember that permission to create pods or workloads in a namespace effectively grants access to its Secrets, because a pod can mount them.
• Mount secrets as files rather than environment variables where possible.
• Consider marking Secrets immutable: true when they should not change, which also protects against accidental edits.

Encryption at rest

The API server can encrypt resources before writing them to etcd, configured with an EncryptionConfiguration file. Options include local keys (aescbc, aesgcm, secretbox) and the KMS v2 provider. Local keys sit in a file on the control plane node, so anyone with access to that node has both the data and the key. The KMS provider uses envelope encryption: each value is encrypted with a data encryption key (DEK), and DEKs are wrapped by a key held in an external key management service, such as AWS KMS, Azure Key Vault or a hardware security module.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - kms:
          apiVersion: v2
          name: cloud-kms
          endpoint: unix:///var/run/kms-plugin/socket.sock
      - identity: {}

The identity provider at the end allows existing unencrypted data to be read during migration. After enabling encryption, rewrite existing Secrets so they are stored encrypted. Managed services make this simpler: EKS, AKS and GKE all offer envelope encryption of Secrets with your own cloud KMS key.

etcd, backups and better options

For important credentials, many teams treat Kubernetes Secrets as a delivery mechanism only, and keep the source of truth in an external store such as HashiCorp Vault, AWS Secrets Manager or Azure Key Vault. The Secrets Store CSI Driver can mount values straight into pods without creating Secret objects at all, while the External Secrets Operator syncs them into Secrets. See container secrets for the wider picture.

Tip: never commit Secret manifests to git, even base64-encoded. If you use GitOps, store encrypted forms (for example Sealed Secrets or SOPS) or references to an external store.