Sealed Secrets

Encrypt Kubernetes Secrets so they can be safely stored in Git using Bitnami Sealed Secrets.

Time: ~15 minutes Difficulty: Intermediate

What You Will Learn

• Why base64 encoding is not encryption
• Installing the Sealed Secrets controller with Helm
• Using kubeseal to encrypt Secrets with asymmetric cryptography
• How the controller decrypts SealedSecrets into regular Secrets
• Scoping sealed secrets to namespace vs cluster-wide
• Rotating encryption keys safely

Prerequisites

Install the kubeseal CLI:

# macOS
brew install kubeseal

# Linux
wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.24.0/kubeseal-0.24.0-linux-amd64.tar.gz
tar xfz kubeseal-0.24.0-linux-amd64.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

# Verify
kubeseal --version

Why Sealed Secrets

Regular Kubernetes Secrets are base64 encoded, not encrypted. Anyone with read access to the namespace can decode them. This means you cannot safely commit Secrets to Git, which breaks GitOps workflows.

Sealed Secrets solve this problem:

• The controller generates a public/private key pair
• You encrypt Secrets with the public key using kubeseal
• Only the controller can decrypt them (private key never leaves the cluster)
• Encrypted SealedSecrets are safe to commit to Git
• The controller automatically decrypts them into regular Secrets

Install Sealed Secrets Controller

Navigate to the demo directory:

cd demos/sealed-secrets

kubectl apply -f manifests/namespace.yaml

# Install the controller in the sealed-secrets namespace
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update

helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace sealed-secrets \
  --create-namespace \
  --set-string fullnameOverride=sealed-secrets

Wait for the controller to be ready:

kubectl get pods -n sealed-secrets -w

Deploy

Fetch the Public Key

The controller generates a public key for encryption. Fetch it:

kubeseal --fetch-cert \
  --controller-namespace sealed-secrets \
  --controller-name sealed-secrets > /tmp/sealed-secrets-pub.pem

cat /tmp/sealed-secrets-pub.pem

This public key can be safely shared. Anyone can encrypt Secrets with it, but only the controller can decrypt.

Seal a Secret

Create a regular Secret and encrypt it:

# Create a Secret (dry-run, don't apply)
kubectl create secret generic db-credentials \
  --namespace sealed-secrets-demo \
  --from-literal=DB_USER=appuser \
  --from-literal=DB_PASSWORD=s3cret-passw0rd \
  --from-literal=DB_HOST=postgres.sealed-secrets-demo.svc \
  --dry-run=client -o yaml > /tmp/secret.yaml

# Seal it with kubeseal
kubeseal --controller-namespace sealed-secrets \
  --controller-name sealed-secrets \
  --format yaml < /tmp/secret.yaml > /tmp/sealed-secret.yaml

cat /tmp/sealed-secret.yaml

The output shows a SealedSecret with encrypted data. Apply it:

kubectl apply -f /tmp/sealed-secret.yaml

Deploy the Application

kubectl apply -f manifests/deployment.yaml
kubectl apply -f manifests/service.yaml

Verify

Check that the SealedSecret was decrypted into a regular Secret:

# The SealedSecret resource
kubectl get sealedsecrets -n sealed-secrets-demo

# The controller created a regular Secret
kubectl get secrets -n sealed-secrets-demo
kubectl describe secret db-credentials -n sealed-secrets-demo

Verify the pods can read the secret values:

kubectl exec -n sealed-secrets-demo deploy/demo-app -- env | grep DB_

You should see the database credentials as environment variables.

Inspect the Encrypted Data

Compare the SealedSecret to the original Secret:

# Original Secret (base64, easily decoded)
cat /tmp/secret.yaml

# SealedSecret (encrypted, safe for Git)
cat /tmp/sealed-secret.yaml

Try decoding the base64 Secret:

echo "czNjcmV0LXBhc3N3MHJk" | base64 -d
# s3cret-passw0rd (easily decoded!)

The SealedSecret encrypted data cannot be decrypted without the controller’s private key.

What is Happening

manifests/
  namespace.yaml          # sealed-secrets-demo namespace
  secret-original.yaml    # Example regular Secret (DO NOT commit to Git!)
  sealed-secret.yaml      # Placeholder SealedSecret (safe to commit)
  deployment.yaml         # nginx Deployment reading secret as env vars
  service.yaml            # ClusterIP Service

Installed via Helm:
  sealed-secrets-controller  # Runs in sealed-secrets namespace

Encryption flow:

1. Controller generates RSA key pair on startup
2. Public key is available via kubeseal --fetch-cert
3. User creates a Secret (dry-run, not applied)
4. kubeseal encrypts the Secret with the public key
5. User commits SealedSecret to Git (safe, encrypted)
6. User applies SealedSecret to the cluster
7. Controller decrypts it with the private key
8. Controller creates a regular Secret
9. Pods consume the Secret normally

Scope:

By default, SealedSecrets are scoped to the namespace and name specified in metadata. This means:

• The encrypted data is bound to the namespace and Secret name
• You cannot copy a SealedSecret to a different namespace or rename it
• This prevents attacks where someone moves your SealedSecret elsewhere

You can change the scope with kubeseal flags:

• --scope strict (default): namespace + name binding
• --scope namespace-wide: can rename, but locked to namespace
• --scope cluster-wide: can use anywhere (least secure)

Experiment

1. Rotate the secret and re-seal it:

   kubectl create secret generic db-credentials \
     --namespace sealed-secrets-demo \
     --from-literal=DB_USER=appuser \
     --from-literal=DB_PASSWORD=new-passw0rd-123 \
     --from-literal=DB_HOST=postgres.sealed-secrets-demo.svc \
     --dry-run=client -o yaml \
     | kubeseal --controller-namespace sealed-secrets \
       --controller-name sealed-secrets \
       --format yaml | kubectl apply -f -
   
   # Restart the deployment to pick up the new secret
   kubectl rollout restart deploy/demo-app -n sealed-secrets-demo
   kubectl rollout status deploy/demo-app -n sealed-secrets-demo
   
   # Verify
   kubectl exec -n sealed-secrets-demo deploy/demo-app -- env | grep DB_PASSWORD

2. Try to decrypt the SealedSecret without the controller (should fail):

   # Fetch the encrypted value
   kubectl get sealedsecret db-credentials -n sealed-secrets-demo -o yaml
   
   # No way to decrypt it without the controller's private key

3. Test namespace scoping by trying to copy the SealedSecret to another namespace:

   kubectl create namespace test-copy
   kubectl get sealedsecret db-credentials -n sealed-secrets-demo -o yaml | \
     sed 's/sealed-secrets-demo/test-copy/g' | kubectl apply -f -
   
   # The controller will fail to decrypt it because the scope is bound to sealed-secrets-demo
   kubectl get events -n test-copy --sort-by='.lastTimestamp'
   kubectl delete namespace test-copy

4. Check the controller logs to see decryption events:

   kubectl logs -n sealed-secrets -l app.kubernetes.io/name=sealed-secrets

5. Create a cluster-wide scoped SealedSecret:

   kubectl create secret generic api-key \
     --namespace sealed-secrets-demo \
     --from-literal=KEY=sk-live-abc123def456 \
     --dry-run=client -o yaml \
     | kubeseal --controller-namespace sealed-secrets \
       --controller-name sealed-secrets \
       --scope cluster-wide \
       --format yaml | kubectl apply -f -
   
   # This can be renamed or moved to another namespace
   kubectl get secret api-key -n sealed-secrets-demo

Cleanup

# Delete the demo namespace
kubectl delete namespace sealed-secrets-demo

# Uninstall the controller
helm uninstall sealed-secrets -n sealed-secrets
kubectl delete namespace sealed-secrets

# Clean up temp files
rm -f /tmp/secret.yaml /tmp/sealed-secret.yaml /tmp/sealed-secrets-pub.pem

Further Reading

See docs/deep-dive.md for a detailed explanation of Sealed Secrets architecture, key rotation strategies, backup and recovery, offline encryption workflows, and comparison with other secret management solutions like Vault and External Secrets Operator.

Next Step

Move on to Vertical Pod Autoscaler to learn how VPA automatically right-sizes container resource requests based on actual usage.