Sealed Secrets: Deep Dive

This document explains the architecture, cryptographic model, and operational considerations behind Bitnami Sealed Secrets. It covers scoping strategies, key management, rotation workflows, backup and recovery, and comparison with other secret management approaches like Vault and External Secrets Operator.

If you are looking for step-by-step instructions, see the README.

---

Why Sealed Secrets Exists

Kubernetes Secrets are base64 encoded, not encrypted. Anyone with read access to the etcd database or the Kubernetes API can decode them. This creates a fundamental problem for GitOps: you cannot safely commit Secrets to Git.

Base64 encoding is trivial to reverse:

echo "czNjcmV0LXBhc3N3MHJk" | base64 -d
# s3cret-passw0rd

Every developer with repository access can see your production credentials. Every CI/CD pipeline that clones the repository has access to every secret. Audit trails disappear. You lose the benefits of Git for the most critical part of your application configuration.

The GitOps Dilemma

GitOps relies on declarative configuration stored in Git. Everything about your application lives in version control: deployments, services, ingress rules, ConfigMaps. But Secrets break this model. You can commit them in plaintext (bad idea) or keep them outside Git (breaks declarative purity).

Common workarounds before Sealed Secrets:

• Secret manifests ignored in .gitignore: manual kubectl apply required, no version history, no audit trail, GitOps tools cannot manage them
• Encrypted files in Git: GPG or SOPS encrypted files, requires decryption keys on every cluster and in every CI/CD pipeline
• External secret stores: Vault or cloud secret managers work but add operational complexity and require network calls at runtime
• CI/CD injection: secrets injected at deployment time, outside of Git, violates GitOps principles

Sealed Secrets solves this cleanly. You encrypt Secrets with a public key, commit the encrypted SealedSecret to Git, and the controller in the cluster decrypts them with the private key. The private key never leaves the cluster.

---

How Sealed Secrets Works

Sealed Secrets uses asymmetric cryptography. The controller generates an RSA key pair at startup. The public key is available to anyone who wants to encrypt a Secret. The private key stays in the controller and is used to decrypt SealedSecrets into regular Kubernetes Secrets.

The Workflow

1. Controller startup: the Sealed Secrets controller generates a 4096-bit RSA key pair. The private key is stored in a Kubernetes Secret in the controller’s namespace (sealed-secrets by default). The public key is served via the controller’s API.

2. Fetch the public key: developers run kubeseal --fetch-cert to retrieve the current public key. This key can be shared freely. Anyone with the public key can encrypt Secrets, but only the controller can decrypt them.

3. Encrypt a Secret: developers create a regular Kubernetes Secret manifest (using kubectl create secret --dry-run=client -o yaml) and pipe it through kubeseal. The tool encrypts each value in the Secret’s data field using the public key and outputs a SealedSecret resource.

4. Commit to Git: the SealedSecret is safe to commit. The encrypted values are base64-encoded ciphertext. Without the private key, they cannot be decrypted.

5. Apply to cluster: GitOps tools (ArgoCD, Flux) or manual kubectl apply creates the SealedSecret resource in the cluster.

6. Controller watches SealedSecrets: the controller detects the new SealedSecret, decrypts it using the private key, and creates a regular Kubernetes Secret with the decrypted values.

7. Pods consume the Secret: applications reference the regular Secret using secretKeyRef or volume mounts. They never know Sealed Secrets exists.

Developer                Git Repository           Cluster
    |                         |                      |
    |-- create Secret -------->|                      |
    |   (dry-run)              |                      |
    |                          |                      |
    |-- kubeseal encrypt ----->|                      |
    |                          |                      |
    |                          |-- GitOps sync ------>|
    |                          |                      |
    |                          |   Controller detects |
    |                          |   SealedSecret       |
    |                          |                      |
    |                          |   Controller decrypts|
    |                          |   with private key   |
    |                          |                      |
    |                          |   Regular Secret     |
    |                          |   created            |
    |                          |                      |
    |                          |   Pod consumes Secret|

---

The SealedSecret CRD

The controller installs a Custom Resource Definition called SealedSecret. It is API group bitnami.com/v1alpha1.

The demo’s placeholder SealedSecret shows the structure:

# From manifests/sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: sealed-secrets-demo
spec:
  encryptedData:
    DB_USER: AgA...placeholder...
    DB_PASSWORD: AgA...placeholder...
    DB_HOST: AgA...placeholder...
  template:
    metadata:
      name: db-credentials
      namespace: sealed-secrets-demo
    type: Opaque

encryptedData

Each key in encryptedData corresponds to a key in the resulting Secret. The value is the encrypted ciphertext. The ciphertext starts with AgA (the version prefix for the encryption scheme). The actual encrypted payload follows.

Kubeseal encrypts each value independently. The namespace and Secret name are factored into the encryption (for strict scoping, more on this below), but the key names are not.

template

The template field defines metadata for the Secret that the controller creates. This includes the Secret’s name, namespace, labels, annotations, and type. The controller copies these fields to the generated Secret.

If you want to add labels or annotations to the Secret, put them in the template. The controller only manages the data and metadata fields. It does not touch other aspects of the Secret.

Scope Binding

The encryption algorithm binds the ciphertext to the namespace and name specified in the metadata. This is a security feature. It prevents someone from copying your SealedSecret to a different namespace or renaming it and gaining access to secrets they should not have.

---

Scoping Modes: strict, namespace-wide, cluster-wide

Sealed Secrets supports three scoping modes. The mode determines what metadata is included in the encryption input, which controls where the SealedSecret can be used.

strict (default)

The encryption includes the namespace and the Secret name. The SealedSecret can only be decrypted if applied in the exact namespace with the exact name.

When you run kubeseal without a --scope flag:

kubectl create secret generic db-credentials \
  --namespace sealed-secrets-demo \
  --from-literal=DB_PASSWORD=s3cret-passw0rd \
  --dry-run=client -o yaml \
  | kubeseal --controller-namespace sealed-secrets \
    --controller-name sealed-secrets -o yaml

The resulting SealedSecret is bound to the sealed-secrets-demo namespace and the name db-credentials. If you change the namespace in the metadata and apply it, the controller fails to decrypt it. If you rename it to db-credentials-copy, decryption fails.

This is the most secure mode. An attacker cannot move your SealedSecret to a namespace they control and extract the values.

namespace-wide

The encryption includes only the namespace, not the name. The SealedSecret can be renamed but must stay in the same namespace.

kubeseal --scope namespace-wide ...

Use this when you have multiple Secrets in a namespace that share the same values and you want to reuse the SealedSecret by renaming it. Less common in practice.

cluster-wide

The encryption includes neither the namespace nor the name. The SealedSecret can be used anywhere in the cluster.

kubeseal --scope cluster-wide ...

Use this for secrets that need to be available in multiple namespaces, like image pull credentials or API keys shared across teams.

The trade-off is obvious: anyone with the ability to create a SealedSecret in any namespace can decrypt cluster-wide SealedSecrets by copying them.

Choosing a Scope

Scope	Use Case
strict	Secrets tied to a specific application in a specific namespace
namespace-wide	Shared secrets across multiple Secrets in one namespace
cluster-wide	Shared secrets across namespaces (image pull credentials)

Default to strict unless you have a clear reason to use a wider scope.

---

Key Management and Rotation

The controller’s private key is the most critical asset in the system. If it leaks, anyone can decrypt all your SealedSecrets. Key management is essential.

Key Storage

The controller stores the private key in a Kubernetes Secret named sealed-secrets-keyXXXXXXXXX (where the X characters are a hash of the key) in the namespace where the controller runs. The key is base64-encoded PEM.

Kubernetes Secret encryption at rest (if enabled) protects the key on disk. Kubernetes RBAC protects access to the Secret via the API.

The controller loads the key at startup. If the key Secret does not exist, the controller generates a new key pair. This happens on the first run or if the key is deleted.

Automatic Key Rotation

The controller automatically generates a new key every 30 days by default. This is configurable via the --key-renew-period flag.

Old keys are not deleted. The controller keeps all keys and uses the newest key for new encryptions. Existing SealedSecrets encrypted with old keys remain decryptable because the controller retries all keys until one works.

This allows gradual re-encryption. You can leave old SealedSecrets in place and re-encrypt them on your own schedule.

Manual Key Rotation

To rotate keys manually, delete the newest key Secret. The controller generates a new one on its next reconciliation cycle.

After rotation, fetch the new public key:

kubeseal --fetch-cert \
  --controller-namespace sealed-secrets \
  --controller-name sealed-secrets > /tmp/new-pub.pem

Re-encrypt your Secrets with the new key:

kubectl create secret generic db-credentials \
  --namespace sealed-secrets-demo \
  --from-literal=DB_PASSWORD=new-password \
  --dry-run=client -o yaml \
  | kubeseal --cert /tmp/new-pub.pem \
    --controller-namespace sealed-secrets \
    --controller-name sealed-secrets -o yaml \
  | kubectl apply -f -

Key Backup

Backup the key Secrets regularly. If the controller is deleted and the keys are lost, all SealedSecrets become undecryptable. You need to re-create every Secret manually.

To backup keys:

kubectl get secret -n sealed-secrets -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > sealed-secrets-keys-backup.yaml

Store this file securely. Encrypt it with GPG or store it in a vault. Anyone with this file can decrypt all your SealedSecrets.

Disaster Recovery

To restore from backup:

1. Install the Sealed Secrets controller in the new cluster
2. Delete the automatically generated key (if any)
3. Apply the backed-up key Secret
4. Restart the controller to load the restored key
5. Apply your SealedSecret manifests from Git

The controller decrypts them with the restored key, and everything works as before.

---

Offline Encryption Workflows

You do not need access to the cluster to encrypt Secrets. If you have the public key, you can encrypt anywhere.

Fetch and Distribute the Public Key

Fetch the public key once and commit it to your repository or distribute it to developers:

kubeseal --fetch-cert \
  --controller-namespace sealed-secrets \
  --controller-name sealed-secrets > pub-sealed-secrets.pem

Commit pub-sealed-secrets.pem to Git. It is safe. It is the public key.

Encrypt Offline

Developers can now encrypt Secrets without kubectl access to the cluster:

kubectl create secret generic db-credentials \
  --namespace sealed-secrets-demo \
  --from-literal=DB_PASSWORD=s3cret-passw0rd \
  --dry-run=client -o yaml \
  | kubeseal --cert pub-sealed-secrets.pem -o yaml > sealed-secret.yaml

The --cert flag tells kubeseal to use the local public key instead of fetching it from the controller.

This is useful for CI/CD pipelines. The pipeline encrypts Secrets during the build process and commits the SealedSecret to the repository. The cluster never needs to expose the controller’s API externally.

Security Implications

The public key distribution model is critical to Sealed Secrets’ design. The public key is not sensitive. Distributing it widely is fine. But the private key must remain in the cluster. Never export the private key Secret and hand it to developers or CI/CD systems.

---

Secret Updates and Re-Encryption

When a secret value changes, you re-encrypt and re-apply the SealedSecret. The controller detects the change and updates the underlying Kubernetes Secret.

The demo README shows the pattern:

kubectl create secret generic db-credentials \
  --namespace sealed-secrets-demo \
  --from-literal=DB_PASSWORD=new-passw0rd-123 \
  --dry-run=client -o yaml \
  | kubeseal --controller-namespace sealed-secrets \
    --controller-name sealed-secrets \
    --format yaml | kubectl apply -f -

The controller decrypts the new SealedSecret and overwrites the existing Secret’s data fields. Pods see the new values based on how they consume the Secret (environment variables require a restart, volume mounts refresh within 1-2 minutes).

Re-Encryption After Key Rotation

After rotating the controller’s key, existing SealedSecrets still work because the controller retains old keys. But newly encrypted SealedSecrets use the new key.

To re-encrypt all SealedSecrets with the new key:

1. Fetch the new public key
2. For each SealedSecret, decrypt it to a Secret, re-encrypt with the new key, and apply

There is no automated re-encryption tool in the core project. This is intentional. Re-encryption is a sensitive operation. The plaintext passes through your local machine or CI/CD system.

Some teams write scripts to automate this:

# Fetch current public key
kubeseal --fetch-cert > new-pub.pem

# For each SealedSecret in Git
for file in $(find . -name "*sealedsecret.yaml"); do
  # This requires access to decrypt (either via controller API or backup key)
  kubectl get secret $(yq '.metadata.name' $file) \
    -n $(yq '.metadata.namespace' $file) -o yaml \
    | kubeseal --cert new-pub.pem -o yaml > $file
done

This is a maintenance task, not an operational requirement. Old SealedSecrets continue to work indefinitely.

---

Production Considerations

Moving from a demo to production requires addressing several operational areas.

Controller High Availability

The Sealed Secrets controller can run with multiple replicas. All replicas share the same private key (loaded from the same Secret). Only one replica actively processes SealedSecrets at a time (leader election), but all replicas serve the public key API.

To deploy in HA mode:

helm upgrade sealed-secrets sealed-secrets/sealed-secrets \
  --namespace sealed-secrets \
  --set replicaCount=3

If the active controller fails, Kubernetes leader election promotes a standby replica. Decryption continues with minimal downtime.

Resource Limits

The demo Deployment does not show resource limits because it is installed via Helm. In production, the controller is lightweight. Typical settings:

resources:
  requests:
    cpu: 50m
    memory: 64Mi
  limits:
    cpu: 200m
    memory: 128Mi

The controller does not process high request volumes. It watches SealedSecret resources, decrypts them once, and creates Secrets. After that, no ongoing work happens unless the SealedSecret is updated.

Backup Automation

Automate key backups. Run a CronJob that backs up the key Secrets to a secure location (S3, GCS, Vault, encrypted Git repository). Schedule this daily or weekly.

Example CronJob logic:

kubectl get secret -n sealed-secrets \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml | gpg --encrypt --recipient admin@example.com > backup.yaml.gpg
# Upload backup.yaml.gpg to S3

Test your restore procedure regularly.

Monitoring

Watch the controller logs for decryption errors. The controller logs every SealedSecret it processes. If decryption fails (wrong scope, key mismatch, corrupted data), the error appears in the logs.

The controller does not expose Prometheus metrics by default. Consider adding a sidecar or external monitoring for production visibility.

Namespace Strategy

If you use a single Sealed Secrets controller for the entire cluster, every SealedSecret in every namespace is encrypted with the same key. This is fine for most organizations.

For multi-tenant environments where teams should not share a key, deploy multiple controllers in different namespaces. Each controller has its own key pair. Teams fetch the public key from their assigned controller and encrypt SealedSecrets scoped to their namespace.

RBAC and Least Privilege

The controller needs RBAC permissions to create and update Secrets in all namespaces where SealedSecrets exist. The default Helm installation creates a ClusterRole and ClusterRoleBinding granting these permissions.

Restrict access to the controller’s namespace. Only cluster administrators should have access to the private key Secrets.

---

Common Pitfalls and Troubleshooting

Wrong Scope

Symptom: the controller fails to decrypt a SealedSecret with an error like unable to decrypt sealed secret.

Cause: the SealedSecret was encrypted for a different namespace or name than where it is being applied. Check the scope mode used during encryption.

Fix: re-encrypt the Secret with the correct namespace and name, or use --scope namespace-wide or --scope cluster-wide if appropriate.

Key Not Found

Symptom: the controller logs show no key could decrypt secret.

Cause: the SealedSecret was encrypted with a public key that does not match any private key in the controller.

Fix: check if the key was rotated or the controller was reinstalled without restoring the backup. Restore the key from backup or re-encrypt with the current public key.

Namespace Mismatch

Symptom: the SealedSecret appears in the cluster but no Secret is created.

Cause: the controller only processes SealedSecrets in namespaces it watches. By default, it watches all namespaces. If the controller is configured to watch specific namespaces, SealedSecrets in other namespaces are ignored.

Fix: check the controller’s --namespaces flag. Deploy the SealedSecret to a watched namespace or expand the controller’s watch scope.

Forgotten Key Backup

Symptom: the controller was deleted, keys were lost, and SealedSecrets cannot be decrypted.

Cause: no backup of the key Secret exists.

Fix: there is no fix. You must manually re-create the Secrets from the original plaintext values and re-encrypt them. This is painful. Automate backups from day one.

Corrupted SealedSecret

Symptom: the controller fails to decrypt with a cryptographic error.

Cause: the YAML was edited by hand and the base64 ciphertext was corrupted, or the SealedSecret was copied incorrectly.

Fix: re-encrypt the Secret from scratch. Do not hand-edit the encryptedData field. Always use kubeseal to generate SealedSecrets.

Pods Not Seeing Updated Secrets

Symptom: the Secret was updated but pods still see old values.

Cause: pods consume Secrets as environment variables. Environment variables are set at container start time and do not refresh.

Fix: restart the pods (kubectl rollout restart deployment/demo-app). Or switch to volume mounts, which refresh automatically within 1-2 minutes.

---

Trade-Offs and Alternatives

Sealed Secrets is one of several approaches to Kubernetes secret management. Understanding the alternatives helps you choose the right tool.

Sealed Secrets vs Vault

Vault is a full-featured secret management platform. It provides dynamic secrets, fine-grained policies, audit logs, lease management, and integrations with databases, cloud providers, and PKI systems.

Sealed Secrets encrypts static secrets so they can be stored in Git. It does not provide dynamic secrets, audit logs, or centralized policy control.

Advantages of Sealed Secrets:

• Simpler to operate (single controller, no external dependencies)
• Secrets live in Git with the rest of your configuration
• No runtime dependency on an external service (Vault downtime does not block secret access once the Secret is created)
• Lower resource footprint (no Vault server, no sidecar agents)

Disadvantages compared to Vault:

• No dynamic secrets (every credential is static, must be rotated manually)
• No audit trail of secret access (you know who committed the SealedSecret, but not which pod read the Secret)
• No fine-grained access control (anyone who can apply a SealedSecret in a namespace can create any Secret in that namespace)
• Secret rotation requires re-encryption and re-deployment

Sealed Secrets vs External Secrets Operator

External Secrets Operator (ESO) syncs secrets from external stores (Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) into Kubernetes Secrets. It polls the external store for changes and keeps Secrets in sync.

Sealed Secrets stores encrypted secrets in Git. There is no external store.

Advantages of Sealed Secrets:

• No external dependency (no Vault, no cloud secret manager)
• Secrets are versioned in Git alongside application configuration
• Simpler architecture (one controller, no external API calls at runtime)

Disadvantages compared to ESO:

• No dynamic secrets or automatic rotation from the source
• No integration with existing enterprise secret stores
• No refresh mechanism (ESO polls and updates Secrets automatically)
• If you already use Vault or a cloud secret manager, Sealed Secrets adds redundancy without benefit

Sealed Secrets vs SOPS

SOPS (Secrets OPerationS) encrypts files using AWS KMS, GCP KMS, Azure Key Vault, PGP, or age. You commit encrypted files to Git and decrypt them in CI/CD or on the cluster.

Sealed Secrets is Kubernetes-native. It uses a CRD and a controller. SOPS is file-based. It encrypts YAML or JSON files.

Advantages of Sealed Secrets:

• Kubernetes-native (CRD, controller, kubectl apply workflow)
• No need to decrypt files in CI/CD before applying (the controller handles decryption in-cluster)
• Scoping built into the encryption model

Disadvantages compared to SOPS:

• Tied to Kubernetes (SOPS works for any configuration file, not just Secrets)
• The controller is a single point of failure (SOPS does not require a controller, decryption happens client-side or via initContainers)
• Key management is cluster-specific (SOPS keys can be managed centrally in cloud KMS)

Sealed Secrets vs Cloud-Native Secret Managers

AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault are managed services. They store secrets, handle encryption at rest, provide audit logs, and integrate with cloud IAM.

Sealed Secrets keeps secrets in Git, encrypted client-side, with no cloud dependency.

Advantages of Sealed Secrets:

• No cloud dependency (works on-premises, air-gapped environments, multi-cloud)
• Secrets versioned in Git
• No API quota or rate-limiting concerns

Disadvantages compared to cloud secret managers:

• No centralized audit logs
• No integration with cloud IAM for fine-grained access control
• No managed rotation workflows
• You manage the controller and key backups (cloud providers manage everything)

---

When to Use Sealed Secrets

Sealed Secrets is the right choice when:

• You practice GitOps and want all configuration (including secrets) in Git
• You do not need dynamic secrets or secret rotation workflows
• You want a simple, self-contained solution with no external dependencies
• You run on-premises, air-gapped, or multi-cloud environments where a single external secret store is impractical
• You do not have an existing enterprise secret management system

Sealed Secrets is not the right choice when:

• You need dynamic secrets (database credentials generated per-pod, short-lived API tokens)
• You need centralized audit logs and fine-grained access policies
• You already run Vault or a cloud secret manager and want to use it
• You need automatic secret rotation driven by an external system
• You need secrets synced from an external source without re-encryption

---

The Demo in Context

The demo deploys the Sealed Secrets controller, encrypts a Secret with kubeseal, commits the SealedSecret to manifests, and applies it. The controller decrypts it into a regular Secret, which the demo app consumes.

The application Deployment references the Secret using standard Kubernetes patterns:

# From manifests/deployment.yaml
env:
  - name: DB_USER
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_USER
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_PASSWORD
  - name: DB_HOST
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_HOST

The app has zero knowledge of Sealed Secrets. It consumes a regular Secret. The SealedSecret is infrastructure, not application code.

The manifests include both the original Secret (as a reference) and the SealedSecret:

# From manifests/secret-original.yaml
# WARNING: Do NOT commit this file to Git!
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: sealed-secrets-demo
type: Opaque
stringData:
  DB_USER: "appuser"
  DB_PASSWORD: "s3cret-passw0rd"
  DB_HOST: "postgres.sealed-secrets-demo.svc"

This file exists for demonstration purposes only. In production, you create this with kubectl create secret --dry-run=client, encrypt it with kubeseal, and discard the plaintext YAML. Only the SealedSecret is committed.

The SealedSecret in the manifests is a placeholder:

# From manifests/sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: sealed-secrets-demo
spec:
  encryptedData:
    DB_USER: AgA...placeholder...
    DB_PASSWORD: AgA...placeholder...
    DB_HOST: AgA...placeholder...

In the README walkthrough, you generate a real SealedSecret with kubeseal and apply it. The controller decrypts it, creates the Secret, and the pods start.

---

Key Takeaways

• Sealed Secrets encrypts Kubernetes Secrets with asymmetric cryptography so they can be safely committed to Git.
• The controller generates an RSA key pair. The public key encrypts, the private key decrypts. The private key never leaves the cluster.
• Scoping modes (strict, namespace-wide, cluster-wide) control where a SealedSecret can be decrypted. Default to strict for security.
• The controller automatically rotates keys every 30 days. Old keys are retained so existing SealedSecrets remain decryptable.
• Backup the controller’s private key regularly. If the key is lost, all SealedSecrets become undecryptable.
• Offline encryption workflows let you encrypt Secrets in CI/CD without cluster access. Distribute the public key, keep the private key in-cluster.
• Sealed Secrets is simpler than Vault but offers no dynamic secrets, audit logs, or centralized policy control. It solves the GitOps secrets problem, not the full secret lifecycle problem.
• Compared to External Secrets Operator, Sealed Secrets keeps secrets in Git with no external dependency. ESO integrates with external stores and syncs automatically.
• Compared to SOPS, Sealed Secrets is Kubernetes-native and requires a controller. SOPS is file-based and works with any configuration, not just Secrets.

---

See Also

• README: step-by-step demo instructions
• Vault: full-featured secret management platform
• External Secrets Operator: sync secrets from external stores into Kubernetes
• Sealed Secrets Documentation: official repository with detailed usage, scoping, and controller configuration
• Sealed Secrets Design: cryptographic design and threat model