External Secrets Operator: Deep Dive

This document explains the architecture, CRDs, and operational considerations behind the External Secrets Operator (ESO). It covers the concepts from the demo and extends into areas like PushSecret, secret templates, generator CRDs, ownership policies, and migration strategies.

If you are looking for step-by-step instructions, see the README.

---

The Problem ESO Solves

Kubernetes Secrets are the standard way to pass sensitive data to pods. But secrets have to come from somewhere. The previous demo showed one approach: pods call the Vault API directly. That works, but it couples every application to Vault.

ESO decouples applications from secret stores. It runs as a controller in the cluster, watches for ExternalSecret custom resources, fetches secrets from external stores, and writes them into native Kubernetes Secrets. Applications consume those Secrets normally. They never know Vault (or AWS Secrets Manager, or any other provider) exists.

The demo application makes this explicit:

env:
  - name: DB_HOST
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_HOST
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_PASSWORD

The pod references a plain Kubernetes Secret called db-credentials. It uses standard secretKeyRef. No Vault SDK. No API calls. No sidecars. ESO created that Secret and keeps it in sync with Vault.

---

ESO Architecture

ESO has three main components that run as Deployments in the cluster.

The Controller

The core reconciliation engine. It watches ExternalSecret resources and reconciles them: fetch data from the external store, create or update the Kubernetes Secret, and report status. The controller is the heart of ESO.

The Webhook

A validating and mutating admission webhook. It validates ExternalSecret and SecretStore resources at creation time, catching misconfigurations before they reach the controller. It also handles conversion between API versions.

The Cert Controller

Manages the TLS certificates used by the webhook. Admission webhooks in Kubernetes require TLS. The cert controller generates and rotates these certificates automatically.

When you install ESO via Helm:

helm install external-secrets external-secrets/external-secrets \
  --namespace eso-demo \
  --set installCRDs=true

All three Deployments are created. The installCRDs=true flag registers the Custom Resource Definitions (SecretStore, ClusterSecretStore, ExternalSecret, ClusterExternalSecret, PushSecret, and the generator CRDs).

The Reconciliation Loop

ESO follows the standard Kubernetes controller pattern. The reconciliation loop for an ExternalSecret works like this:

1. Controller notices a new or changed ExternalSecret resource
2. Reads the referenced SecretStore to get provider configuration
3. Authenticates to the external provider using credentials from the SecretStore
4. Fetches the secret data specified in the ExternalSecret’s data or dataFrom
5. Applies any template transformations
6. Creates or updates the target Kubernetes Secret
7. Sets the ExternalSecret status (SecretSynced, SecretSyncedError, etc.)
8. Schedules the next sync based on refreshInterval

If any step fails, the controller logs the error, sets the status to reflect the failure, and retries on the next reconciliation cycle.

---

SecretStore vs ClusterSecretStore

These two CRDs define the connection to an external secret provider. The difference is scope.

SecretStore (Namespace-Scoped)

A SecretStore lives in a single namespace. ExternalSecrets in that namespace can reference it. ExternalSecrets in other namespaces cannot.

The demo uses a namespace-scoped SecretStore:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: eso-demo
spec:
  provider:
    vault:
      server: "http://vault.vault-demo.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          key: token

Key fields:

• provider.vault.server: the Vault API endpoint
• provider.vault.path: the secrets engine mount path
• provider.vault.version: v1 or v2 (for KV engine version)
• auth.tokenSecretRef: references a Kubernetes Secret containing the Vault token used for authentication

The auth token is stored in a separate Secret:

apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: eso-demo
stringData:
  token: root

In production, you would not use a root token. You would use a Vault token scoped to a specific policy, or use Kubernetes auth instead of token auth.

ClusterSecretStore (Cluster-Scoped)

A ClusterSecretStore is a cluster-wide resource. ExternalSecrets in any namespace can reference it. This is useful when a central platform team manages the connection to the secret store and individual teams create ExternalSecrets in their own namespaces.

The spec is identical to SecretStore. The only difference is the kind and the absence of a namespace in the metadata.

When an ExternalSecret references a ClusterSecretStore, it sets secretStoreRef.kind: ClusterSecretStore instead of SecretStore.

When to Use Which

Scenario	Use
Single team, single namespace	SecretStore
Multi-team, central secret management	ClusterSecretStore
Different credentials per namespace	SecretStore per namespace
Same provider, same credentials everywhere	ClusterSecretStore

---

ExternalSecret: data vs dataFrom

The ExternalSecret CRD declares which secrets to sync. Two approaches exist for specifying what to fetch: data and dataFrom.

data: Selective Field Extraction

The data field lets you pick individual keys from the external secret and map them to specific keys in the Kubernetes Secret. The demo’s database credential extraction uses this approach:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: eso-demo
spec:
  refreshInterval: 30s
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: DB_HOST
      remoteRef:
        key: demo/database
        property: host
    - secretKey: DB_USER
      remoteRef:
        key: demo/database
        property: username
    - secretKey: DB_PASSWORD
      remoteRef:
        key: demo/database
        property: password
    - secretKey: DB_PORT
      remoteRef:
        key: demo/database
        property: port

Each entry in the data array maps one remote property to one local key. remoteRef.key is the path in the external store. remoteRef.property is the specific field within that secret. secretKey is the key name in the resulting Kubernetes Secret.

This gives you full control over naming. The Vault secret has a field called host, but the Kubernetes Secret key is DB_HOST. Applications often expect environment variable naming conventions that differ from what the secret store uses.

dataFrom: Full Extraction

The dataFrom field pulls all keys from a path in one declaration. The demo’s API credential extraction uses this:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-credentials
  namespace: eso-demo
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: api-credentials
    creationPolicy: Owner
  dataFrom:
    - extract:
        key: demo/api

The extract directive fetches all key-value pairs from demo/api in Vault and copies them directly into the Kubernetes Secret. If Vault has key=sk-live-abc123 and provider=stripe, the Kubernetes Secret gets both keys with those exact names.

extract vs find

Within dataFrom, there are two modes:

• extract: fetches all keys from a single secret path. This is what the demo uses. You specify the exact path and get everything at that path.

• find: searches across multiple secrets using name patterns or tags. For example, you could find all secrets whose path matches a regex, or all secrets tagged with a specific label. This is powerful for dynamic environments where secret paths are not known ahead of time.

Combining data and dataFrom

You can use both data and dataFrom in the same ExternalSecret. The results are merged into a single Kubernetes Secret. If there are key conflicts, data entries take precedence over dataFrom entries.

---

Secret Templates

ESO supports Go templates for transforming secret data before writing it to the Kubernetes Secret. This is useful when applications expect secrets in a specific format that differs from how they are stored in the external provider.

Template Syntax

The target.template field accepts a Go template. Template variables come from the fetched secret data.

spec:
  target:
    name: db-connection
    template:
      type: Opaque
      data:
        connection_string: |
          postgresql://{{ .DB_USER }}:{{ .DB_PASSWORD }}@{{ .DB_HOST }}:{{ .DB_PORT }}/mydb
  data:
    - secretKey: DB_USER
      remoteRef:
        key: demo/database
        property: username
    - secretKey: DB_PASSWORD
      remoteRef:
        key: demo/database
        property: password
    - secretKey: DB_HOST
      remoteRef:
        key: demo/database
        property: host
    - secretKey: DB_PORT
      remoteRef:
        key: demo/database
        property: port

The resulting Kubernetes Secret contains a single key connection_string with the value postgresql://appuser:s3cret-passw0rd@postgres.example.com:5432/mydb.

Template Functions

ESO templates support standard Go template functions plus additional helpers:

• String manipulation: upper, lower, trim, replace
• Encoding: base64encode, base64decode
• JSON: fromJson, toJson
• PKCS12/JKS: for certificate format conversion

Templates also let you set the Secret’s type field. For example, you can create a kubernetes.io/dockerconfigjson Secret for image pull credentials by templating the .dockerconfigjson key in the correct format.

Template Metadata

Templates can set labels and annotations on the generated Secret:

spec:
  target:
    template:
      metadata:
        labels:
          managed-by: external-secrets
        annotations:
          last-synced: "{{ .timestamp }}"

---

Refresh Intervals and Sync Behavior

The refreshInterval field controls how often ESO polls the external store for changes. The demo uses two different intervals:

• db-credentials: 30 seconds
• api-credentials: 1 minute

How Refresh Works

ESO does not receive push notifications from external stores. It polls. On each refresh cycle:

1. ESO fetches the current value from the external store
2. Compares it to the existing Kubernetes Secret
3. If they differ, updates the Kubernetes Secret
4. Updates the ExternalSecret status with the sync timestamp

If the values have not changed, ESO still makes the API call to the external store but does not update the Kubernetes Secret. This means the refresh interval directly affects the load on your external secret store.

Choosing a Refresh Interval

Shorter intervals mean faster propagation of changes but more API calls. Longer intervals reduce load but delay updates.

Interval	Use Case
10-30s	Rapidly rotating secrets, development environments
1-5m	Production secrets that change occasionally
15-60m	Stable secrets that rarely change
0	Sync once, never refresh (use with caution)

A refresh interval of 0 (or omitted) syncs the secret once and never checks again. This is appropriate for secrets that are set once and never change, but it means you lose automatic sync.

Secret Updates and Pod Behavior

When ESO updates a Kubernetes Secret, the effect on pods depends on how the pod consumes the Secret:

• Environment variables: the pod does NOT see the change until it restarts. Environment variables are set at container start time.
• Volume mounts: Kubernetes updates the mounted file within 1-2 minutes (controlled by the kubelet sync period). The application sees the change if it re-reads the file.

The demo application uses secretKeyRef (environment variables), so it needs a restart to pick up changes.

---

PushSecret

PushSecret is the reverse of ExternalSecret. Instead of pulling secrets from an external store into Kubernetes, PushSecret writes Kubernetes Secrets to an external store.

Use cases:

• A CI/CD pipeline generates a credential in-cluster and needs to store it in Vault for other systems to consume
• Bootstrapping: creating initial secrets in an external store from Kubernetes resources
• Syncing secrets between clusters via a shared external store

PushSecret Spec

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: push-to-vault
spec:
  secretStoreRefs:
    - name: vault-backend
      kind: SecretStore
  selector:
    secret:
      name: my-k8s-secret
  data:
    - match:
        secretKey: api-key
        remoteRef:
          remoteKey: pushed/api-key
          property: value

This takes the api-key field from the my-k8s-secret Kubernetes Secret and writes it to pushed/api-key in Vault.

PushSecret is alpha-level in ESO. The API may change. But it fills an important gap in the secrets lifecycle.

---

Supported Providers

ESO supports a wide range of external secret stores. Each provider has its own authentication mechanisms and configuration.

HashiCorp Vault

What this demo uses. Supports token auth, Kubernetes auth, AppRole, LDAP, and JWT/OIDC auth methods. Works with KV v1, KV v2, and PKI secrets engines.

AWS Secrets Manager and Parameter Store

ESO authenticates to AWS using static credentials, IRSA (IAM Roles for Service Accounts), or pod identity. Secrets can be fetched from Secrets Manager (for JSON blobs) or SSM Parameter Store (for individual values).

GCP Secret Manager

ESO uses a GCP service account key or Workload Identity for authentication. Secrets are fetched by name and version.

Azure Key Vault

ESO authenticates via client credentials, managed identity, or workload identity federation. Supports secrets, keys, and certificates from Azure Key Vault.

1Password

ESO connects to 1Password via the 1Password Connect Server. Items from 1Password vaults are synced into Kubernetes Secrets. This is popular for teams that already use 1Password for credential management.

Other Providers

ESO also supports IBM Cloud Secrets Manager, CyberArk Conjur, Doppler, Keeper Security, Scaleway, Senhasegura, Oracle Vault, and others. The provider ecosystem grows with each release.

Provider Portability

One of ESO’s strengths is that switching providers requires changing the SecretStore, not the ExternalSecrets or the applications. The ExternalSecret references a SecretStore by name. If you migrate from Vault to AWS Secrets Manager, you create a new SecretStore pointing to AWS, update the ExternalSecrets to reference it (and adjust key paths), and the applications remain untouched.

---

Generator CRDs

ESO includes generator CRDs that produce secrets without fetching them from an external store. Generators are useful for bootstrap scenarios and testing.

Password Generator

Generates random passwords with configurable length, character sets, and complexity requirements. The generated password is stored in a Kubernetes Secret and can be pushed to an external store via PushSecret.

apiVersion: generators.external-secrets.io/v1alpha1
kind: Password
metadata:
  name: db-password
spec:
  length: 32
  digits: 6
  symbols: 4
  noUpper: false
  allowRepeat: true

Fake Generator

Generates fake data for testing. Produces realistic-looking but synthetic values for names, emails, addresses, and other fields. Useful for populating development environments with test data.

ECR Authorization Token Generator

Generates short-lived AWS ECR pull credentials. These credentials expire every 12 hours. The generator automatically refreshes them, solving the common problem of ECR image pull failures due to expired tokens.

apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-token
spec:
  region: us-east-1
  auth:
    secretRef:
      accessKeyID:
        name: aws-creds
        key: access-key
      secretAccessKey:
        name: aws-creds
        key: secret-key

---

Ownership and Garbage Collection

The creationPolicy field in the ExternalSecret’s target controls the relationship between the ExternalSecret and the Kubernetes Secret it creates.

Both ExternalSecrets in the demo use creationPolicy: Owner:

target:
  name: db-credentials
  creationPolicy: Owner

Three Creation Policies

Owner (default): ESO creates the Kubernetes Secret and sets an owner reference pointing back to the ExternalSecret. If you delete the ExternalSecret, Kubernetes garbage collection deletes the Secret too. ESO has full control over the Secret’s lifecycle.

Merge: ESO does not create the Secret. It expects the Secret to already exist and merges its data into it. This is useful when other controllers or processes also write to the same Secret. ESO only manages the keys it is responsible for. Deleting the ExternalSecret does not delete the Secret.

Orphan: ESO creates the Secret but does not set an owner reference. Deleting the ExternalSecret leaves the Secret behind. This is useful when you want to bootstrap a Secret with ESO but then manage it independently.

Deletion Policies

Separate from creation policies, ESO also has deletion policies that control what happens to the Secret’s data when the ExternalSecret is deleted:

• Retain: keep the Secret data as-is
• Delete: remove the Secret
• Merge: remove only the keys that ESO manages

---

Comparison with Alternatives

ESO is not the only way to bridge external secret stores and Kubernetes. Understanding the alternatives helps you choose the right tool.

Vault Agent Injector

The Vault Agent Injector runs as a mutating admission webhook. When a pod is created with specific annotations, the injector adds a Vault Agent sidecar container. The sidecar authenticates to Vault, fetches secrets, and writes them to a shared in-memory volume. The application reads secrets from files.

Advantages over ESO:

• Secrets never exist as Kubernetes Secret objects (higher security posture)
• Supports Vault-specific features like dynamic secrets and leases natively
• The sidecar renews leases and re-fetches secrets automatically

Disadvantages compared to ESO:

• Tightly coupled to Vault (no provider portability)
• Requires sidecar per pod (resource overhead)
• Applications must read files, not environment variables
• Every pod spec needs annotations, increasing complexity
• Pod startup is delayed while the sidecar fetches secrets

CSI Secret Store Driver

The Secrets Store CSI Driver mounts secrets from external stores as volumes. Provider-specific plugins handle the actual fetching. Vault, AWS, GCP, and Azure all have CSI provider plugins.

Advantages over ESO:

• Secrets are mounted directly into pods (no intermediate Kubernetes Secret by default, though sync-to-secret is optional)
• Uses the standard CSI interface

Disadvantages compared to ESO:

• Provider plugins vary in maturity and features
• Volume mount only (no environment variable support without sync-to-secret)
• No built-in refresh mechanism comparable to ESO’s polling
• Each provider plugin is a separate DaemonSet

Sealed Secrets

Bitnami Sealed Secrets takes a completely different approach. Secrets are encrypted client-side with a public key, committed to Git as SealedSecret resources, and decrypted in-cluster by the Sealed Secrets controller.

Advantages:

• Secrets can be safely stored in Git (GitOps friendly)
• No external secret store required
• Simple, single-purpose tool

Disadvantages compared to ESO:

• No dynamic secrets
• No audit trail of secret access
• No automatic rotation
• Single point of failure (the controller’s private key)
• Does not integrate with existing enterprise secret stores

When to Use ESO

ESO is the right choice when:

• You want applications decoupled from the secret store
• You need provider portability
• You want standard Kubernetes Secret consumption (env vars and volumes)
• You manage secrets centrally in an external store
• You do not want sidecars or DaemonSets per pod or node
• You need automatic refresh and sync

ESO is not the right choice when:

• You need secrets to never exist as Kubernetes Secret objects
• You need dynamic secrets with lease management in the application
• You only use Vault and want deep Vault integration

---

Migration from Sealed Secrets

Many teams start with Sealed Secrets for GitOps and later adopt an external secret store. ESO provides a migration path.

Step-by-Step Migration Strategy

1. Deploy ESO alongside Sealed Secrets. Both can coexist in the same cluster. They manage different Secret objects.

2. Create SecretStore and ExternalSecrets for the secrets you want to migrate. Use different target Secret names initially to avoid conflicts.

3. Update application references to point to the ESO-managed Secrets.

4. Delete the SealedSecret resources once applications are using the ESO-managed Secrets.

5. Uninstall Sealed Secrets controller when all secrets are migrated.

Considerations

• SealedSecrets are encrypted with the controller’s key pair. You cannot simply convert a SealedSecret to an ExternalSecret. You need the plaintext values in the external store first.
• Plan for a transition period where both systems run. Monitor both.
• Test the migration in a non-production environment first.

---

The Demo in Context

The demo flow creates a clear separation between the secret store and the application.

The SecretStore connects ESO to Vault using a token:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: eso-demo
spec:
  provider:
    vault:
      server: "http://vault.vault-demo.svc:8200"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: vault-token
          key: token

The db-credentials ExternalSecret selectively maps four fields, renaming them to match environment variable conventions:

data:
  - secretKey: DB_HOST
    remoteRef:
      key: demo/database
      property: host
  - secretKey: DB_USER
    remoteRef:
      key: demo/database
      property: username
  - secretKey: DB_PASSWORD
    remoteRef:
      key: demo/database
      property: password
  - secretKey: DB_PORT
    remoteRef:
      key: demo/database
      property: port

The api-credentials ExternalSecret extracts everything at demo/api:

dataFrom:
  - extract:
      key: demo/api

The application Deployment consumes both Secrets through secretKeyRef:

env:
  - name: DB_HOST
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_HOST
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: api-credentials
        key: key
  - name: API_PROVIDER
    valueFrom:
      secretKeyRef:
        name: api-credentials
        key: provider

The application code is a shell script that prints environment variables and loops. It has zero awareness of Vault or ESO. If you replaced Vault with AWS Secrets Manager tomorrow, you would change the SecretStore and update the ExternalSecret key paths. The Deployment stays the same.

---

Production Considerations

Authentication

The demo uses a Vault root token stored in a Kubernetes Secret. In production:

• Use Vault Kubernetes auth in the SecretStore so ESO authenticates with its own ServiceAccount token, not a static token
• Scope the Vault policy to only the paths ESO needs
• Rotate credentials regularly if using static tokens

RBAC

ESO needs RBAC permissions to create and update Secrets in the namespaces where ExternalSecrets exist. The Helm chart creates the necessary ClusterRoles and bindings. Review them to ensure they follow least privilege.

Monitoring

Watch these signals:

• ExternalSecret status conditions (SecretSynced, SecretSyncedError)
• ESO controller logs for authentication failures
• Kubernetes events on ExternalSecret resources
• Metrics: ESO exposes Prometheus metrics for sync operations, errors, and latency

Rate Limiting

Aggressive refresh intervals across many ExternalSecrets can overwhelm the external provider. Calculate the total request rate: (number of ExternalSecrets) / (average refresh interval). If you have 500 ExternalSecrets refreshing every 30 seconds, that is roughly 17 requests per second to Vault. Plan accordingly.

Namespace Strategy

For multi-team clusters:

• Use ClusterSecretStore for shared provider configuration
• Let each team create ExternalSecrets in their namespace
• Use Vault policies (or equivalent) to restrict which paths each namespace can access

---

Key Takeaways

• ESO is a Kubernetes controller that syncs external secrets into native Kubernetes Secret objects. Applications stay decoupled from the secret store.
• SecretStore is namespace-scoped. ClusterSecretStore is cluster-scoped. Choose based on your organizational model.
• data gives you field-level control with renaming. dataFrom.extract pulls everything from a path. dataFrom.find searches across paths.
• Secret templates transform fetched values before writing them to the Kubernetes Secret. Use Go template syntax.
• refreshInterval controls polling frequency. Balance freshness against API load.
• creationPolicy controls ownership. Owner enables garbage collection. Merge preserves existing Secrets. Orphan creates without ownership.
• PushSecret writes Kubernetes Secrets back to external stores.
• ESO is provider-agnostic. Switching from Vault to AWS Secrets Manager changes the SecretStore, not the application.
• Compared to Vault Agent Injector and CSI Secret Store Driver, ESO avoids sidecars and DaemonSets but does create Kubernetes Secret objects.

---

See Also

• README: step-by-step demo instructions
• Vault: the Vault deployment that this demo connects to
• ESO Documentation: official reference for all CRDs, providers, and configuration
• ESO API Reference: complete CRD specification