ConfigMaps & Secrets: Deep Dive

This document explains how Kubernetes ConfigMaps and Secrets work under the hood. It covers storage internals, update propagation, encryption, external secret management, and the trade-offs between different injection methods.

How ConfigMaps and Secrets Are Stored

ConfigMaps and Secrets are first-class API objects stored in etcd. They live in a specific namespace and follow the same lifecycle as any other Kubernetes resource.

A ConfigMap holds arbitrary key-value pairs in its data field. Values are plain UTF-8 strings. Binary data goes in binaryData as base64.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: config-demo
data:
  APP_ENV: "development"
  APP_LOG_LEVEL: "debug"
  APP_PORT: "8080"
  APP_FEATURE_FLAGS: "dark-mode=true,beta-api=false"

A Secret is structurally identical to a ConfigMap, but with one behavioral difference: the API server base64-encodes values in the data field. The stringData field accepts plain text and converts it automatically.

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: config-demo
type: Opaque
stringData:
  DB_HOST: "postgres.config-demo.svc"
  DB_USER: "appuser"
  DB_PASSWORD: "s3cret-passw0rd"
  DB_NAME: "myapp"

stringData vs data

The stringData field is a write-only convenience. When you kubectl apply a Secret with stringData, the API server converts it to base64 and stores it in data. If you then kubectl get secret -o yaml, you will only see data with base64 values. The original stringData is gone.

The data field requires you to provide base64 values yourself:

data:
  DB_PASSWORD: czNjcmV0LXBhc3N3MHJk    # echo -n 's3cret-passw0rd' | base64

Use stringData when writing manifests by hand. Use data when generating manifests programmatically or when you need exact byte control.

Secret Types

Kubernetes defines several Secret types, each with different validation rules:

Type	Purpose	Required Keys
Opaque	General-purpose	None
kubernetes.io/dockerconfigjson	Container registry auth	.dockerconfigjson
kubernetes.io/tls	TLS certificates	tls.crt, tls.key
kubernetes.io/basic-auth	Basic authentication	username, password
kubernetes.io/ssh-auth	SSH authentication	ssh-privatekey
kubernetes.io/service-account-token	Service account token	Auto-populated

The demo uses both Opaque and dockerconfigjson:

apiVersion: v1
kind: Secret
metadata:
  name: registry-auth
  namespace: config-demo
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6e319

The type field triggers validation. A kubernetes.io/tls Secret without tls.crt will be rejected by the API server.

Two Injection Methods

Kubernetes provides two ways to deliver ConfigMap and Secret data to containers: environment variables and volume mounts. Each has different update semantics.

Environment Variables (envFrom and env)

The envFrom field injects every key from a ConfigMap or Secret as an environment variable. The env field with valueFrom injects individual keys.

containers:
  - name: app
    envFrom:
      - configMapRef:
          name: app-config        # All keys become env vars
    env:
      - name: DATABASE_HOST
        valueFrom:
          secretKeyRef:
            name: db-credentials
            key: DB_HOST           # Single key injection
      - name: VERSION
        valueFrom:
          configMapKeyRef:
            name: app-version
            key: VERSION           # Single key from ConfigMap

The critical limitation: environment variables are set at container start and never updated. If you change the ConfigMap after the pod is running, the environment variables keep their original values. The pod must be restarted to pick up changes.

With envFrom, key names map directly to environment variable names. If a ConfigMap key is not a valid environment variable name (contains dots, starts with a digit), it is silently skipped. No error, no warning. This catches people off guard.

Volume Mounts

Volume mounts project ConfigMap or Secret data as files in the container filesystem. The demo mounts nginx configuration and HTML from a ConfigMap:

containers:
  - name: nginx
    volumeMounts:
      - name: nginx-conf
        mountPath: /etc/nginx/conf.d/default.conf
        subPath: nginx.conf
      - name: html
        mountPath: /usr/share/nginx/html/index.html
        subPath: index.html
      - name: db-creds
        mountPath: /etc/secrets
        readOnly: true
volumes:
  - name: nginx-conf
    configMap:
      name: nginx-config
  - name: html
    configMap:
      name: nginx-config
  - name: db-creds
    secret:
      secretName: db-credentials

When you mount an entire directory (no subPath), each key becomes a file. The Secret db-credentials mounted at /etc/secrets creates files like /etc/secrets/DB_USER, /etc/secrets/DB_PASSWORD, etc.

Hot-Reload Mechanics

This is where ConfigMaps and Secrets behave very differently depending on how you mount them.

Projected Volumes (Full Directory Mount)

When you mount a ConfigMap or Secret as a full directory (no subPath), the kubelet uses a symlink-based atomic update mechanism:

1. The kubelet watches the API server for ConfigMap/Secret changes.
2. When a change is detected, it writes the new content to a new timestamped directory.
3. It atomically swaps the symlink from the old directory to the new one.

The result: all files update at once, atomically. The propagation delay is the sum of:

• kubelet sync period (default: 60 seconds)
• ConfigMap cache TTL (configurable, default depends on watch mode)

Total delay is typically 30 to 90 seconds.

subPath Mounts Block Updates

The demo uses subPath to mount individual files:

volumeMounts:
  - name: nginx-conf
    mountPath: /etc/nginx/conf.d/default.conf
    subPath: nginx.conf    # <-- This prevents hot-reload

A volume mount using subPath is never updated. The kubelet copies the file at pod start and does not touch it again. This is a very common source of confusion.

The reason: subPath mounts bind a specific file or directory, bypassing the symlink mechanism. There is no symlink to atomically swap.

The Fix: Projected Volumes

If you need hot-reload with individual file mounts, use projected volumes instead:

volumes:
  - name: config
    projected:
      sources:
        - configMap:
            name: nginx-config
            items:
              - key: nginx.conf
                path: nginx.conf

Projected volumes support atomic updates while allowing you to select specific keys.

How the Kubelet Watches for Changes

The kubelet has two modes for syncing ConfigMap and Secret data:

1. Watch-based (default since v1.19): The kubelet opens a watch stream against the API server. Changes are pushed in near-real-time. This is efficient but requires one watch per ConfigMap/Secret.

2. TTL-based cache: The kubelet caches ConfigMap/Secret data and refreshes at a configurable interval. Less API server load but slower propagation.

3. Get-based: No caching. The kubelet fetches from the API server on every sync period. Simple but creates high API server load.

The mode is set via --config-map-and-secret-change-detection-strategy on the kubelet.

Immutable ConfigMaps

The demo declares an immutable ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-version
  namespace: config-demo
immutable: true
data:
  VERSION: "1.2.3"
  BUILD_SHA: "abc123def456"

Setting immutable: true has two effects:

1. Protection: Any attempt to modify the data is rejected by the API server. You must delete and recreate.

2. Performance: The kubelet stops watching the ConfigMap for changes. In clusters with thousands of ConfigMaps, this significantly reduces API server load. Each watch consumes memory on the API server. Immutable ConfigMaps eliminate that overhead entirely.

This is particularly valuable for configuration that should never change during a deployment, like build metadata, version strings, or feature flags tied to a release.

The trade-off: you cannot update in place. You must delete, recreate, and restart any pods that reference it.

Secret Encryption at Rest

By default, Secrets are stored in etcd as base64-encoded plaintext. Anyone with etcd access can read them. This is not encryption.

EncryptionConfiguration

Kubernetes supports encrypting Secrets at rest through EncryptionConfiguration. You create a configuration file on the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}    # Fallback for reading unencrypted secrets

The API server encrypts Secret data before writing to etcd and decrypts on read. The identity provider at the bottom allows reading old unencrypted Secrets during migration.

Supported providers:

Provider	Strength	Notes
identity	None	Default, plaintext
aescbc	Strong	AES-CBC with PKCS7 padding
aesgcm	Strong	AES-GCM, must rotate keys frequently
kms v1/v2	Strongest	Delegates to external KMS (Vault, AWS KMS, etc.)
secretbox	Strong	Uses XSalsa20 and Poly1305

The kms provider is the recommended approach for production. It keeps the encryption key in an external system (HSM, cloud KMS) and only sends data encryption keys to the API server.

What Encryption at Rest Does NOT Protect Against

Encryption at rest protects against someone reading raw etcd data. It does not protect against:

• Users with RBAC access to the Secret (they get decrypted values via the API)
• Container processes that receive Secret data as env vars or files
• Logs that accidentally print Secret values
• etcd backups that include unencrypted snapshots (if made before encryption was enabled)

External Secret Operators

For production systems, external secret operators pull secrets from dedicated vault systems and sync them into Kubernetes Secrets.

External Secrets Operator (ESO)

ESO defines two CRDs:

• SecretStore / ClusterSecretStore: Connection to the external provider (Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager).
• ExternalSecret: Declares which external secrets to sync and how to map them to Kubernetes Secret keys.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-credentials          # Resulting K8s Secret name
  data:
    - secretKey: DB_PASSWORD      # Key in the K8s Secret
      remoteRef:
        key: prod/database        # Path in Vault
        property: password        # Field in Vault

ESO periodically refreshes the Kubernetes Secret from the external source. If the external secret changes, the Kubernetes Secret is updated.

HashiCorp Vault Agent Injector

The Vault agent injector takes a different approach. It uses a mutating admission webhook to inject a Vault Agent sidecar into pods. The sidecar authenticates with Vault, retrieves secrets, and writes them as files in a shared volume.

annotations:
  vault.hashicorp.com/agent-inject: "true"
  vault.hashicorp.com/role: "myapp"
  vault.hashicorp.com/agent-inject-secret-db-password: "secret/data/db"

The key difference: ESO creates Kubernetes Secrets (visible via kubectl get secret), while Vault Agent Injector writes files directly into the pod (no Kubernetes Secret object).

Choosing Between Them

Factor	ESO	Vault Agent
Kubernetes Secret created?	Yes	No
Multi-provider support	Yes (many)	Vault only
Secret rotation	Polling-based	Agent-based
Additional sidecar?	No	Yes (per pod)
Works with env vars?	Yes	No (files only)

envFrom vs env: Trade-offs

envFrom

envFrom:
  - configMapRef:
      name: app-config

Injects all keys at once. Simple. But you lose control over naming. If the ConfigMap has a key called PATH, it will override the system PATH variable. There is no key filtering or renaming.

You can set a prefix to namespace the variables:

envFrom:
  - configMapRef:
      name: app-config
    prefix: APP_CONFIG_

Now APP_ENV becomes APP_CONFIG_APP_ENV.

env with valueFrom

env:
  - name: DATABASE_HOST
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: DB_HOST

More verbose but more explicit. You control the variable name. You can mix sources. You can mark keys as optional:

env:
  - name: OPTIONAL_KEY
    valueFrom:
      configMapKeyRef:
        name: maybe-exists
        key: some-key
        optional: true    # Pod starts even if ConfigMap missing

Without optional: true, a missing ConfigMap or key prevents the pod from starting.

File-Based ConfigMaps

The demo demonstrates multi-line file data in a ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: config-demo
data:
  nginx.conf: |
    server {
        listen 8080;
        server_name localhost;
        location / {
            root /usr/share/nginx/html;
            index index.html;
        }
        location /health {
            access_log off;
            return 200 "healthy\n";
            add_header Content-Type text/plain;
        }
    }
  index.html: |
    <!DOCTYPE html>
    <html>
    <head><title>Config Demo</title></head>
    <body>
      <h1>ConfigMap File Mount Demo</h1>
    </body>
    </html>

Each key in the data field becomes a separate file when volume-mounted. The YAML pipe (|) preserves newlines. This is the standard pattern for injecting configuration files, HTML templates, scripts, or any multi-line content.

The maximum size of a ConfigMap is 1 MiB. This is an etcd limit. If your configuration exceeds this, split it across multiple ConfigMaps or use a different storage mechanism.

ConfigMap and Secret Size Limits

Both ConfigMaps and Secrets are capped at 1 MiB by etcd. But there is a practical difference in how that plays out.

Secret data is base64-encoded, which adds ~33% overhead. A 1 MiB Secret can only hold about 750 KiB of actual data.

For large TLS certificate bundles or configuration files, this limit matters. Consider mounting from PersistentVolumes or init containers that fetch from external storage.

Common Pitfalls

1. Forgetting That subPath Blocks Updates

If you use subPath and expect hot-reload, you will be disappointed. Use projected volumes or restart pods after changes.

2. Base64 Is Not Encryption

Anyone with namespace RBAC access can decode Secrets. Treat RBAC as your first line of defense, not base64.

3. ConfigMap Key Naming

Keys with dots (app.properties) work in volume mounts (they become filenames) but fail silently with envFrom because dots are not valid in environment variable names.

4. Missing ConfigMap Blocks Pod Start

If a pod references a ConfigMap that does not exist and the reference is not marked optional, the pod stays in ContainerCreating forever. The events show MountVolume.SetUp failed.

5. Immutable Means Immutable

You cannot flip immutable back to false. You cannot modify the data. Delete and recreate is the only path.

Summary of Update Behavior

Injection Method	Auto-Updates?	Delay	Notes
envFrom / env	No	N/A	Requires pod restart
Volume mount (full dir)	Yes	30-90s	Atomic via symlink swap
Volume mount + subPath	No	N/A	File copied at start
Projected volume	Yes	30-90s	Supports key selection

Related Resources

• Kubernetes Docs: ConfigMaps
• Kubernetes Docs: Secrets
• Kubernetes Docs: Encrypt Data at Rest
• External Secrets Operator