HashiCorp Vault: Deep Dive

This document explains the architecture, internals, and production considerations behind HashiCorp Vault. It covers the concepts that the demo touches on, and goes well beyond them into secrets engines, auth methods, dynamic secrets, and high-availability deployment patterns.

If you are looking for step-by-step instructions, see the README.

---

Vault Architecture

Vault is a secrets management platform. At a high level, it does three things: stores secrets, controls access to those secrets, and logs every interaction. Under the hood, it is more complex than that sentence suggests.

Core Components

Vault runs as a single binary. Inside it, several subsystems cooperate.

The Barrier is the most important concept. Vault encrypts everything before writing it to the storage backend. The barrier sits between Vault’s internal logic and the storage layer. Data passes through the barrier on the way in (encrypted) and on the way out (decrypted). Nothing in the storage backend is readable without the master key.

The Storage Backend is where encrypted data lives. Vault itself is stateless in the sense that it does not store data internally. It delegates persistence to a backend. Common choices include Consul, integrated Raft storage, S3, and PostgreSQL. In this demo, dev mode uses an in-memory backend:

helm install vault hashicorp/vault \
  --namespace vault-demo \
  --set "server.dev.enabled=true" \
  --set "server.dev.devRootToken=root" \
  --set "injector.enabled=false"

The server.dev.enabled=true flag starts Vault unsealed with data stored only in memory. Restart the pod and everything is gone.

The HTTP API is how all clients interact with Vault. The CLI, the web UI, and application code all talk to the same REST API on port 8200. Every operation is an API call. When the demo pod authenticates:

SA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

VAULT_TOKEN=$(wget -qO- --post-data "{\"jwt\": \"$SA_TOKEN\", \"role\": \"demo-app\"}" \
  http://vault.vault-demo.svc:8200/v1/auth/kubernetes/login 2>/dev/null | \
  sed 's/.*"client_token":"\([^"]*\)".*/\1/')

That is a POST to /v1/auth/kubernetes/login. Every subsequent read is a GET with the token in the X-Vault-Token header.

Seal and Unseal

When Vault starts in production mode, it is sealed. A sealed Vault can respond to API requests, but it cannot decrypt anything. It knows data exists in the storage backend, but it cannot read it.

The unsealing process works through Shamir’s Secret Sharing. During initialization, Vault generates a master key and splits it into N shares. A configurable threshold (M of N) of those shares must be provided to reconstruct the master key and unseal Vault. For example, you might have 5 shares with a threshold of 3. Any 3 keyholders can unseal Vault.

The master key itself does not encrypt data directly. It encrypts an encryption key, which encrypts the data. This indirection allows Vault to rekey (change the master key shares) without re-encrypting all stored data.

Auto-unseal replaces Shamir with a cloud KMS service. Instead of splitting the master key into shares, Vault wraps it with a KMS key (AWS KMS, GCP Cloud KMS, Azure Key Vault, or an HSM). On startup, Vault calls the KMS to unwrap the master key. No human intervention required.

Dev Mode vs Production Mode

Dev mode, which this demo uses, is a convenience shortcut.

Aspect	Dev Mode	Production Mode
Seal state	Starts unsealed	Starts sealed, requires unseal
Storage	In-memory only	Persistent backend (Raft, Consul, etc.)
TLS	Disabled	Required
Root token	Printed at startup	Generated during init, should be revoked
Audit	Not configured	Mandatory for compliance

Dev mode exists for development and testing. It is not hardened. The root token is root. TLS is off. The storage is ephemeral. Production deployments need every item in the right column.

---

Secrets Engines

A secrets engine is a component that stores, generates, or encrypts data. Vault ships with many built-in engines. You enable them at a path, and all operations under that path go to that engine.

KV (Key-Value)

The KV engine stores arbitrary key-value pairs. There are two versions.

KV v1 is a simple, non-versioned store. Writing to a key overwrites the previous value with no history. Reads return the current value. Simple and fast, but you lose previous values.

KV v2 adds versioning. Every write creates a new version. You can read previous versions, configure the maximum number of versions to retain, and soft-delete (destroy specific versions). This demo uses KV v2, which is the default in dev mode:

vault kv put secret/demo/database \
  username=appuser \
  password=s3cret-passw0rd \
  host=postgres.example.com \
  port=5432

The secret/ prefix maps to a KV v2 engine. Under the hood, KV v2 stores data at secret/data/demo/database and metadata at secret/metadata/demo/database. This distinction matters for policies, as we will see later.

Transit

The transit engine performs cryptographic operations without exposing keys. You send plaintext in, get ciphertext back. The encryption key never leaves Vault. This is “encryption as a service.”

Common use cases:

• Encrypting application data before writing it to a database
• Decrypting data at read time
• Signing and verifying payloads
• Generating random bytes or HMACs

The transit engine supports key rotation. When you rotate a key, new encryptions use the latest version, but old ciphertext remains decryptable. You can also rewrap old ciphertext to use the new key version without exposing plaintext.

PKI (Public Key Infrastructure)

The PKI engine generates X.509 certificates. You configure a Certificate Authority (root or intermediate), and Vault issues certificates on demand.

This solves the certificate lifecycle problem. Instead of requesting certificates through a ticketing system and waiting days, applications request short-lived certificates from Vault. If a certificate has a 24-hour TTL, compromise exposure is limited. Rotation happens automatically.

A typical PKI hierarchy in Vault:

1. An offline root CA (generated once, stored securely)
2. An intermediate CA managed by Vault
3. Leaf certificates issued by the intermediate for services and pods

Database

The database engine generates dynamic credentials. Instead of storing a static username and password, Vault connects to a database and creates a new user account for each request. Each credential set has a TTL. When the TTL expires, Vault revokes the credentials and drops the database user.

This is a powerful pattern. Every pod gets unique credentials. If a credential leaks, it expires quickly. You can trace any database query back to the specific pod that made it, because each pod has its own username.

Supported databases include PostgreSQL, MySQL, MongoDB, MSSQL, Oracle, Elasticsearch, Redis, and many others through plugin support.

---

Policies

Policies define what a client can do. They are written in HCL (HashiCorp Configuration Language) and specify paths with allowed capabilities.

HCL Syntax

A policy is a set of path blocks. Each path block names an API path and lists capabilities. The demo stores policies in a ConfigMap for reference:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-policies
  namespace: vault-demo
data:
  app-readonly.hcl: |
    # Read-only access to the demo secrets
    path "secret/data/demo/*" {
      capabilities = ["read", "list"]
    }
  app-admin.hcl: |
    # Full access to the demo secrets
    path "secret/data/demo/*" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "secret/metadata/demo/*" {
      capabilities = ["read", "list", "delete"]
    }

Capabilities

There are five standard capabilities:

Capability	HTTP Verb	Description
create	POST	Write to a path that does not exist
read	GET	Read from a path
update	PUT/POST	Write to a path that already exists
delete	DELETE	Delete a path
list	LIST	List entries under a path

Two special capabilities exist:

• deny overrides everything. If any policy attached to a token denies a path, access is denied regardless of other policies that allow it.
• sudo grants access to root-protected paths. Some administrative endpoints require sudo in addition to other capabilities.

Path Patterns

Paths support two wildcards:

• * matches any single path segment. secret/data/demo/* matches secret/data/demo/database but not secret/data/demo/team/database.
• + matches a single path component in the middle of a path. secret/data/+/database matches secret/data/team1/database and secret/data/team2/database.

The KV v2 Path Gotcha

KV v2 adds data/ and metadata/ segments to paths. When you run vault kv put secret/demo/database, the actual API path is secret/data/demo/database. Policies must use the full path including data/ or metadata/. This is a common source of confusion.

The app-readonly policy in this demo targets secret/data/demo/* (not secret/demo/*) for exactly this reason. The app-admin policy also includes secret/metadata/demo/* to allow listing and deleting version metadata.

Default Deny

Vault follows a default-deny model. If no policy explicitly grants access to a path, the request is denied. This is the opposite of many systems that grant access by default and require explicit deny rules.

---

Auth Methods

Auth methods are mechanisms that verify a client’s identity and map it to policies. Vault supports many auth methods. Each one handles a different identity source.

Token Auth

Every Vault interaction uses a token. All other auth methods ultimately produce a token. The token auth method is always enabled and cannot be disabled.

Tokens have properties: policies, TTL, renewal limits, and an accessor (a reference that can be used to look up or revoke the token without knowing the token itself). Tokens can be service tokens (persisted, renewable) or batch tokens (lightweight, not persisted, not renewable).

The root token has unlimited access. In production, generate it during initialization, use it for initial setup, then revoke it. Create new root tokens only when needed using the vault operator generate-root process.

Kubernetes Auth

This is the auth method used in the demo. It allows pods to authenticate to Vault using their Kubernetes ServiceAccount token.

The configuration in the demo:

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

vault write auth/kubernetes/role/demo-app \
  bound_service_account_names=demo-app \
  bound_service_account_namespaces=vault-demo \
  policies=app-readonly \
  ttl=1h

A role binds a ServiceAccount identity to Vault policies. The role named demo-app says: if a pod presents a JWT from ServiceAccount demo-app in namespace vault-demo, issue a Vault token with the app-readonly policy and a one-hour TTL.

How the Kubernetes Auth Method Works Internally

The authentication flow has several steps. Understanding them helps with troubleshooting.

1. Pod obtains its ServiceAccount token. Kubernetes automatically mounts a signed JWT at /var/run/secrets/kubernetes.io/serviceaccount/token. This token is a projected volume, rotated periodically by the kubelet.

2. Pod sends the JWT to Vault. The pod POSTs to /v1/auth/kubernetes/login with the JWT and the role name.

3. Vault validates the JWT. Vault sends a TokenReview request to the Kubernetes API server. The API server verifies the token’s signature, confirms it has not expired, and returns the ServiceAccount name and namespace.

4. Vault checks the role. Vault compares the ServiceAccount name and namespace from the TokenReview response against the role’s bound_service_account_names and bound_service_account_namespaces fields. If they match, authentication succeeds.

5. Vault issues a token. Vault creates a new token with the policies listed in the role and returns it to the pod.

Pod                     Vault                   K8s API Server
 |                        |                           |
 |-- POST /login (JWT) -->|                           |
 |                        |-- TokenReview (JWT) ----->|
 |                        |<-- SA name, namespace ----|
 |                        |                           |
 |                        | role lookup: match?        |
 |                        |                           |
 |<-- Vault token --------|                           |

The Vault pod needs network access to the Kubernetes API server for this to work. The demo configures the host using the $KUBERNETES_PORT_443_TCP_ADDR environment variable, which Kubernetes injects into every pod.

AppRole Auth

AppRole is designed for machine-to-machine authentication. It uses two credentials: a role ID (like a username, relatively static) and a secret ID (like a password, ephemeral and tightly controlled).

AppRole is a good fit for CI/CD pipelines. The pipeline knows its role ID. A trusted process delivers a secret ID just before the pipeline needs to authenticate. The secret ID can be single-use.

LDAP Auth

LDAP auth lets users authenticate with their corporate directory credentials. Vault queries the LDAP server to verify the username and password, then maps LDAP groups to Vault policies. This is common in enterprises where Active Directory or FreeIPA manages user identities.

OIDC Auth

OIDC auth integrates with identity providers like Keycloak, Okta, or Azure AD. Users authenticate through a browser-based flow. Vault validates the OIDC token, extracts claims (like group membership), and maps those claims to policies.

---

Dynamic Secrets

Dynamic secrets are generated on demand and automatically revoked. They are one of Vault’s most powerful features.

How Dynamic Database Credentials Work

Consider a PostgreSQL database. Instead of creating a static appuser account, you configure Vault with:

1. A connection to the database (using a privileged account)
2. A creation statement (SQL template for creating users)
3. A revocation statement (SQL template for dropping users)
4. A TTL (how long the credentials live)

When an application requests credentials, Vault:

1. Generates a random username and password
2. Connects to PostgreSQL and runs the creation SQL
3. Returns the username, password, and lease ID to the application
4. Starts a TTL timer

When the TTL expires (or the lease is revoked), Vault connects to PostgreSQL and runs the revocation SQL to drop the user.

The result: every application instance gets unique, short-lived credentials. There is no shared password to leak. There is no password to rotate manually. And every database query can be traced to a specific application instance.

Leases

Every dynamic secret has a lease. The lease tracks:

• Lease ID: a unique identifier for this specific secret instance
• TTL: how long the secret is valid
• Renewable: whether the application can extend the TTL

Applications can renew a lease before it expires. Renewal extends the TTL without generating a new secret. There is typically a maximum TTL beyond which renewal is not possible, forcing the application to re-authenticate and get fresh credentials.

If an application crashes without revoking its lease, the TTL still expires and Vault revokes the credentials. No orphaned accounts.

---

Lease Management and Renewal

Lease management is central to how Vault handles the lifecycle of secrets.

Renewal

A client with a renewable lease can call /v1/sys/leases/renew with the lease ID. Vault extends the TTL, typically by the original increment. Renewal is not guaranteed. If the maximum TTL has been reached, Vault denies the renewal and the client must request a new secret.

Revocation

Clients can explicitly revoke a lease by calling /v1/sys/leases/revoke. Administrators can revoke all leases under a prefix with /v1/sys/leases/revoke-prefix. When a lease is revoked, Vault calls back to the secrets engine (for example, the database engine drops the user).

Token TTLs vs Secret TTLs

Both tokens and secrets have TTLs, and they interact. If a token expires, all secrets issued under that token are revoked. This creates a cascading revocation model. Revoking a token cleans up everything associated with it.

In the demo, the Vault token issued to the pod has a 1-hour TTL:

vault write auth/kubernetes/role/demo-app \
  bound_service_account_names=demo-app \
  bound_service_account_namespaces=vault-demo \
  policies=app-readonly \
  ttl=1h

If the pod holds dynamic secrets under that token, they are all revoked when the token expires (if they have not expired already on their own).

---

Audit Devices

Vault can log every authenticated interaction. These logs are called audit logs, and the components that produce them are called audit devices.

Vault supports three types of audit devices:

• File: writes JSON audit logs to a file path
• Syslog: sends logs to a syslog endpoint
• Socket: sends logs over a network socket (TCP or UDP)

Every API request and response is logged. The logs include the request path, the client token accessor (not the token itself), the request data, and the response. Sensitive values in the request and response are HMAC’d using a salt, so the audit log does not contain plaintext secrets. But you can verify whether a specific value was read by HMAC’ing it with the same salt and comparing.

In production, Vault requires at least one audit device to be functional. If all audit devices fail (for example, the disk is full), Vault stops processing requests. This is a deliberate design choice: Vault would rather stop serving than serve without an audit trail.

---

High Availability

Vault supports high availability through leader election. Only one Vault node handles requests at a time (the active node). Other nodes are standby nodes. If the active node fails, a standby node takes over.

Integrated Raft Storage

The recommended HA storage backend is Vault’s integrated Raft storage. It uses the Raft consensus protocol to replicate data across Vault nodes. Raft provides:

• Automatic leader election
• Data replication across all nodes
• No external dependency (unlike Consul)

A typical production deployment runs 3 or 5 Vault nodes. Three nodes tolerate one failure. Five nodes tolerate two failures. Using an odd number avoids split-brain scenarios.

With Raft, each Vault node stores a copy of the encrypted data locally. Writes go to the leader, which replicates them to followers. Reads can be served from the leader or (with performance standby nodes in Enterprise) from followers.

Standby Nodes and Request Forwarding

Standby nodes receive client requests but forward them to the active node. From the client’s perspective, any Vault node can accept requests. The Kubernetes Service in front of Vault pods distributes traffic, and standby nodes handle the forwarding transparently.

---

Production Deployment Considerations

Moving from dev mode to production requires addressing several areas.

Auto-Unseal with Cloud KMS

Manual unsealing with Shamir shares requires human intervention on every restart. For Kubernetes deployments where pods can restart at any time, this is unacceptable. Auto-unseal delegates the master key protection to a cloud KMS:

• AWS KMS: Vault wraps the master key with an AWS KMS key
• GCP Cloud KMS: Same pattern with a GCP key
• Azure Key Vault: Same pattern with an Azure key
• HSM (PKCS#11): For on-premises deployments with hardware security modules

The Vault pod needs IAM permissions (or equivalent) to call the KMS. On OpenShift, this can be configured via pod identity or mounted credentials.

TLS

Production Vault must use TLS. All API communication should be encrypted in transit. Options:

• Vault manages its own TLS certificate (configured in the listener stanza)
• A reverse proxy or service mesh terminates TLS in front of Vault
• On OpenShift, a Route with TLS termination can front the Vault service

For internal service-to-service communication, Vault can use its own PKI engine to issue its own certificate. That is a satisfying bit of self-referential infrastructure.

Namespace Isolation

Vault Enterprise supports namespaces, which are isolated environments within a single Vault cluster. Each namespace has its own secrets engines, auth methods, and policies. This allows multi-team or multi-tenant usage without risk of cross-contamination.

In the open-source version, isolation is achieved through careful policy design. The app-readonly policy in the demo scopes access to secret/data/demo/*. A different team would have policies scoped to secret/data/their-team/*.

Resource Sizing

Vault is not resource-hungry for small deployments, but sizing depends on usage patterns. Considerations:

• CPU: cryptographic operations (transit engine, PKI) are CPU-bound
• Memory: Vault caches frequently accessed secrets
• Disk: Raft storage grows with the number of secrets and versions
• Network: audit log volume can be significant under high request rates

Backup and Recovery

With integrated Raft storage, backups are taken with vault operator raft snapshot save. The snapshot is an encrypted file. Restore with vault operator raft snapshot restore. Automate this on a schedule.

Monitoring

Vault exposes telemetry data. Key metrics to watch:

• vault.core.handle_request: request latency
• vault.expire.num_leases: number of active leases
• vault.runtime.alloc_bytes: memory usage
• vault.audit.log_request_failure: audit device failures (critical)
• vault.barrier.get / vault.barrier.put: storage backend performance

---

The Demo in Context

The demo deploys Vault in dev mode, stores KV v2 secrets, creates policies, enables Kubernetes auth, and has a pod authenticate and read secrets.

The test app manifest creates a ServiceAccount and a Deployment:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-app
  namespace: vault-demo
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo-app
  namespace: vault-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: demo-app
  template:
    metadata:
      labels:
        app: demo-app
    spec:
      serviceAccountName: demo-app
      containers:
        - name: app
          image: busybox:1.36
          command:
            - /bin/sh
            - -c
            - |
              echo "Demo app running with ServiceAccount: demo-app"
              echo "Waiting for secrets to be available..."
              sleep infinity

The serviceAccountName: demo-app is critical. That is what produces the JWT that gets mounted at /var/run/secrets/kubernetes.io/serviceaccount/token. That JWT is what the pod sends to Vault for authentication. Without the correct ServiceAccount, the Vault role binding fails and authentication is denied.

The policies ConfigMap preserves the HCL files as data:

apiVersion: v1
kind: ConfigMap
metadata:
  name: vault-policies
  namespace: vault-demo
data:
  app-readonly.hcl: |
    path "secret/data/demo/*" {
      capabilities = ["read", "list"]
    }
  app-admin.hcl: |
    path "secret/data/demo/*" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }
    path "secret/metadata/demo/*" {
      capabilities = ["read", "list", "delete"]
    }

This ConfigMap is for documentation purposes. The actual policies are loaded into Vault via vault policy write commands run inside the pod. In a production setup, you would automate policy loading as part of your Vault configuration management, potentially using Terraform’s Vault provider.

---

Common Patterns Beyond the Demo

Vault Agent Injector

The Vault Agent Injector is a Kubernetes mutating admission webhook. It intercepts pod creation requests and injects a Vault Agent sidecar container. The sidecar handles authentication and writes secrets to a shared volume. The application reads secrets from files, with no Vault SDK or API calls needed.

The demo disabled the injector (injector.enabled=false) to keep things simple. In production, the injector is a common pattern for legacy applications that cannot be modified to call the Vault API.

Vault CSI Provider

The Vault CSI Provider integrates with the Kubernetes Secrets Store CSI Driver. It mounts Vault secrets as volumes in pods. Like the injector, it keeps Vault details out of application code. Unlike the injector, it uses the CSI interface rather than sidecars.

External Secrets Operator

ESO (covered in demo 29) takes a different approach. Instead of injecting secrets into pods, it syncs Vault secrets into native Kubernetes Secret objects. Applications consume standard secretKeyRef environment variables. ESO handles authentication, polling, and synchronization in the background.

---

Vault vs Alternatives

Vault vs Kubernetes Secrets

Kubernetes Secrets are base64 encoded, not encrypted at rest by default (unless you configure etcd encryption). Anyone with namespace access can read them. No audit trail. No dynamic secrets. No fine-grained policies.

Vault provides encryption at rest and in transit, per-path policies, full audit logging, dynamic secrets, and lease management. The trade-off is operational complexity.

Vault vs Cloud-Native Secret Managers

AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault are managed services. They handle the operational burden for you. Vault is self-hosted (or uses HCP Vault, the managed offering). Vault’s advantage is portability, richer feature set (transit, PKI, dynamic database credentials), and fine-grained policy control. The cloud services win on simplicity and operational overhead.

Vault vs Sealed Secrets

Bitnami Sealed Secrets encrypt Kubernetes Secrets client-side so they can be stored safely in Git. The Sealed Secrets controller in the cluster decrypts them. This solves the “secrets in Git” problem but does not provide dynamic secrets, audit logging, or fine-grained access control. Sealed Secrets is a simpler tool solving a narrower problem.

---

Key Takeaways

• Vault encrypts all data through the barrier before it reaches the storage backend. The seal/unseal mechanism protects the encryption key.
• Dev mode is for learning and testing only. Production requires persistent storage, TLS, auto-unseal, and audit devices.
• KV v2 adds versioning and introduces the data/ and metadata/ path segments that policies must account for.
• The Kubernetes auth method uses the TokenReview API to validate ServiceAccount JWTs. The Vault role maps ServiceAccount identity to policies.
• Dynamic secrets (database, PKI) generate unique, short-lived credentials per request. Leases track their lifecycle.
• Policies follow a default-deny model. Capabilities control what operations are allowed on each path.
• Production deployments need auto-unseal, TLS, audit logging, HA with Raft, and careful policy design.

---

See Also

• README: step-by-step demo instructions
• External Secrets: sync Vault secrets into Kubernetes Secrets using ESO
• Vault Documentation: official reference for all secrets engines, auth methods, and configuration
• Vault API Reference: complete API specification