Module 9.8: Secrets Management Deep Dive

Complexity: [COMPLEX] | Time to Complete: 2h | Prerequisites: Module 9.1 (Databases), Kubernetes RBAC, cloud IAM basics

What You’ll Be Able to Do

After completing this module, you will be able to make defensible design choices about secret storage, delivery, rotation, and Kubernetes integration:

• Implement External Secrets Operator to synchronize cloud secrets (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) into Kubernetes
• Configure automatic secret rotation workflows that update Kubernetes secrets without pod restarts
• Deploy HashiCorp Vault on Kubernetes with cloud KMS auto-unseal and the Vault Secrets Operator
• Design multi-cloud secret management architectures that work consistently across EKS, GKE, and AKS clusters

---

Why This Module Matters

Hypothetical scenario: a payments team ships a weekend hotfix and accidentally commits a .env file that contains a cloud access key, a database password, and a webhook signing secret. The repository is private, but the key is copied into a build log, a contractor’s laptop has a stale clone, and an application container still exposes the password as an environment variable after the Git mistake is removed. The incident is no longer one secret in one file; it is a lifecycle failure across generation, storage, distribution, rotation, audit, and revocation.

The root cause is not just developer carelessness. It is an architecture that lets long-lived static credentials exist in many places at once: source control, CI variables, local laptops, container images, Kubernetes Secret objects, application logs, and cloud consoles. A secret that is easy to copy is hard to revoke completely, because every consumer must be found, every cache must be refreshed, every dependent pod must reload, and every audit trail must prove that the old value stopped being used.

Modern secrets management does not mean “put the password in a nicer box.” It means designing a system where humans rarely see plaintext, workloads authenticate with short-lived identity instead of stored cloud keys, rotation is routine rather than heroic, and access can be audited after the fact. This module teaches the full Kubernetes integration path across AWS Secrets Manager, AWS Systems Manager Parameter Store, Google Secret Manager, Azure Key Vault, External Secrets Operator, Secrets Store CSI Driver, and HashiCorp Vault so you can defend the design in a real platform review.

---

The Secret Lifecycle

Secrets management is easiest to reason about as a lifecycle: generation, storage, distribution, rotation, audit, and revocation. Generation decides whether the secret is a random value, a database credential, a TLS private key, a cloud token, or a dynamic lease minted for one workload. Storage decides whether that value lives in AWS Secrets Manager, AWS Systems Manager Parameter Store, Google Secret Manager, Azure Key Vault, Vault, Kubernetes etcd, or Git as encrypted ciphertext. Distribution decides how the application sees the value: an environment variable, a mounted file, a native SDK call, or a dynamic credential broker.

Rotation is where many otherwise polished designs fail. Changing the value in a cloud secret store is only the first step; the new value must reach Kubernetes, the application process must reload it, and the old credential must remain valid long enough for a safe transition or be revoked quickly enough to limit exposure. Audit answers a different question: not “is the secret encrypted,” but “who or what accessed it, from where, under which identity, and was that access expected.” Revocation closes the loop by disabling old versions, deleting temporary database users, removing unused IAM permissions, and proving that stale consumers are gone.

The dangerous shortcut is to treat the secret object as the lifecycle. A Kubernetes Secret can store a password, and a cloud secret manager can store a password, but neither automatically gives you safe generation, distribution, rotation, audit, and revocation. AWS Secrets Manager gives you native rotation workflows for many secret types through Lambda-backed rotation or managed rotation, but your pods still need a reload path. Google Secret Manager gives strong versioning semantics and IAM-based access, but an application pinned to an old version will not magically move to a new alias unless you design for that. Azure Key Vault stores secrets, keys, and certificates with recoverability controls such as soft-delete, but a workload still needs Microsoft Entra-based access and a rotation propagation model.

The Kubernetes angle is that cluster objects are excellent orchestration metadata and poor long-term secret repositories. GitOps wants declarative manifests, but plaintext secrets do not belong in Git. Pods want fast local access, but long-lived environment variables are copied into the process environment at start time and are awkward to rotate without restart. Operators want consistent workflows across EKS, GKE, and AKS, but each cloud has a different identity system, audit log surface, quota model, and pricing model. The practical answer is usually a layered design: cloud or Vault as the source of truth, workload identity for keyless access, ESO or CSI for Kubernetes delivery, and application reload mechanics for rotation.

The Secrets Management Analogy

Think of a secret like a master key in a large building. Locking the key in a cabinet is storage; deciding who can open the cabinet is access control; replacing the lock on a schedule is rotation; checking the sign-out sheet is audit; and changing the doors so the old key stops working is revocation. A mature platform does all five, because a beautiful cabinet does not help if every contractor has a photocopy.

---

The Kubernetes Secrets Problem

What Kubernetes Secrets Actually Are

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=        # base64("admin")
  password: cDRzc3cwcmQ=    # base64("p4ssw0rd")

Kubernetes Secrets are base64-encoded, not encrypted, so the following example shows why encoded manifest data should never be treated as protected ciphertext:

alias k=kubectl
k get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
# Output: p4ssw0rd

The base64 encoding exists because Kubernetes API objects are structured data and some secret values are binary. It is not a security boundary, and Kubernetes documentation explicitly warns that encoded values are only obscured. The more important default is that Secret objects are stored in the API server’s backing data store, etcd, and they are unencrypted there unless the control plane is configured with encryption at rest. Anyone with direct etcd access, broad API permissions, or the ability to create pods that mount secrets in a namespace can potentially recover values.

What Kubernetes Does and Does Not Provide

Feature	Kubernetes Native	What You Actually Need
Storage	etcd (encrypted at rest if configured)	External vault with audit logging
Access control	RBAC (namespace-level)	Attribute-based access with MFA
Rotation	Manual (delete and recreate)	Automatic with zero-downtime
Auditing	API audit logs (if enabled)	Who accessed what, when, from where
Dynamic secrets	Not supported	Short-lived, auto-expiring credentials
Git safety	Plaintext in manifests	Encrypted at rest in Git

This table should not make you dismiss Kubernetes Secrets entirely. They are still the native interface many workloads, Helm charts, and controllers expect, and the kubelet can project them into a pod as files with filesystem update semantics. The problem is treating a Kubernetes Secret as the source of truth. A safer model treats it as a local delivery artifact that can be recreated from an external source, restricted by namespace RBAC, protected by etcd encryption, and monitored through Kubernetes audit logs.

The RBAC implication is subtle and important. Granting get on a Secret lets the subject retrieve a specific value; granting list or watch on secrets can expose every value in that scope because the API response includes secret data. Granting a developer permission to create arbitrary pods in a namespace can also be equivalent to granting indirect secret read access, because that pod can mount any Secret allowed in the namespace and print it. Production clusters therefore split duties: application teams can deploy workloads, but only platform-owned controllers and tightly scoped service accounts can read broad secret sets.

Etcd encryption at rest is necessary but not sufficient. Kubernetes supports an EncryptionConfiguration, including KMS provider options, so the API server stores encrypted secret data instead of plaintext in etcd. That protects against a class of datastore compromise, but it does not protect against an authorized API read, an overbroad service account, a pod that echoes secrets to logs, or an application that loads secrets into environment variables forever. For Kubernetes 1.35 curriculum work, treat KMS-backed encryption, least-privilege RBAC, namespace isolation, audit logging, and external stores as the minimum baseline rather than optional hardening.

Plain environment variables deserve special scrutiny. They are convenient, widely supported, and often required by legacy applications, but they are captured when the process starts and do not update just because the Kubernetes Secret changes. They can also appear in crash diagnostics, process inspection tools, or overly verbose debug output. Mounted files are usually easier to rotate because the application can watch a path, reopen the file, or reload configuration on a signal, but even file mounts require deliberate application behavior. Secrets management is therefore an application design topic, not just a cluster administration topic.

---

Managed Secret Stores Across AWS, GCP, and Azure

The three major clouds all provide managed secret stores, but they are not interchangeable in the details that matter during incidents. AWS Secrets Manager is designed for application secrets that need versioning, IAM policy control, CloudTrail audit events, KMS-backed envelope encryption, and rotation workflows. AWS Systems Manager Parameter Store can also hold encrypted SecureString parameters, and its standard tier has no additional charge, but it is a configuration and parameter service first. Use Parameter Store for low-change configuration values and simple encrypted parameters; use Secrets Manager when rotation workflow, secret lifecycle metadata, or application-secret semantics are central to the design.

AWS Secrets Manager protects secret values with envelope encryption through AWS KMS. That distinction matters because the KMS key protects data keys, while Secrets Manager stores and serves the secret value through its own API. Automatic rotation commonly uses a Lambda function that performs staged steps such as creating a pending value, applying it to the backing service, testing it, and marking it current. The managed-service benefit is strong, but it introduces a dependency chain: the rotation Lambda needs network access to the database or service, permissions to update the target, and enough observability to troubleshoot failed rotation windows.

Google Secret Manager models a secret as metadata plus one or more versions, and each version stores the actual payload. That version model is useful during rollback because workloads can reference a specific version, use aliases, or consume the latest enabled version depending on the operational pattern. Google encrypts secrets in transit and at rest by default and supports customer-managed encryption keys through Cloud KMS for teams with stronger key-control requirements. For GKE, Workload Identity Federation lets Kubernetes service accounts call Google Cloud APIs without long-lived service account key files, which is the keyless path you want for ESO controllers or applications that call Secret Manager directly.

Azure Key Vault is broader than a password store. It manages secrets, cryptographic keys, and certificates, and it integrates with Microsoft Entra ID for authentication plus Azure RBAC or access policies for authorization. Soft-delete and purge protection are operationally important because accidental deletion of a vault, key, certificate, or secret should be recoverable instead of instantly permanent. For AKS, Microsoft Entra Workload ID uses projected service account tokens and OIDC federation so pods can access Azure resources such as Key Vault without embedding a client secret in the cluster.

The encryption backing is conceptually similar across clouds even though the products differ. AWS Secrets Manager uses AWS KMS keys and envelope encryption; Google Secret Manager encrypts at rest by default and can use Cloud KMS customer-managed encryption keys; Azure Key Vault stores vault data encrypted at rest using keys protected by HSM-backed systems. In all three clouds, customer-managed keys improve control and separation of duties, but they also add failure modes. If a KMS key is disabled, access policy is broken, or a region-specific key is unavailable, the secret store can become unreachable even though the secret itself still exists.

Cost at moderate scale is usually dominated by three knobs: number of stored objects or active versions, request volume, and optional premium features. AWS Secrets Manager charges per secret per month and per API calls, so a controller that polls thousands of secrets every few seconds can turn a simple platform pattern into measurable spend. Parameter Store standard parameters avoid additional storage charges within documented limits, while advanced parameters and higher-throughput usage add charges. Google Secret Manager bills active secret versions per location and access operations, so version sprawl and user-managed multi-location replication affect cost. Azure Key Vault pricing is operations-oriented for secrets and has separate considerations for keys, certificates, Premium tier, and Managed HSM usage.

At platform scale, the cost lens changes design choices. A direct SDK call from every pod on every request is a poor pattern because it increases latency, quota pressure, and billable secret access operations. A short ESO refresh interval across many namespaces may feel safer, but it can create unnecessary API traffic when secrets rotate monthly or quarterly. A better design caches within the application or syncs through an operator at a reasonable interval, then uses explicit rollout triggers for urgent rotation. The target is not the lowest possible bill; it is a controlled bill where spending tracks risk reduction instead of accidental polling.

---

External Secrets Operator (ESO): The Standard Approach

ESO is the most widely adopted solution for syncing secrets from cloud secret managers into Kubernetes Secrets. It runs as an operator in your cluster and periodically fetches secrets from external sources.

ESO fits GitOps because the manifest in Git describes where to fetch a secret and how to shape it, without storing the plaintext value. The source of truth stays in AWS Secrets Manager, Parameter Store, Google Secret Manager, Azure Key Vault, Vault, or another supported backend. The operator reconciles an ExternalSecret into a native Kubernetes Secret, which means existing Deployments, Helm charts, Ingress controllers, and applications can keep using familiar secretKeyRef and volume mounts. That compatibility is the reason ESO is the default answer for many platform teams.

The tradeoff is that ESO deliberately creates a Kubernetes Secret. That means the synced value lands in etcd, is governed by Kubernetes RBAC, and is visible to anyone who can read that Secret in the namespace. This is acceptable when etcd encryption, namespace isolation, and RBAC are correctly configured; it is not acceptable when a compliance requirement says the secret must never be persisted in the cluster datastore. ESO is a synchronization pattern, not a magic bypass around Kubernetes secret exposure.

Architecture

graph TD
    A[AWS Secrets Manager] --- B[External Secrets Operator]
    C[GCP Secret Manager] --- B
    B -- Creates/Updates --> D[K8s Secret managed by ESO]
    D -- Volume mount / env var --> E[Application Pod]

Pause and predict: Given this architecture, what’s a critical operational consideration for ESO concerning network connectivity and permissions? How would you secure the communication path between ESO and your cloud secret manager?

The critical consideration is that ESO becomes a privileged bridge between Kubernetes and the external store. If it cannot reach the provider endpoint, rotations stop propagating. If its identity is too broad, a compromise of the ESO controller can read secrets for many applications. Production designs therefore use private endpoints where practical, network policies around the controller namespace, provider-side IAM scoped to exact secret paths, and separate SecretStore resources for teams that should not share blast radius.

Installing ESO

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set installCRDs=true

ClusterSecretStore Configuration

A ClusterSecretStore defines how ESO authenticates with the external secret provider. It is cluster-scoped, meaning any namespace can use it.

Cluster-scoped stores are convenient for a central platform team, but they should not become a universal skeleton key. A ClusterSecretStore can be referenced across namespaces, so its provider credentials and access policy must be intentionally narrow or paired with admission policy that restricts who can reference it. A namespaced SecretStore is often better for tenant-owned applications because it keeps provider access, Kubernetes RBAC, and namespace ownership aligned. The decision is less about syntax and more about who owns the risk when a namespace is compromised.

# AWS Secrets Manager with IRSA
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
---
# GCP Secret Manager with Workload Identity
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: my-project
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production
          serviceAccountRef:
            name: gcp-secrets-sa
            namespace: external-secrets
---
# Azure Key Vault with Workload Identity
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: azure-key-vault
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-vault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: azure-secrets-sa
        namespace: external-secrets

The provider authentication examples show the keyless direction. On EKS, IRSA and EKS Pod Identity map a Kubernetes service account to AWS IAM permissions so the ESO controller can call Secrets Manager without a static access key stored in Kubernetes. On GKE, Workload Identity Federation maps Kubernetes service accounts into Google IAM principals or service-account impersonation flows. On AKS, Microsoft Entra Workload ID uses service account token projection and OIDC federation to let the controller call Key Vault. The shared principle is that the cluster holds a projected workload identity token, not a long-lived cloud credential.

This identity layer is also the right place to enforce separation between environments. A production ESO service account should not be able to read development secrets, and a development namespace should not be able to reference the production ClusterSecretStore. In AWS, that means IAM resource constraints and secret naming discipline. In GCP, that means IAM roles on specific Secret Manager resources or projects. In Azure, that means Key Vault access scoped through Azure RBAC or access policies. Kubernetes RBAC then limits which teams can create or modify ExternalSecret objects that reference those stores.

ExternalSecret: Syncing Individual Secrets

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    deletionPolicy: Retain
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
    - secretKey: host
      remoteRef:
        key: production/database
        property: host
    - secretKey: connection-string
      remoteRef:
        key: production/database
        property: connection_string

Stop and think: You have an existing application expecting secrets in a specific format, e.g., a single config.json file. How would you use ESO to fetch multiple individual secrets from AWS Secrets Manager and combine them into this single config.json within a Kubernetes Secret?

The refreshInterval is a design decision, not boilerplate. ESO’s periodic refresh means the operator updates the target Kubernetes Secret when the external value changes and the next reconciliation observes it. A five-minute interval is reasonable for many applications because most rotations are planned, but it is still a polling loop against a paid and quota-limited provider API. For emergency rotation, pair a sane interval with a manual refresh annotation, a rollout trigger, or an application reload controller rather than setting every ExternalSecret to an aggressive interval.

ExternalSecret: Templating

ESO can transform secret data using Go templates, which is useful when the external store and the consuming application use different shapes:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-url
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:5432/{{ .dbname }}?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
    - secretKey: host
      remoteRef:
        key: production/database
        property: host
    - secretKey: dbname
      remoteRef:
        key: production/database
        property: dbname

Templating is valuable because application configuration rarely matches provider storage exactly. A team may store username, password, host, port, and database name as separate JSON properties in AWS Secrets Manager, separate versions in Google Secret Manager, or separate Key Vault secrets in Azure. ESO can shape those inputs into a connection string, a .dockerconfigjson, or an application-specific config file. The risk is that templates can hide coupling; if a downstream app expects one generated string, every rotated field must remain compatible with that string format.

For multi-cloud platforms, normalize conventions before you normalize tools. Decide whether secret paths include environment, application, region, and purpose; decide whether JSON objects are allowed or whether each key is a separate secret; decide who owns labels and tags; and decide how stale versions are disabled. ESO can talk to all three clouds, but it cannot make inconsistent naming and ownership models safe. The cleanest clusters usually have boring conventions such as /prod/payments/api/database, projects/prod/secrets/payments-api-db, or https://platform-prod.vault.azure.net/secrets/payments-api-db, mapped into a predictable Kubernetes Secret name.

---

Secrets Store CSI Driver

The Secrets Store CSI Driver mounts secrets directly from a vault as files in a pod, bypassing Kubernetes Secrets entirely. The secret exists only in the pod’s filesystem and the vault — in the standard CSI-only pattern, it does not land in etcd.

CSI is the right mental model when the pod needs a file, not a Kubernetes API object. On Linux, the driver mounts secret content through an in-memory filesystem path and provider plugins retrieve values from AWS, Azure, GCP, Vault, or another supported backend. The pod references a SecretProviderClass, and the driver fetches the configured objects when the pod starts. If you enable sync as Kubernetes Secret, you regain compatibility with secretKeyRef, but you also reintroduce etcd persistence and the RBAC exposure that CSI-only designs were trying to avoid.

Architecture Difference from ESO

graph TD
    Vault_CSI[Vault] --> CSI_Driver[CSI Driver]
    CSI_Driver --> Pod_Filesystem[Pod Filesystem]
    Pod_Filesystem -- Mounted as files --> Pod_CSI[Application Pod]

Pause and predict: If a secret never lands in etcd when using the CSI Driver, what are the primary security advantages and potential operational challenges compared to ESO? Consider auditability and secret rotation.

The security advantage is smaller blast radius in the Kubernetes API. An attacker with get secrets permission cannot read a CSI-only value because there is no native Secret object to fetch. The operational challenge is that every pod mount becomes a provider access path, every node needs the driver and provider components healthy, and applications must read files rather than environment variables if you want rotation without restart. CSI reduces one exposure path, but it increases dependence on node-level plumbing and application reload behavior.

Installing the CSI Driver

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set syncSecret.enabled=true

# Install AWS provider
k apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: db-secrets
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "production/database"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password
  secretObjects:
    - secretName: db-credentials-synced
      type: Opaque
      data:
        - objectName: db-username
          key: username
        - objectName: db-password
          key: password

Pod Using CSI Mounted Secrets

apiVersion: v1
kind: Pod
metadata:
  name: api-server
  namespace: production
spec:
  serviceAccountName: app-sa
  containers:
    - name: api
      image: mycompany/api-server:3.0.0
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials-synced
              key: username
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: db-secrets

Stop and think: Your security team mandates that secrets should never be exposed as environment variables, only mounted as files. However, an older legacy application only reads secrets from environment variables. How might you adapt the CSI Driver approach to meet both requirements, or what alternative would you consider?

ESO vs CSI Driver: When to Use Each

Factor	ESO	Secrets Store CSI
Secret in etcd	Yes (K8s Secret)	Optional (only if syncSecret enabled)
Multiple pods share secret	Yes (via K8s Secret)	Each pod mounts independently
Secret refresh	Automatic (refreshInterval)	Requires pod restart or rotation
Template/transform	Yes (Go templates)	Limited
Git-friendly	ExternalSecret in Git (no plaintext)	SecretProviderClass in Git (no plaintext)
Vault-native rotation	Works with any rotation	Better with CSI rotation reconciler
Best for	Most use cases	Zero-trust (no secrets in etcd)

For most teams, ESO is the better choice. It is simpler, more flexible, and works well with GitOps. Use Secrets Store CSI when your security requirements prohibit secrets from existing in etcd at all.

That recommendation assumes the application can tolerate a Kubernetes Secret existing as a local delivery object. If the workload is an Ingress controller that needs TLS certificates, a database client that can read credentials from files, or a security-sensitive service in a regulated namespace, CSI may be a better fit. If the workload is a common Helm chart that expects secretKeyRef, an application that needs templated configuration, or a platform with many teams and GitOps workflows, ESO is usually easier to operate consistently. The best platform often supports both, with clear policy for when each is allowed.

---

Rotation Propagation and Application Reloads

Rotation has three separate clocks. The first clock is the cloud or Vault source of truth: AWS Secrets Manager may execute a Lambda-backed rotation schedule, Google Secret Manager may create a new enabled version and publish rotation notifications, Azure Key Vault may store a new secret version or renew a certificate, and Vault may issue a new dynamic lease. The second clock is Kubernetes delivery: ESO reconciles according to refresh policy and interval, while Secrets Store CSI can periodically republish mounted contents when rotation is enabled. The third clock is the application: a process might read a file on every request, cache credentials in a connection pool, or read an environment variable once at startup.

Most rotation outages happen because teams update the first clock and forget the third. A rotated database password in AWS Secrets Manager does not update existing PostgreSQL connections. A new Google Secret Manager version does not make a Java process rebuild its connection pool. A Key Vault certificate renewal does not guarantee an Ingress controller has reloaded the file. A Vault dynamic credential may expire on schedule while the application still tries to reuse an old connection. The correct design includes a reload mechanism, not just a rotation mechanism.

For ESO, propagation starts when the operator notices a changed remote value and updates the Kubernetes Secret. If the consuming pod mounts that Secret as a volume, Kubernetes updates the projected volume contents after the kubelet observes the change, and the application can watch the file path. If the consuming pod reads the Secret as environment variables, the process will not see the new value until the pod restarts. Teams commonly add a controller such as a reloader to trigger rolling restarts when selected Secrets change, but that should be a conscious availability decision because every rotation becomes a deployment event.

For Secrets Store CSI Driver, mounted content can be updated when automatic rotation is enabled and the provider supports it. The driver documentation distinguishes file mounts, synced Kubernetes Secrets, and environment-variable consumption. File-reading applications still need to watch the mounted file or reload periodically. Environment-variable consumers still need restart even if CSI syncs a Kubernetes Secret. This is why CSI does not automatically solve the legacy-app problem; it gives you a better delivery path if the app can consume files or if you can wrap the app with a reloadable entrypoint.

For Vault dynamic secrets, the lifecycle is lease-based rather than version-based. Vault can generate a unique database username and password for a role, attach a TTL, renew the lease, and revoke the credential when the lease expires or is explicitly revoked. That gives excellent blast-radius reduction, but it also means applications must renew leases or request new credentials before expiration. A short TTL is not automatically safer if the application cannot refresh connections gracefully; it can become an availability risk that looks like random database authentication failure.

The pattern that works across AWS, GCP, Azure, and Vault is two-phase rotation. First, publish the new credential while the old one still works, then update or restart consumers in controlled waves, then revoke the old credential after telemetry shows the new one is in use. Some managed databases and rotation templates support alternating-user rotation so the old and new database users overlap safely. Where that is not available, you need a maintenance window, connection draining, or application code that can retry with refreshed credentials. Secret rotation is reliability engineering wearing a security badge.

---

Dynamic Secrets with HashiCorp Vault

Dynamic secrets are generated on-demand and automatically expire. Instead of a static database password that lives forever, Vault creates a temporary database user with a 1-hour TTL every time a pod requests credentials.

Vault is different from the cloud-native managers because it can be both a secret store and a credential broker. A static secret store gives you a value that already exists; a dynamic secrets engine creates a value at request time, records a lease, and cleans it up later. For database credentials, that means Vault can create a real database user with role-specific privileges, return the username and password to the workload, and revoke the user when the lease expires. This is powerful because audit trails can map suspicious database activity to a specific lease and workload instead of one shared application password.

Dynamic Secret Lifecycle

sequenceDiagram
    participant P as Pod
    participant V as Vault
    participant D as Database

    P->>V: Request credentials
    V->>D: Create temporary user (TTL: 1h)
    D-->>V: Temporary user created
    V-->>P: Credentials (username, password)
    P->>D: Use credentials (for 1 hour)
    loop After TTL expires
        V->>D: Revoke user
        D-->>V: User revoked
        P->>V: Request new credentials (or renew lease)
    end

Pause and predict: What potential issues could arise if a pod crashes and restarts frequently when using Vault’s dynamic secrets with a very short TTL (e.g., 5 minutes)? How might you design your application or Vault policy to handle this gracefully?

Vault Setup for Database Dynamic Secrets

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/production-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-readonly,app-readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@app-postgres.abc123.us-east-1.rds.amazonaws.com:5432/appdb?sslmode=require" \
  username="vault_admin" \
  password="vault-admin-password"

# Create a role that generates read-only credentials
vault write database/roles/app-readonly \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Create a readwrite role
vault write database/roles/app-readwrite \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="4h"

Vault Agent Sidecar for Dynamic Secrets

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-server"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/app-readonly"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/app-readonly" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end -}}
    spec:
      serviceAccountName: api-server
      containers:
        - name: api
          image: mycompany/api-server:3.0.0
          command:
            - /bin/sh
            - -c
            - "source /vault/secrets/db-creds && ./start-server"

Stop and think: You need to provide different database credentials (read-only vs. read-write) to two different containers within the same pod based on their function. How would you modify the Vault Agent annotations and container configuration to achieve this isolation?

Cloud KMS Auto-Unseal for Vault HA

A production Vault cluster should not depend on operators manually entering Shamir key shares every time a pod restarts, a node drains, or an HA standby takes over. Configure an auto-unseal seal stanza such as seal "awskms", seal "gcpckms", or seal "azurekeyvault" so Vault can ask the cloud KMS or Key Vault service to decrypt its root key material during startup. This keeps the unseal authority outside Vault storage while removing a human from the recovery path; in Kubernetes, the Vault server pod’s workload identity must be authorized to call the KMS key with narrowly scoped permissions, or the pod may be healthy while Vault remains sealed.

Vault Secrets Operator for Native Secret Sync

Vault Secrets Operator is the CRD-based Vault integration pattern for applications that expect native Kubernetes Secrets. Platform teams define connection and authentication resources such as VaultConnection and VaultAuth; application teams then use VaultStaticSecret for KV paths or VaultDynamicSecret for generated credentials, and the VSO controller syncs the result into a Kubernetes Secret. Contrast that with the Vault Agent Sidecar pattern above: the sidecar injects rendered files into the pod filesystem, while VSO syncs Vault data into a native Secret that existing secretKeyRef, volume, and Helm chart workflows can consume.

Vault vs Cloud Secret Managers

Feature	HashiCorp Vault	AWS Secrets Manager	GCP Secret Manager	Azure Key Vault
Dynamic secrets	Yes (database, AWS, PKI)	No dynamic leases; static secret values	No	No
Secret rotation	Built-in (TTL + revocation)	Lambda-backed or managed rotation on a configurable schedule	Rotation with Cloud Functions	Auto-rotation (certificates)
PKI/certificates	Yes (built-in CA)	Via ACM (separate service)	Via CAS	Via Key Vault certificates
Multi-cloud	Yes	AWS only	GCP only	Azure only
Self-hosted	Yes (or HCP Vault)	N/A (managed)	N/A (managed)	N/A (managed)
Complexity	High (operate Vault cluster)	Low	Low	Medium
Pricing model	OSS or managed Vault subscription plus operations	Per stored secret and API calls	Per active version/location and access operations	Per operations, plus key/certificate/HSM dimensions

Recommendation: choose the simplest managed store that satisfies your lifecycle requirements, but move to Vault when dynamic credentials or cross-cloud control outweigh operational complexity.

• Single cloud, simple needs: Use the cloud-native secret manager with ESO
• Multi-cloud or dynamic secrets needed: Use Vault
• Small team, few secrets: Cloud-native is easiest
• Enterprise with strict compliance: Vault gives the most control

Cloud-native secret managers usually win on operational simplicity because the provider runs the control plane, integrates with IAM, and exposes audit events in the same cloud account. Vault wins when the requirement is dynamic credentials, consistent multi-cloud policy, private PKI, or an abstraction layer that does not privilege one hyperscaler. The cost tradeoff is not just subscription or per-secret charges; it includes the human cost of operating Vault HA, unseal strategy, backup, disaster recovery, policy authoring, and plugin lifecycle. A self-hosted Vault outage can become an application outage if workloads depend on dynamic credentials and cannot renew leases.

---

Sealed Secrets: GitOps-Safe Encryption

Sealed Secrets encrypts secrets so they can be safely stored in Git. Only the Sealed Secrets controller in the cluster can decrypt them.

How It Works

sequenceDiagram
    participant D as Developer
    participant G as Git Repo
    participant C as Cluster
    D->>G: 1. kubeseal encrypt (SealedSecret YAML)
    G->>C: 2. GitOps sync
    Note over C: 3. Controller decrypts
    Note over C: 4. Creates K8s Secret

Installing Sealed Secrets

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Install kubeseal CLI
brew install kubeseal

Creating a Sealed Secret

# Create a regular secret (do NOT apply it)
k create secret generic db-credentials \
  --from-literal=username=appadmin \
  --from-literal=password=super-secret-password \
  --dry-run=client -o yaml > /tmp/secret.yaml

# Seal it (encrypts with the cluster's public key)
kubeseal --format yaml < /tmp/secret.yaml > sealed-secret.yaml

# The sealed version is safe to commit to Git
cat sealed-secret.yaml

# This is safe to commit to Git
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    username: AgB7w2K...long-encrypted-string...==
    password: AgCx9f3...long-encrypted-string...==
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

Sealed Secrets Limitations

Limitation	Impact	Mitigation
Cluster-specific encryption	Sealed Secret from cluster A cannot be decrypted in cluster B	Export and share the sealing key, or use SOPS instead
No rotation mechanism	Secret value stays the same until manually re-sealed	Combine with ESO for rotation
Key management	Losing the sealing key means losing all sealed secrets	Back up the sealing key to a secure location

---

SOPS: Mozilla’s Alternative to Sealed Secrets

SOPS (Secrets OPerationS) encrypts YAML/JSON files using cloud KMS keys, PGP, or age. Unlike Sealed Secrets, SOPS is not Kubernetes-specific — it encrypts files that can be decrypted by anyone with the KMS key.

SOPS with AWS KMS

# Install SOPS
brew install sops

# Create a .sops.yaml configuration
cat > .sops.yaml << 'EOF'
creation_rules:
  - path_regex: .*secrets.*\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/mrk-abc123
  - path_regex: .*secrets.*\.yaml$
    gcp_kms: projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key
EOF

# Create a secret file
cat > secrets.yaml << 'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
stringData:
  username: appadmin
  password: super-secret-password
EOF

# Encrypt it
sops --encrypt secrets.yaml > secrets.enc.yaml

# The encrypted file can be committed to Git
# Argo CD / Flux can decrypt it using SOPS integration

SOPS vs Sealed Secrets

Feature	SOPS	Sealed Secrets
Encryption backend	KMS, PGP, age	Cluster-specific RSA key
Multi-cluster	Same KMS key works everywhere	Different key per cluster
GitOps integration	Argo CD SOPS plugin, Flux SOPS	Native Kubernetes controller
Edit encrypted files	sops secrets.enc.yaml opens in editor	Must re-seal entire secret
Non-K8s files	Encrypts any YAML/JSON	Kubernetes Secrets only

---

Putting It All Together: A Complete Secrets Architecture

graph TD
    Devs[Developers <br/> kubeseal/sops] -- Encrypted secrets in Git --> GitOps[GitOps Argo CD <br/> - SealedSecrets <br/> - SOPS decrypt]
    GitOps -- Sync to cluster --> ESO[ESO <br/> dynamic refresh]
    ESO -- Syncs from --> CloudMgrs[AWS Secrets Mgr <br/> GCP Secret Mgr <br/> Azure Key Vault]
    ESO -- Creates K8s Secrets --> Pods[Application Pods <br/> - env vars <br/> - volume mounts]
    Pods -- Requests from --> Vault[Vault dynamic <br/> - DB creds 1h <br/> - PKI certs 24h]

Layer	Tool	Purpose
Git encryption	Sealed Secrets or SOPS	Safe to commit secrets to Git
External sync	ESO	Sync cloud secrets to K8s Secrets
Dynamic secrets	Vault	Short-lived credentials with auto-revocation
Runtime mount	Secrets Store CSI	Mount directly, bypassing etcd
Rotation trigger	Reloader	Restart pods when secrets change

The complete architecture is intentionally layered because each layer handles a different failure mode. Git encryption tools protect repository workflows but do not rotate live application credentials. ESO gives existing Kubernetes-native workloads a familiar Secret object but does not prevent etcd exposure if the cluster is misconfigured. CSI avoids native Secret persistence in the CSI-only path but requires file-based consumption and node driver health. Vault dynamic secrets reduce credential lifetime but add lease-management complexity. Cloud secret managers centralize storage and audit, but they do not automatically teach applications how to reload.

In a multi-cloud platform, the portable part is the control pattern, not the provider API. EKS workloads should use IRSA or EKS Pod Identity to avoid AWS access keys. GKE workloads should use Workload Identity Federation rather than service account key files. AKS workloads should use Microsoft Entra Workload ID instead of client secrets stored in Kubernetes. ESO and CSI then become delivery mechanisms that use those identities. KEDA scalers, database operators, Ingress controllers, and application Deployments should consume secrets through the same governed path instead of each team inventing a separate credential mount.

The audit design should be explicit. Kubernetes audit logs should show who changed ExternalSecret, SecretStore, ClusterSecretStore, SecretProviderClass, and native Secret objects. Cloud audit logs should show which workload identity read which secret, and whether access came from the expected cluster identity. Vault audit devices should record lease issuance and revocation without exposing plaintext. Alerting should focus on abnormal access patterns: secrets read from a new namespace, a sudden increase in GetSecretValue calls, Key Vault operations from an unexpected identity, or Secret Manager access outside the deployment window.

Cost and reliability are also part of the architecture. Use provider API calls to fetch or sync secrets at boundaries, not inside high-volume request paths. Keep ESO refresh intervals reasonable, use labels or annotations to trigger urgent syncs, and avoid one global ClusterSecretStore that every namespace can reference. Keep versions under control by destroying or disabling old GCP versions when rollback windows close, deleting unused AWS secrets rather than storing zombie values forever, and cleaning up Key Vault objects after retention and compliance rules allow. A secrets platform that nobody can afford to operate will eventually be bypassed.

---

Patterns & Anti-Patterns

Proven Patterns

Pattern 1: external store as source of truth, Kubernetes as delivery plane. Store canonical values in AWS Secrets Manager, Parameter Store, Google Secret Manager, Azure Key Vault, or Vault, then use ESO or CSI to deliver them to workloads. This scales because platform teams can centralize IAM, audit, rotation, and retention while application teams keep declarative manifests in Git. It works best when secret names, labels, paths, and ownership rules are standardized before hundreds of teams create their own conventions.

Pattern 2: keyless controller access through workload identity. Run ESO, CSI providers, and applications under Kubernetes service accounts mapped to cloud identities rather than storing static cloud credentials in Kubernetes. On EKS, choose IRSA or EKS Pod Identity according to the cluster and organizational model. On GKE, use Workload Identity Federation. On AKS, use Microsoft Entra Workload ID. This pattern scales because credential issuance becomes short-lived and auditable, and the compromise of one namespace does not automatically expose a reusable cloud access key.

Pattern 3: rotation with reload choreography. Treat every rotation as a rollout workflow that includes source update, Kubernetes propagation, application reload, telemetry confirmation, and old-secret revocation. ESO refresh or CSI rotation handles only the middle of that chain. Mature teams write runbooks that say whether an app watches files, restarts on Secret changes, refreshes SDK clients, drains database pools, or uses Vault lease renewal. This pattern scales because emergency rotation becomes a rehearsed operational motion instead of a Slack scramble.

Pattern 4: namespace-scoped blast-radius boundaries. Use separate cloud secret paths, Key Vaults, IAM policies, SecretStore resources, and Kubernetes service accounts for environments and tenants that should not share risk. A production payments namespace should not reference the same store identity as a development analytics namespace. At small scale this feels repetitive, but it pays back when one service account is misconfigured or one namespace is compromised. The blast radius is bounded by design rather than by hope.

Pattern 5: file mounts for high-sensitivity values. Prefer mounted secret files for applications that can reload configuration, especially when credentials are rotated regularly or when policy forbids environment-variable exposure. Files can be watched, reopened, and replaced without embedding values in process environment state. CSI-only mounts can also avoid native Kubernetes Secret persistence. This pattern scales when application frameworks agree on predictable file paths and reload signals.

Anti-Patterns

Anti-pattern 1: secrets in environment variables forever. Teams fall into this because environment variables are easy, Twelve-Factor-style examples are everywhere, and many legacy applications only support them. The failure mode is stale values: a rotated Kubernetes Secret does not update a running process environment, so the application keeps using the old password until restart. The better alternative is file-based consumption with reload, native SDK retrieval with caching, or a controlled reloader that restarts pods when selected Secrets change.

Anti-pattern 2: one giant shared secret. A team stores every credential for an application in one JSON blob because it is convenient to fetch and template. The failure mode is excessive blast radius: rotating one field forces every consumer to reload, access to one key implies access to all keys, and audit logs cannot easily distinguish which credential was needed. The better alternative is to group only values with the same owner, rotation cadence, and access policy, then template them at the edge when the application truly needs a combined file.

Anti-pattern 3: static cloud credentials for operators. A platform team creates an AWS access key, GCP service account key, or Azure client secret and stores it in a Kubernetes Secret so ESO or a CSI provider can authenticate. This feels simple during a proof of concept, but it creates a high-value static credential inside the very cluster the tool is meant to protect. The better alternative is workload identity: IRSA or EKS Pod Identity, GKE Workload Identity Federation, and Microsoft Entra Workload ID.

Anti-pattern 4: no etcd encryption because secrets are external. Teams assume ESO means “secrets live outside the cluster” and forget that ESO creates native Kubernetes Secrets by design. The failure mode is plaintext or weakly protected secret data in etcd, plus broad RBAC that lets too many subjects read those values. The better alternative is to enable KMS-backed encryption at rest, restrict get, list, and watch on Secrets, and use CSI-only mounts where policy forbids etcd persistence.

Anti-pattern 5: aggressive polling as a substitute for rotation design. A team sets ESO refresh intervals to 15 or 30 seconds because faster feels safer. The failure mode is provider API cost, quota pressure, noisy reconciliation, and no guarantee that the application process actually reloads. The better alternative is a moderate refresh interval, explicit manual refresh for urgent changes, and application-level reload or restart automation.

Anti-pattern 6: Git encryption as the whole secrets platform. SOPS and Sealed Secrets protect manifests at rest in Git, but they do not provide cloud audit logs, dynamic leases, provider-side rotation, or runtime access decisions. Teams fall into this pattern because encrypted YAML is easy to add to an existing GitOps repository. The better alternative is to use Git encryption for bootstrap or narrow exceptions, then move long-lived application secrets into managed stores or Vault with ESO/CSI delivery.

---

Decision Framework

Use this matrix when choosing the source of truth and the Kubernetes delivery path. The most common mistake is asking “which tool is best” without separating storage, identity, delivery, and reload. A team can choose AWS Secrets Manager as the source, EKS Pod Identity as authentication, ESO as delivery, and a reloader as propagation; another can choose Azure Key Vault, Entra Workload ID, CSI file mounts, and application file watches. The right answer is a composition.

Decision	Prefer This	When It Fits	Tradeoff
AWS static application secret with rotation	AWS Secrets Manager	You need rotation workflow, CloudTrail audit, KMS-backed encryption, and application-secret metadata	Higher per-secret/API cost than simple parameters
AWS simple encrypted parameter	SSM Parameter Store SecureString	You need low-change config or simple secrets and can accept Parameter Store semantics	Standard tier lacks automatic rotation workflow and advanced features
GCP application secret	Google Secret Manager	You want versioned secrets, IAM control, replication choices, and GKE Workload Identity Federation	Active versions and access operations affect cost
Azure application secret/cert/key	Azure Key Vault	You need secrets plus keys or certificates, Microsoft Entra auth, soft-delete, and Azure-native audit	Operations pricing and vault access model require planning
Multi-cloud dynamic credentials	HashiCorp Vault	You need database dynamic users, leases, revocation, PKI, or consistent multi-cloud policy	Higher operational complexity and availability responsibility
Kubernetes delivery for most apps	ESO	Apps expect native Kubernetes Secrets, GitOps should hold only references, and templating is useful	Secret data lands in etcd and must be protected by RBAC/encryption
Kubernetes delivery for strict no-etcd policy	Secrets Store CSI Driver	Apps can consume files and policy forbids native Secret persistence	Node driver/provider health and app reload behavior matter more
Git-stored encrypted bootstrap values	SOPS or Sealed Secrets	You need to keep limited encrypted values in Git for bootstrap or disconnected workflows	Does not replace runtime rotation, audit, or dynamic credentials

flowchart TD
    A[Need to deliver a secret to a Kubernetes workload] --> B{Is the value dynamic or short-lived?}
    B -- Yes --> C[Use Vault dynamic secrets or cloud-native temporary identity]
    B -- No --> D{Single cloud source of truth?}
    D -- AWS --> E{Need native rotation workflow?}
    E -- Yes --> F[AWS Secrets Manager]
    E -- No --> G[SSM Parameter Store SecureString]
    D -- GCP --> H[Google Secret Manager]
    D -- Azure --> I[Azure Key Vault]
    D -- Multi-cloud --> J[Vault or per-cloud stores with consistent policy]
    F --> K{Can the app consume mounted files?}
    G --> K
    H --> K
    I --> K
    J --> K
    C --> K
    K -- Yes, and no-etcd policy matters --> L[Secrets Store CSI Driver]
    K -- No, app expects K8s Secret/env/template --> M[External Secrets Operator]
    L --> N[Design file watch or restart path]
    M --> O[Enable etcd encryption, RBAC, and reload automation]

The decision framework should be revisited whenever the consumer changes. A database password for a legacy app may start with ESO and a rolling restart because the app only reads environment variables. The same platform may use CSI-only mounts for TLS private keys on an Ingress controller, direct SDK calls for a service that can cache Google Secret Manager versions, and Vault dynamic credentials for administrative database jobs. Consistency matters, but forcing every workload through one delivery mechanism usually creates exceptions that are less safe than a documented two-tool policy.

---

Did You Know?

1. Kubernetes warns that Secrets are stored unencrypted in etcd by default unless encryption at rest is explicitly configured. That is why a managed cloud cluster’s convenience does not remove the need to understand the control-plane encryption model and RBAC boundaries.

2. Secrets Store CSI Driver rotation and ESO refresh solve different layers of the same problem. ESO updates a Kubernetes Secret on reconciliation, while CSI rotation updates mounted content and optionally synced Secrets, but an environment-variable consumer still needs a pod restart.

3. Google Secret Manager bills active versions per location, so forgotten old versions are not just operational clutter. Version retention should match rollback needs, and user-managed replication should be chosen deliberately because each selected location changes the cost model.

4. Azure Key Vault soft-delete gives teams a recovery window for deleted vault objects, but recovery is not the same as a complete restore of every dependent integration. Treat vault deletion and purge permissions as production-critical controls rather than ordinary cleanup permissions.

---

Common Mistakes

Mistake	Why It Happens	How to Fix It
Treating base64 as encryption	”The secret is encoded, so it is safe”	base64 is encoding, not encryption; anyone can decode it
Storing secrets in ConfigMaps	Developer confusion between ConfigMap and Secret	Use Secrets (they get masked in logs and have RBAC separation)
Not enabling etcd encryption at rest	Not configured by default	Enable EncryptionConfiguration with a KMS v2 provider; use secretbox only when local key storage is the accepted fallback
Using the same secret across all environments	”Simpler to manage one secret”	Separate secrets per environment; use ESO with environment-specific paths
Not monitoring secret access	”We have RBAC, that is enough”	Enable Kubernetes audit logging; alert on secret read events from unexpected sources
Committing plaintext secrets to Git then deleting them	”I removed it, so it is gone”	Git history preserves everything; rotate the secret immediately, use git-filter-repo to purge
Running Vault without HA	”It is just a dev cluster”	Vault is a critical dependency; in production, run HA mode (typically 3+ replicas)
Setting ESO refreshInterval too low	”Faster sync is better”	Below 1 minute creates unnecessary API calls and costs; 5-15 minutes is usually fine

---

Quiz

1. You are auditing a newly provisioned Kubernetes cluster and notice that the team is storing database passwords in standard Kubernetes Secret objects. The lead developer argues this is safe because the values are unreadable when viewed in the manifest. Why is this assumption dangerous, and what minimal configuration changes must you enforce to secure these secrets?

The developer’s assumption is dangerous because Kubernetes Secrets are merely base64-encoded, not encrypted, meaning anyone with kubectl get secret permissions can instantly decode them. Furthermore, these secrets are stored in plaintext within the etcd database by default. To secure these secrets minimally, enable etcd encryption at rest with an EncryptionConfiguration, preferably using a KMS v2 provider so key-encryption-key control stays outside the API server host; secretbox is the local-key fallback when a KMS plugin is not available. You must also strictly restrict RBAC permissions to ensure only necessary ServiceAccounts and users can read the secrets, and enable Kubernetes API audit logging to track access. Finally, implementing an external secrets manager like ESO or the CSI driver means the true source of the secret remains external, even though ESO still syncs a Kubernetes Secret into etcd.

2. Your platform engineering team needs to integrate an external cloud vault with your Kubernetes cluster. One engineer suggests using the External Secrets Operator (ESO), while another insists on the Secrets Store CSI Driver to satisfy a strict "zero-trust" compliance requirement. What fundamental architectural difference between these two tools justifies the CSI Driver for zero-trust environments?

The fundamental architectural difference lies in how the secret data is surfaced to the application pod. The External Secrets Operator (ESO) fetches the secret from the external vault and creates a standard Kubernetes Secret object stored in the etcd database, which is then mounted or read by the pod. In contrast, the Secrets Store CSI Driver mounts the secret directly from the external vault into the pod’s ephemeral filesystem, entirely bypassing the creation of a Kubernetes Secret. This better aligns with a strict zero-trust requirement because the secret does not typically land in the cluster’s etcd datastore, significantly reducing the attack surface and the risk of etcd compromise exposing the credentials.

3. A recent security breach occurred when a contractor's laptop was stolen, exposing a static database password that had been valid for 11 months. You are tasked with implementing a solution using HashiCorp Vault. How does Vault's dynamic secrets feature prevent this specific type of breach, and what happens automatically when the time-to-live (TTL) expires?

Vault’s dynamic secrets feature prevents this type of breach by generating temporary, on-demand credentials rather than relying on long-lived static passwords. When an application requests database access, Vault dynamically creates a unique database user with a strict time-to-live (TTL), such as one hour. If a laptop containing these credentials is stolen, the blast radius is severely limited because the credentials will expire shortly anyway. When the TTL expires, Vault automatically reaches out to the database and revokes the user, ensuring the credential is mathematically dead without requiring any human intervention to rotate it.

4. You are designing a GitOps pipeline for a multi-cloud environment spanning EKS, GKE, and AKS. You need to store encrypted secrets in a single Git repository and sync them across all clusters. A colleague suggests using Sealed Secrets, but you propose SOPS instead. Why is SOPS the better architectural choice for this specific multi-cluster scenario?

SOPS is the better architectural choice for a multi-cloud, multi-cluster environment because it uses external Key Management Services (KMS) like AWS KMS, GCP KMS, or Azure Key Vault to encrypt files. This means a single encrypted file in Git can be decrypted by any cluster that has been granted access to the centralized KMS key. Sealed Secrets, on the other hand, relies on a cluster-specific RSA key pair generated by its internal controller. If you used Sealed Secrets, you would either have to encrypt the secret multiple times (once for each cluster’s public key) or manually export and share the private sealing key across all clusters, which defeats its operational simplicity.

5. A junior operator configures the External Secrets Operator (ESO) to sync credentials from AWS Secrets Manager with a `refreshInterval` of 30 seconds, arguing that faster synchronization improves security. Within a few hours, the cluster begins experiencing intermittent secret syncing failures and unexpected cloud billing charges. What is the root cause of this issue, and why is a longer interval recommended?

The root cause of the syncing failures and billing charges is API rate limiting and per-request costs imposed by the cloud provider. At a 30-second interval, ESO continuously polls the AWS Secrets Manager API, generating thousands of unnecessary requests per hour which can quickly hit service quotas and incur significant usage fees. A longer interval of 5 to 15 minutes is recommended because secret rotation is typically a planned, infrequent operational event rather than an emergency. If immediate propagation is truly required after a rotation, you should implement a push-based notification system, such as a CloudWatch Event triggering a webhook, rather than relying on aggressive polling.

6. During a code review, you notice a developer accidentally committed an AWS access key in a `.env` file. Recognizing the mistake, the developer immediately creates a new commit that deletes the file and pushes the change to the central repository, claiming the issue is resolved. Is the secret now safe, and what mandatory incident response steps must you take?

No, the secret is absolutely not safe because Git is a version control system designed to preserve the complete history of every file change. Even though the .env file was deleted in the latest commit, the AWS access key remains fully accessible in the repository’s history and can be easily extracted by attackers or automated scanning tools. To respond to this incident, you must immediately assume the key is compromised and rotate it within AWS IAM to invalidate the exposed credential. Afterward, you must use tools like git-filter-repo or BFG Repo Cleaner to permanently purge the secret from the entire Git commit history, and ensure all developers force-pull the cleaned repository.

---

Hands-On Exercise: Multi-Layer Secrets Management

Setup

# Create kind cluster
kind create cluster --name secrets-lab

# This module uses the standard Kubernetes shorthand in runnable blocks.
alias k=kubectl

# Install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set installCRDs=true
k wait --for=condition=ready pod -l app.kubernetes.io/name=external-secrets \
  --namespace external-secrets --timeout=120s

# Install Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system
k wait --for=condition=ready pod -l app.kubernetes.io/name=sealed-secrets \
  --namespace kube-system --timeout=120s

Task 1: Create and Seal a Secret

Use kubeseal to encrypt a secret that is safe to store in Git, then verify that the cluster controller can decrypt it back into a native Kubernetes Secret.

Solution

# Install kubeseal CLI if not present
# brew install kubeseal  # or download from GitHub releases

# Create a secret manifest (NOT applied to cluster)
k create secret generic app-secrets \
  --namespace default \
  --from-literal=api-key=sk-live-abc123def456 \
  --from-literal=webhook-secret=whsec-xyz789 \
  --dry-run=client -o yaml > /tmp/plain-secret.yaml

# Seal the secret
kubeseal --format yaml \
  --controller-name sealed-secrets \
  --controller-namespace kube-system \
  < /tmp/plain-secret.yaml > /tmp/sealed-secret.yaml

# Verify the sealed version does not contain plaintext
echo "=== Sealed Secret (safe to commit) ==="
cat /tmp/sealed-secret.yaml

# Apply the sealed secret
k apply -f /tmp/sealed-secret.yaml

# Verify the controller created the K8s Secret
sleep 5
k get secret app-secrets
k get secret app-secrets -o jsonpath='{.data.api-key}' | base64 -d
echo ""

Task 2: Set Up a Fake Secret Store with ESO

Since we do not have a real cloud provider in this lab, use ESO’s Fake provider to demonstrate the reconciliation workflow without introducing static cloud credentials.

Solution

# Fake SecretStore (for lab only -- uses in-cluster data)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: fake-store
  namespace: default
spec:
  provider:
    fake:
      data:
        - key: "/production/database"
          value: '{"username":"app_user","password":"dynamic-pass-892","host":"db.example.com","port":"5432"}'
        - key: "/production/redis"
          value: '{"host":"redis.example.com","port":"6379","auth_token":"redis-token-456"}'
---
# ExternalSecret that syncs from the fake store
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-creds
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: fake-store
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: /production/database
        property: username
    - secretKey: password
      remoteRef:
        key: /production/database
        property: password
    - secretKey: host
      remoteRef:
        key: /production/database
        property: host

k apply -f /tmp/eso-fake.yaml

# Wait for sync
sleep 10

# Verify ESO created the secret
k get externalsecret database-creds
k get secret db-credentials
k get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
echo ""

Task 3: Use ESO Templates to Generate a Connection String

Create an ExternalSecret that templates multiple fields into a single connection string, then inspect the generated Kubernetes Secret to confirm the expected shape.

Solution

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: fake-store
    kind: SecretStore
  target:
    name: database-url
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/appdb?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: /production/database
        property: username
    - secretKey: password
      remoteRef:
        key: /production/database
        property: password
    - secretKey: host
      remoteRef:
        key: /production/database
        property: host
    - secretKey: port
      remoteRef:
        key: /production/database
        property: port

k apply -f /tmp/eso-template.yaml
sleep 10

k get secret database-url -o jsonpath='{.data.DATABASE_URL}' | base64 -d
echo ""
# Should output: postgresql://app_user:dynamic-pass-892@db.example.com:5432/appdb?sslmode=require

Task 4: Deploy a Pod That Uses the Synced Secret

Deploy a pod that reads the ESO-managed secret as an environment variable, then consider why this convenient pattern still needs restart-based rotation handling.

Solution

apiVersion: v1
kind: Pod
metadata:
  name: secret-consumer
  namespace: default
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: busybox:1.36
      command:
        - /bin/sh
        - -c
        - |
          echo "=== Secret Consumer ==="
          echo "DB Username: $DB_USERNAME"
          echo "DB Host: $DB_HOST"
          echo "DB Password length: $(echo -n $DB_PASSWORD | wc -c) characters"
          echo "Connection String: $DATABASE_URL"
          echo "=== Sealed Secret ==="
          echo "API Key: $API_KEY"
          echo "=== Done ==="
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: host
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-url
              key: DATABASE_URL
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: api-key

k apply -f /tmp/secret-consumer.yaml
for i in {1..30}; do
  k get pod/secret-consumer >/dev/null 2>&1 && break
  sleep 1
done
k wait --for=condition=ready pod/secret-consumer --timeout=60s
sleep 3
k logs secret-consumer

Task 5: Verify Secret Status and Health

Check the status of all ExternalSecret and SealedSecret resources so you can distinguish successful reconciliation from objects that merely exist.

Solution

echo "=== ExternalSecret Status ==="
k get externalsecrets -o wide

echo ""
echo "=== SealedSecret Status ==="
k get sealedsecrets -o wide

echo ""
echo "=== All Secrets (non-system) ==="
k get secrets --field-selector type!=kubernetes.io/service-account-token

echo ""
echo "=== ESO SecretStore Status ==="
k get secretstores -o wide

Success Criteria

• SealedSecret is applied and the controller creates a K8s Secret
• ESO fake SecretStore syncs secrets to K8s Secrets
• Templated ExternalSecret generates a valid connection string
• Pod reads secrets from both Sealed Secrets and ESO
• All ExternalSecrets show SecretSynced status

Cleanup

kind delete cluster --name secrets-lab

---

Next Module

Module 9.9: Cloud-Native API Gateways & WAF — Learn how cloud API gateways compare to Kubernetes Gateway API, how to integrate WAF protection, and how to handle OAuth2/OIDC proxying for your services.

Sources

• Kubernetes Secrets — Native Secret behavior, base64 encoding, default etcd storage warning, and size considerations.
• Good practices for Kubernetes Secrets — Kubernetes guidance for encryption at rest, RBAC, external stores, and access restrictions.
• Encrypting Confidential Data at Rest — Kubernetes EncryptionConfiguration, KMS provider, and encryption verification guidance.
• ExternalSecret API — ESO refresh policy, refresh interval, target templating, and sync behavior.
• ClusterSecretStore API — ESO cluster-scoped store behavior and provider configuration model.
• Secrets Store CSI Driver Concepts — CSI mount flow, provider model, and security implications.
• Secrets Store CSI Driver Auto Rotation — Mounted-content rotation, synced Secret behavior, and environment-variable restart caveat.
• AWS Secrets Manager Encryption — AWS KMS envelope encryption behavior for secret values.
• AWS Secrets Manager Rotation by Lambda — Lambda-backed rotation workflow and rotation steps.
• AWS Secrets Manager Rotation Schedules — Rotation windows, rate expressions, and schedule constraints.
• AWS Secrets Manager Pricing — Current per-secret and API-call pricing model; verify region-specific pricing before estimating production spend.
• AWS Systems Manager Parameter Store Tiers — Standard versus Advanced parameter limits, size, policies, and cost distinction.
• AWS Parameter Store KMS SecureString — How SecureString parameters use AWS KMS.
• Amazon EKS Pod Identity — EKS Pod Identity behavior, benefits, limits, and credential isolation.
• IAM Roles for Service Accounts — IRSA behavior and OIDC-based temporary AWS credentials for pods.
• Google Secret Manager Overview — Secret versions, IAM, replication, encryption, and CMEK behavior.
• Google Secret Manager Pricing — Active version, replication-location, access-operation, and rotation-notification pricing model.
• Workload Identity Federation for GKE — GKE workload identity behavior and keyless access to Google Cloud APIs.
• GKE Secret Manager Workload Identity Tutorial — Official GKE pattern for accessing Secret Manager without service account key files.
• Azure Key Vault Overview — Key Vault secrets, keys, certificates, encryption at rest, access control, and monitoring.
• Azure Key Vault Keys, Secrets, and Certificates — Object identifiers, versions, and supported object types.
• Azure Key Vault Soft-Delete — Soft-delete and purge behavior for vaults and vault objects.
• Azure Key Vault Pricing — Operations-oriented pricing model and billing definitions; verify current regional pricing before estimation.
• AKS Microsoft Entra Workload ID — AKS workload identity with projected service account tokens and OIDC federation.
• Vault Database Secrets Engine — Dynamic database credentials, leases, static roles, and revocation semantics in Vault.
• Vault Lease Concepts — Lease creation, renewal, and revocation behavior for dynamic secrets.
• Vault Secrets Operator — VSO CRDs for Vault connection, authentication, static secrets, and dynamic secrets.
• Vault Seal Configuration — Seal stanza behavior and Shamir fallback when no auto-unseal configuration exists.
• Vault AWS KMS Seal — AWS KMS auto-unseal stanza and activation behavior.
• Vault GCP Cloud KMS Seal — GCP Cloud KMS auto-unseal stanza and authentication model.
• Vault Azure Key Vault Seal — Azure Key Vault auto-unseal stanza and activation behavior.
• Flux Kustomization Decryption with SOPS — Current reference for SOPS-backed GitOps decryption workflows in Kubernetes.