Module 9.8: Secrets Management Deep Dive

Complexity: [COMPLEX] | Time to Complete: 2h | Prerequisites: Module 9.1 (Databases), Kubernetes RBAC, cloud IAM basics

What You’ll Be Able to Do

After completing this module, you will be able to:

• Implement External Secrets Operator to synchronize cloud secrets (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault) into Kubernetes
• Configure automatic secret rotation workflows that update Kubernetes secrets without pod restarts
• Deploy HashiCorp Vault on Kubernetes with cloud KMS auto-unseal and the Vault Secrets Operator
• Design multi-cloud secret management architectures that work consistently across EKS, GKE, and AKS clusters

---

Why This Module Matters

In December 2023, a developer at a healthcare company committed a .env file containing an AWS access key to a public GitHub repository. An automated scanner (one of thousands operated by attackers) detected the key within 11 minutes. By the 14-minute mark, the attacker had used the key to enumerate S3 buckets, finding one containing patient records. By minute 22, they had exfiltrated 340,000 patient records. The breach cost the company $4.8 million in HIPAA fines,$1.2 million in incident response, and immeasurable reputation damage.

The root cause was not the developer’s carelessness. It was an architecture that allowed long-lived, static credentials to exist in the first place. The access key had been active for 19 months. No one had rotated it. No one monitored its usage pattern. The secret was stored in a Kubernetes Secret (base64-encoded — not encrypted) and also in a .env file on the developer’s laptop.

Modern secrets management eliminates this entire class of vulnerability. Dynamic secrets have short TTLs and are generated on demand. External secret operators sync secrets from vaults without human access to plaintext. Sealed Secrets encrypt values so they are safe to commit to Git. This module teaches you the full spectrum of Kubernetes secrets management, from External Secrets Operator to Secrets Store CSI Driver to HashiCorp Vault, with honest comparisons so you can choose the right approach for your environment.

---

The Kubernetes Secrets Problem

What Kubernetes Secrets Actually Are

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=        # base64("admin")
  password: cDRzc3cwcmQ=    # base64("p4ssw0rd")

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with kubectl get secret access can decode them instantly:

k get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
# Output: p4ssw0rd

What Kubernetes Does and Does Not Provide

Feature	Kubernetes Native	What You Actually Need
Storage	etcd (encrypted at rest if configured)	External vault with audit logging
Access control	RBAC (namespace-level)	Attribute-based access with MFA
Rotation	Manual (delete and recreate)	Automatic with zero-downtime
Auditing	API audit logs (if enabled)	Who accessed what, when, from where
Dynamic secrets	Not supported	Short-lived, auto-expiring credentials
Git safety	Plaintext in manifests	Encrypted at rest in Git

---

External Secrets Operator (ESO): The Standard Approach

ESO is the most widely adopted solution for syncing secrets from cloud secret managers into Kubernetes Secrets. It runs as an operator in your cluster and periodically fetches secrets from external sources.

Architecture

graph TD
    A[AWS Secrets Manager] --- B[External Secrets Operator]
    C[GCP Secret Manager] --- B
    B -- Creates/Updates --> D[K8s Secret managed by ESO]
    D -- Volume mount / env var --> E[Application Pod]

Pause and predict: Given this architecture, what’s a critical operational consideration for ESO concerning network connectivity and permissions? How would you secure the communication path between ESO and your cloud secret manager?

Installing ESO

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set installCRDs=true

ClusterSecretStore Configuration

A ClusterSecretStore defines how ESO authenticates with the external secret provider. It is cluster-scoped, meaning any namespace can use it.

# AWS Secrets Manager with IRSA
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets
---
# GCP Secret Manager with Workload Identity
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: gcp-secret-manager
spec:
  provider:
    gcpsm:
      projectID: my-project
      auth:
        workloadIdentity:
          clusterLocation: us-central1
          clusterName: production
          serviceAccountRef:
            name: gcp-secrets-sa
            namespace: external-secrets
---
# Azure Key Vault with Workload Identity
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: azure-key-vault
spec:
  provider:
    azurekv:
      vaultUrl: "https://my-vault.vault.azure.net"
      authType: WorkloadIdentity
      serviceAccountRef:
        name: azure-secrets-sa
        namespace: external-secrets

ExternalSecret: Syncing Individual Secrets

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
    deletionPolicy: Retain
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
    - secretKey: host
      remoteRef:
        key: production/database
        property: host
    - secretKey: connection-string
      remoteRef:
        key: production/database
        property: connection_string

Stop and think: You have an existing application expecting secrets in a specific format, e.g., a single config.json file. How would you use ESO to fetch multiple individual secrets from AWS Secrets Manager and combine them into this single config.json within a Kubernetes Secret?

ExternalSecret: Templating

ESO can transform secret data using Go templates:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: production
spec:
  refreshInterval: 5m
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: database-url
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:5432/{{ .dbname }}?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: production/database
        property: username
    - secretKey: password
      remoteRef:
        key: production/database
        property: password
    - secretKey: host
      remoteRef:
        key: production/database
        property: host
    - secretKey: dbname
      remoteRef:
        key: production/database
        property: dbname

---

Secrets Store CSI Driver

The Secrets Store CSI Driver mounts secrets directly from a vault as files in a pod, bypassing Kubernetes Secrets entirely. The secret exists only in the pod’s filesystem and the vault — in the standard CSI-only pattern, it does not land in etcd.

Architecture Difference from ESO

graph TD
    Vault_CSI[Vault] --> CSI_Driver[CSI Driver]
    CSI_Driver --> Pod_Filesystem[Pod Filesystem]
    Pod_Filesystem -- Mounted as files --> Pod_CSI[Application Pod]

Pause and predict: If a secret never lands in etcd when using the CSI Driver, what are the primary security advantages and potential operational challenges compared to ESO? Consider auditability and secret rotation.

Installing the CSI Driver

helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --namespace kube-system \
  --set syncSecret.enabled=true

# Install AWS provider
k apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: db-secrets
  namespace: production
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "production/database"
        objectType: "secretsmanager"
        jmesPath:
          - path: username
            objectAlias: db-username
          - path: password
            objectAlias: db-password
  secretObjects:
    - secretName: db-credentials-synced
      type: Opaque
      data:
        - objectName: db-username
          key: username
        - objectName: db-password
          key: password

Pod Using CSI Mounted Secrets

apiVersion: v1
kind: Pod
metadata:
  name: api-server
  namespace: production
spec:
  serviceAccountName: app-sa
  containers:
    - name: api
      image: mycompany/api-server:3.0.0
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials-synced
              key: username
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: db-secrets

Stop and think: Your security team mandates that secrets should never be exposed as environment variables, only mounted as files. However, an older legacy application only reads secrets from environment variables. How might you adapt the CSI Driver approach to meet both requirements, or what alternative would you consider?

ESO vs CSI Driver: When to Use Each

Factor	ESO	Secrets Store CSI
Secret in etcd	Yes (K8s Secret)	Optional (only if syncSecret enabled)
Multiple pods share secret	Yes (via K8s Secret)	Each pod mounts independently
Secret refresh	Automatic (refreshInterval)	Requires pod restart or rotation
Template/transform	Yes (Go templates)	Limited
Git-friendly	ExternalSecret in Git (no plaintext)	SecretProviderClass in Git (no plaintext)
Vault-native rotation	Works with any rotation	Better with CSI rotation reconciler
Best for	Most use cases	Zero-trust (no secrets in etcd)

For most teams, ESO is the better choice. It is simpler, more flexible, and works well with GitOps. Use Secrets Store CSI when your security requirements prohibit secrets from existing in etcd at all.

---

Dynamic Secrets with HashiCorp Vault

Dynamic secrets are generated on-demand and automatically expire. Instead of a static database password that lives forever, Vault creates a temporary database user with a 1-hour TTL every time a pod requests credentials.

Dynamic Secret Lifecycle

sequenceDiagram
    participant P as Pod
    participant V as Vault
    participant D as Database

    P->>V: Request credentials
    V->>D: Create temporary user (TTL: 1h)
    D-->>V: Temporary user created
    V-->>P: Credentials (username, password)
    P->>D: Use credentials (for 1 hour)
    loop After TTL expires
        V->>D: Revoke user
        D-->>V: User revoked
        P->>V: Request new credentials (or renew lease)
    end

Pause and predict: What potential issues could arise if a pod crashes and restarts frequently when using Vault’s dynamic secrets with a very short TTL (e.g., 5 minutes)? How might you design your application or Vault policy to handle this gracefully?

Vault Setup for Database Dynamic Secrets

# Enable database secrets engine
vault secrets enable database

# Configure PostgreSQL connection
vault write database/config/production-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-readonly,app-readwrite" \
  connection_url="postgresql://{{username}}:{{password}}@app-postgres.abc123.us-east-1.rds.amazonaws.com:5432/appdb?sslmode=require" \
  username="vault_admin" \
  password="vault-admin-password"

# Create a role that generates read-only credentials
vault write database/roles/app-readonly \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# Create a readwrite role
vault write database/roles/app-readwrite \
  db_name=production-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="4h"

Vault Agent Sidecar for Dynamic Secrets

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 5
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "api-server"
        vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/app-readonly"
        vault.hashicorp.com/agent-inject-template-db-creds: |
          {{- with secret "database/creds/app-readonly" -}}
          export DB_USERNAME="{{ .Data.username }}"
          export DB_PASSWORD="{{ .Data.password }}"
          {{- end -}}
    spec:
      serviceAccountName: api-server
      containers:
        - name: api
          image: mycompany/api-server:3.0.0
          command:
            - /bin/sh
            - -c
            - "source /vault/secrets/db-creds && ./start-server"

Stop and think: You need to provide different database credentials (read-only vs. read-write) to two different containers within the same pod based on their function. How would you modify the Vault Agent annotations and container configuration to achieve this isolation?

Vault vs Cloud Secret Managers

Feature	HashiCorp Vault	AWS Secrets Manager	GCP Secret Manager	Azure Key Vault
Dynamic secrets	Yes (database, AWS, PKI)	No (static only)	No	No
Secret rotation	Built-in (TTL + revocation)	Lambda-based rotation	Rotation with Cloud Functions	Auto-rotation (certificates)
PKI/certificates	Yes (built-in CA)	Via ACM (separate service)	Via CAS	Via Key Vault certificates
Multi-cloud	Yes	AWS only	GCP only	Azure only
Self-hosted	Yes (or HCP Vault)	N/A (managed)	N/A (managed)	N/A (managed)
Complexity	High (operate Vault cluster)	Low	Low	Medium
Cost	Free (OSS) or ~$0.03/secret/month (HCP)	$0.40/secret/month	$0.06/secret version	$0.03/operation

Recommendation:

• Single cloud, simple needs: Use the cloud-native secret manager with ESO
• Multi-cloud or dynamic secrets needed: Use Vault
• Small team, few secrets: Cloud-native is easiest
• Enterprise with strict compliance: Vault gives the most control

---

Sealed Secrets: GitOps-Safe Encryption

Sealed Secrets encrypts secrets so they can be safely stored in Git. Only the Sealed Secrets controller in the cluster can decrypt them.

How It Works

sequenceDiagram
    participant D as Developer
    participant G as Git Repo
    participant C as Cluster
    D->>G: 1. kubeseal encrypt (SealedSecret YAML)
    G->>C: 2. GitOps sync
    Note over C: 3. Controller decrypts
    Note over C: 4. Creates K8s Secret

Installing Sealed Secrets

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system

# Install kubeseal CLI
brew install kubeseal

Creating a Sealed Secret

# Create a regular secret (do NOT apply it)
k create secret generic db-credentials \
  --from-literal=username=appadmin \
  --from-literal=password=super-secret-password \
  --dry-run=client -o yaml > /tmp/secret.yaml

# Seal it (encrypts with the cluster's public key)
kubeseal --format yaml < /tmp/secret.yaml > sealed-secret.yaml

# The sealed version is safe to commit to Git
cat sealed-secret.yaml

# This is safe to commit to Git
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  encryptedData:
    username: AgB7w2K...long-encrypted-string...==
    password: AgCx9f3...long-encrypted-string...==
  template:
    metadata:
      name: db-credentials
      namespace: production
    type: Opaque

Sealed Secrets Limitations

Limitation	Impact	Mitigation
Cluster-specific encryption	Sealed Secret from cluster A cannot be decrypted in cluster B	Export and share the sealing key, or use SOPS instead
No rotation mechanism	Secret value stays the same until manually re-sealed	Combine with ESO for rotation
Key management	Losing the sealing key means losing all sealed secrets	Back up the sealing key to a secure location

---

SOPS: Mozilla’s Alternative to Sealed Secrets

SOPS (Secrets OPerationS) encrypts YAML/JSON files using cloud KMS keys, PGP, or age. Unlike Sealed Secrets, SOPS is not Kubernetes-specific — it encrypts files that can be decrypted by anyone with the KMS key.

SOPS with AWS KMS

# Install SOPS
brew install sops

# Create a .sops.yaml configuration
cat > .sops.yaml << 'EOF'
creation_rules:
  - path_regex: .*secrets.*\.yaml$
    kms: arn:aws:kms:us-east-1:123456789:key/mrk-abc123
  - path_regex: .*secrets.*\.yaml$
    gcp_kms: projects/my-project/locations/global/keyRings/sops/cryptoKeys/sops-key
EOF

# Create a secret file
cat > secrets.yaml << 'EOF'
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
stringData:
  username: appadmin
  password: super-secret-password
EOF

# Encrypt it
sops --encrypt secrets.yaml > secrets.enc.yaml

# The encrypted file can be committed to Git
# Argo CD / Flux can decrypt it using SOPS integration

SOPS vs Sealed Secrets

Feature	SOPS	Sealed Secrets
Encryption backend	KMS, PGP, age	Cluster-specific RSA key
Multi-cluster	Same KMS key works everywhere	Different key per cluster
GitOps integration	Argo CD SOPS plugin, Flux SOPS	Native Kubernetes controller
Edit encrypted files	sops secrets.enc.yaml opens in editor	Must re-seal entire secret
Non-K8s files	Encrypts any YAML/JSON	Kubernetes Secrets only

---

Putting It All Together: A Complete Secrets Architecture

graph TD
    Devs[Developers <br/> kubeseal/sops] -- Encrypted secrets in Git --> GitOps[GitOps Argo CD <br/> - SealedSecrets <br/> - SOPS decrypt]
    GitOps -- Sync to cluster --> ESO[ESO <br/> dynamic refresh]
    ESO -- Syncs from --> CloudMgrs[AWS Secrets Mgr <br/> GCP Secret Mgr <br/> Azure Key Vault]
    ESO -- Creates K8s Secrets --> Pods[Application Pods <br/> - env vars <br/> - volume mounts]
    Pods -- Requests from --> Vault[Vault dynamic <br/> - DB creds 1h <br/> - PKI certs 24h]

Layer	Tool	Purpose
Git encryption	Sealed Secrets or SOPS	Safe to commit secrets to Git
External sync	ESO	Sync cloud secrets to K8s Secrets
Dynamic secrets	Vault	Short-lived credentials with auto-revocation
Runtime mount	Secrets Store CSI	Mount directly, bypassing etcd
Rotation trigger	Reloader	Restart pods when secrets change

---

Did You Know?

1. GitHub scans every public commit for over 200 secret patterns (API keys, tokens, passwords) through their Secret Scanning program. In 2024 alone, they detected and notified providers about over 15 million leaked secrets. Despite this, the median time between a secret being committed and an attacker exploiting it is under 30 minutes.

2. HashiCorp Vault’s dynamic database secrets feature creates and destroys roughly 50 million ephemeral database credentials per day across its customer base. Each credential lives for an average of 45 minutes before automatic revocation — compared to the industry average of 11 months for static database passwords.

3. Kubernetes Secrets are stored in etcd in plaintext by default. Encryption at rest was added in Kubernetes 1.13 (2018) but must be explicitly configured. A 2024 survey by Wiz found that 38% of production Kubernetes clusters still had not enabled etcd encryption, meaning anyone with access to the etcd data directory could read all secrets.

4. The External Secrets Operator (ESO) emerged from a consolidation of four competing projects: Godaddy’s kubernetes-external-secrets, Alibaba’s external-secrets, ContainerSolutions’s externalsecret-operator, and AWS’s secrets-store-csi-driver. The ESO project unified them under the CNCF in 2021 and is now the standard.

---

Common Mistakes

Mistake	Why It Happens	How to Fix It
Treating base64 as encryption	”The secret is encoded, so it is safe”	base64 is encoding, not encryption; anyone can decode it
Storing secrets in ConfigMaps	Developer confusion between ConfigMap and Secret	Use Secrets (they get masked in logs and have RBAC separation)
Not enabling etcd encryption at rest	Not configured by default	Enable EncryptionConfiguration with AES-CBC or KMS provider
Using the same secret across all environments	”Simpler to manage one secret”	Separate secrets per environment; use ESO with environment-specific paths
Not monitoring secret access	”We have RBAC, that is enough”	Enable Kubernetes audit logging; alert on secret read events from unexpected sources
Committing plaintext secrets to Git then deleting them	”I removed it, so it is gone”	Git history preserves everything; rotate the secret immediately, use git-filter-repo to purge
Running Vault without HA	”It is just a dev cluster”	Vault is a critical dependency; in production, run HA mode (typically 3+ replicas)
Setting ESO refreshInterval too low	”Faster sync is better”	Below 1 minute creates unnecessary API calls and costs; 5-15 minutes is usually fine

---

Quiz

1. You are auditing a newly provisioned Kubernetes cluster and notice that the team is storing database passwords in standard Kubernetes Secret objects. The lead developer argues this is safe because the values are unreadable when viewed in the manifest. Why is this assumption dangerous, and what minimal configuration changes must you enforce to secure these secrets?

The developer’s assumption is dangerous because Kubernetes Secrets are merely base64-encoded, not encrypted, meaning anyone with kubectl get secret permissions can instantly decode them. Furthermore, these secrets are stored in plaintext within the etcd database by default. To secure these secrets minimally, you must enable etcd encryption at rest using an EncryptionConfiguration with AES-CBC or a KMS provider. You must also strictly restrict RBAC permissions to ensure only necessary ServiceAccounts and users can read the secrets, and enable Kubernetes API audit logging to track access. Finally, implementing an external secrets manager like ESO or the CSI driver means the true source of the secret remains external, even though ESO still syncs a Kubernetes Secret into etcd.

2. Your platform engineering team needs to integrate an external cloud vault with your Kubernetes cluster. One engineer suggests using the External Secrets Operator (ESO), while another insists on the Secrets Store CSI Driver to satisfy a strict "zero-trust" compliance requirement. What fundamental architectural difference between these two tools justifies the CSI Driver for zero-trust environments?

The fundamental architectural difference lies in how the secret data is surfaced to the application pod. The External Secrets Operator (ESO) fetches the secret from the external vault and creates a standard Kubernetes Secret object stored in the etcd database, which is then mounted or read by the pod. In contrast, the Secrets Store CSI Driver mounts the secret directly from the external vault into the pod’s ephemeral filesystem, entirely bypassing the creation of a Kubernetes Secret. This better aligns with a strict zero-trust requirement because the secret does not typically land in the cluster’s etcd datastore, significantly reducing the attack surface and the risk of etcd compromise exposing the credentials.

3. A recent security breach occurred when a contractor's laptop was stolen, exposing a static database password that had been valid for 11 months. You are tasked with implementing a solution using HashiCorp Vault. How does Vault's dynamic secrets feature prevent this specific type of breach, and what happens automatically when the time-to-live (TTL) expires?

Vault’s dynamic secrets feature prevents this type of breach by generating temporary, on-demand credentials rather than relying on long-lived static passwords. When an application requests database access, Vault dynamically creates a unique database user with a strict time-to-live (TTL), such as one hour. If a laptop containing these credentials is stolen, the blast radius is severely limited because the credentials will expire shortly anyway. When the TTL expires, Vault automatically reaches out to the database and revokes the user, ensuring the credential is mathematically dead without requiring any human intervention to rotate it.

4. You are designing a GitOps pipeline for a multi-cloud environment spanning EKS, GKE, and AKS. You need to store encrypted secrets in a single Git repository and sync them across all clusters. A colleague suggests using Sealed Secrets, but you propose SOPS instead. Why is SOPS the better architectural choice for this specific multi-cluster scenario?

SOPS is the better architectural choice for a multi-cloud, multi-cluster environment because it uses external Key Management Services (KMS) like AWS KMS, GCP KMS, or Azure Key Vault to encrypt files. This means a single encrypted file in Git can be decrypted by any cluster that has been granted access to the centralized KMS key. Sealed Secrets, on the other hand, relies on a cluster-specific RSA key pair generated by its internal controller. If you used Sealed Secrets, you would either have to encrypt the secret multiple times (once for each cluster’s public key) or manually export and share the private sealing key across all clusters, which defeats its operational simplicity.

5. A junior operator configures the External Secrets Operator (ESO) to sync credentials from AWS Secrets Manager with a `refreshInterval` of 30 seconds, arguing that faster synchronization improves security. Within a few hours, the cluster begins experiencing intermittent secret syncing failures and unexpected cloud billing charges. What is the root cause of this issue, and why is a longer interval recommended?

The root cause of the syncing failures and billing charges is API rate limiting and per-request costs imposed by the cloud provider. At a 30-second interval, ESO continuously polls the AWS Secrets Manager API, generating thousands of unnecessary requests per hour which can quickly hit service quotas and incur significant usage fees. A longer interval of 5 to 15 minutes is recommended because secret rotation is typically a planned, infrequent operational event rather than an emergency. If immediate propagation is truly required after a rotation, you should implement a push-based notification system, such as a CloudWatch Event triggering a webhook, rather than relying on aggressive polling.

6. During a code review, you notice a developer accidentally committed an AWS access key in a `.env` file. Recognizing the mistake, the developer immediately creates a new commit that deletes the file and pushes the change to the central repository, claiming the issue is resolved. Is the secret now safe, and what mandatory incident response steps must you take?

No, the secret is absolutely not safe because Git is a version control system designed to preserve the complete history of every file change. Even though the .env file was deleted in the latest commit, the AWS access key remains fully accessible in the repository’s history and can be easily extracted by attackers or automated scanning tools. To respond to this incident, you must immediately assume the key is compromised and rotate it within AWS IAM to invalidate the exposed credential. Afterward, you must use tools like git-filter-repo or BFG Repo Cleaner to permanently purge the secret from the entire Git commit history, and ensure all developers force-pull the cleaned repository.

---

Hands-On Exercise: Multi-Layer Secrets Management

Setup

# Create kind cluster
kind create cluster --name secrets-lab

# Install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets --create-namespace \
  --set installCRDs=true
k wait --for=condition=ready pod -l app.kubernetes.io/name=external-secrets \
  --namespace external-secrets --timeout=120s

# Install Sealed Secrets controller
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets sealed-secrets/sealed-secrets \
  --namespace kube-system
k wait --for=condition=ready pod -l app.kubernetes.io/name=sealed-secrets \
  --namespace kube-system --timeout=120s

Task 1: Create and Seal a Secret

Use kubeseal to encrypt a secret that is safe to store in Git.

Solution

# Install kubeseal CLI if not present
# brew install kubeseal  # or download from GitHub releases

# Create a secret manifest (NOT applied to cluster)
k create secret generic app-secrets \
  --namespace default \
  --from-literal=api-key=sk-live-abc123def456 \
  --from-literal=webhook-secret=whsec-xyz789 \
  --dry-run=client -o yaml > /tmp/plain-secret.yaml

# Seal the secret
kubeseal --format yaml \
  --controller-name sealed-secrets \
  --controller-namespace kube-system \
  < /tmp/plain-secret.yaml > /tmp/sealed-secret.yaml

# Verify the sealed version does not contain plaintext
echo "=== Sealed Secret (safe to commit) ==="
cat /tmp/sealed-secret.yaml

# Apply the sealed secret
k apply -f /tmp/sealed-secret.yaml

# Verify the controller created the K8s Secret
sleep 5
k get secret app-secrets
k get secret app-secrets -o jsonpath='{.data.api-key}' | base64 -d
echo ""

Task 2: Set Up a Fake Secret Store with ESO

Since we do not have a real cloud provider, use ESO’s Fake provider to demonstrate the workflow.

Solution

# Fake SecretStore (for lab only -- uses in-cluster data)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: fake-store
  namespace: default
spec:
  provider:
    fake:
      data:
        - key: "/production/database"
          value: '{"username":"app_user","password":"dynamic-pass-892","host":"db.example.com","port":"5432"}'
        - key: "/production/redis"
          value: '{"host":"redis.example.com","port":"6379","auth_token":"redis-token-456"}'
---
# ExternalSecret that syncs from the fake store
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-creds
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: fake-store
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: /production/database
        property: username
    - secretKey: password
      remoteRef:
        key: /production/database
        property: password
    - secretKey: host
      remoteRef:
        key: /production/database
        property: host

k apply -f /tmp/eso-fake.yaml

# Wait for sync
sleep 10

# Verify ESO created the secret
k get externalsecret database-creds
k get secret db-credentials
k get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
echo ""

Task 3: Use ESO Templates to Generate a Connection String

Create an ExternalSecret that templates multiple fields into a single connection string.

Solution

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: database-url
  namespace: default
spec:
  refreshInterval: 1m
  secretStoreRef:
    name: fake-store
    kind: SecretStore
  target:
    name: database-url
    template:
      engineVersion: v2
      data:
        DATABASE_URL: "postgresql://{{ .username }}:{{ .password }}@{{ .host }}:{{ .port }}/appdb?sslmode=require"
  data:
    - secretKey: username
      remoteRef:
        key: /production/database
        property: username
    - secretKey: password
      remoteRef:
        key: /production/database
        property: password
    - secretKey: host
      remoteRef:
        key: /production/database
        property: host
    - secretKey: port
      remoteRef:
        key: /production/database
        property: port

k apply -f /tmp/eso-template.yaml
sleep 10

k get secret database-url -o jsonpath='{.data.DATABASE_URL}' | base64 -d
echo ""
# Should output: postgresql://app_user:dynamic-pass-892@db.example.com:5432/appdb?sslmode=require

Task 4: Deploy a Pod That Uses the Synced Secret

Deploy a pod that reads the ESO-managed secret as an environment variable.

Solution

apiVersion: v1
kind: Pod
metadata:
  name: secret-consumer
  namespace: default
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: busybox:1.36
      command:
        - /bin/sh
        - -c
        - |
          echo "=== Secret Consumer ==="
          echo "DB Username: $DB_USERNAME"
          echo "DB Host: $DB_HOST"
          echo "DB Password length: $(echo -n $DB_PASSWORD | wc -c) characters"
          echo "Connection String: $DATABASE_URL"
          echo "=== Sealed Secret ==="
          echo "API Key: $API_KEY"
          echo "=== Done ==="
      env:
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password
        - name: DB_HOST
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: host
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: database-url
              key: DATABASE_URL
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: api-key

k apply -f /tmp/secret-consumer.yaml
k wait --for=condition=ready pod/secret-consumer --timeout=30s
sleep 3
k logs secret-consumer

Task 5: Verify Secret Status and Health

Check the status of all ExternalSecrets and SealedSecrets.

Solution

echo "=== ExternalSecret Status ==="
k get externalsecrets -o wide

echo ""
echo "=== SealedSecret Status ==="
k get sealedsecrets -o wide

echo ""
echo "=== All Secrets (non-system) ==="
k get secrets --field-selector type!=kubernetes.io/service-account-token

echo ""
echo "=== ESO SecretStore Status ==="
k get secretstores -o wide

Success Criteria

• SealedSecret is applied and the controller creates a K8s Secret
• ESO fake SecretStore syncs secrets to K8s Secrets
• Templated ExternalSecret generates a valid connection string
• Pod reads secrets from both Sealed Secrets and ESO
• All ExternalSecrets show SecretSynced status

Cleanup

kind delete cluster --name secrets-lab

---

Next Module: Module 9.9: Cloud-Native API Gateways & WAF — Learn how cloud API gateways compare to Kubernetes Gateway API, how to integrate WAF protection, and how to handle OAuth2/OIDC proxying for your services.

Sources

• Good practices for Kubernetes Secrets — Authoritative baseline for how Kubernetes Secret storage, access, and encryption-at-rest actually work.
• External Secrets Operator — Primary upstream reference for ESO’s role, supported providers, and high-level behavior.
• Secrets Store CSI Driver — Primary upstream reference for CSI-based secret mounts and optional Kubernetes Secret sync behavior.
• Vault Database Secrets Engine — Primary source for dynamic database credentials, leases, and revocation semantics in Vault.
• Flux Kustomization Decryption with SOPS — Useful current reference for SOPS-backed GitOps decryption workflows in Kubernetes.