"The Vault" — Secret Management with HashiCorp Vault and Sealed Secrets

It was a Tuesday morning, and Alex, NovaCraft’s newly minted Platform Lead, was staring at a Slack notification that made his blood run cold. A red, flashing alert from their continuous integration pipeline screamed: “SECRET DETECTED IN GIT COMMIT.”

Someone had accidentally committed a production database password to a public-facing Git repository. The security team was already on a war footing, and the all-hands-on-deck incident response was in full swing. Alex’s mind raced. He had spent the last six months evangelizing Kubernetes, migrating services, and building what he thought was a robust platform. But this single, simple mistake had exposed a fundamental weakness in their security posture.

As the initial chaos subsided and the immediate fire was extinguished—the commit was scrubbed, the password rotated, and access logs audited—Alex was left with a lingering sense of unease. He had followed the Kubernetes gospel, using standard Secret objects to manage sensitive information. But the incident had laid bare a painful truth: Kubernetes Secrets, by themselves, are not enough.

The Illusion of Security: Why Kubernetes Secrets Fall Short

For many, the name Secret in Kubernetes implies a level of security that doesn’t quite match reality. While they are a step up from hardcoding credentials in container images or environment variables, they have significant limitations, especially as an organization scales.

A Kubernetes Secret is just a Base64-encoded string. Anyone with API access to the cluster can read it. Base64 is encoding, not encryption. It’s like writing a postcard in a language that anyone with a decoder ring can read.

Here’s a breakdown of the core issues Alex was now grappling with:

Alex realized that to truly secure NovaCraft’s platform, he needed to move beyond the basics. He needed a system that could provide centralized management, strong encryption, dynamic secrets, and a secure workflow for GitOps. His research led him to a powerful trifecta of tools designed to solve these very problems: HashiCorp Vault, Sealed Secrets, and the External Secrets Operator.

This chapter is the story of how Alex went “Beyond the Cluster” to build a production-grade secret management solution. We’ll dive deep into the architecture of these tools, compare their strengths, and walk through a hands-on implementation that you can replicate on your own machine.

The Fort Knox of Secrets: An Introduction to HashiCorp Vault

Alex’s first and most significant discovery was HashiCorp Vault. Vault is an open-source tool that provides a centralized, secure, and highly available solution for managing secrets. It’s not just a key-value store; it’s a complete platform for secrets management, encryption as a service, and identity-based access.

Vault’s Architecture: A Fortress Model

To understand Vault, you need to understand its core architectural principles. Think of it as a digital fortress with multiple layers of defense.

• Storage Backend: Vault doesn’t store data itself. It relies on a separate storage backend for persistence. This could be a cloud storage service like Amazon S3 or Google Cloud Storage, a database like PostgreSQL, or even a local file system. The key is that Vault considers the storage backend to be untrusted. All data is encrypted by Vault before it is written to the storage backend.

• The Barrier: The cryptographic boundary that protects Vault is called the Barrier. Everything that passes through the barrier is encrypted. This is the core of Vault’s security model. Even if an attacker gains access to the storage backend, the data is useless without the encryption keys, which are kept within the barrier.

• Seal/Unseal: When a Vault server starts, it’s in a sealed state. It knows where its data is, but it doesn’t know how to decrypt it. To become operational, it must be unsealed. This process involves providing a set of unseal keys. By default, Vault uses an algorithm called Shamir’s Secret Sharing to split the master key into multiple shards. A certain threshold of these shards is required to reconstruct the master key and unseal the Vault. This prevents a single person from having unilateral access to all secrets.

• Auth Methods: Before a client can interact with Vault, it must authenticate. Vault supports a wide range of auth methods, from traditional username/password and tokens to more advanced methods like AWS IAM, Kubernetes Service Accounts, and GitHub teams. Authentication is about verifying who you are.

• Policies: Once authenticated, Vault uses policies to determine what you are allowed to do. Policies are written in HashiCorp Configuration Language (HCL) and provide fine-grained control over which paths and operations a client can access. Vault operates on a principle of deny by default.

• Secrets Engines: This is where the magic happens. Secrets engines are responsible for storing, generating, or encrypting data. There are many types:

  • KV (Key-Value): The simplest secrets engine, for storing arbitrary secrets.
  • Database: Dynamically generates database credentials with a specific time-to-live (TTL).
  • AWS: Dynamically generates AWS IAM credentials.
  • Transit: Provides encryption as a service, allowing applications to encrypt data without having access to the encryption keys.

The Power of Dynamic Secrets

The concept of dynamic secrets was a game-changer for Alex. Instead of storing static, long-lived credentials in Vault, you configure a secrets engine (like the database or AWS engine) with a set of high-privilege credentials. Then, when an application needs to access the database, it asks Vault for credentials. Vault connects to the database, generates a new set of credentials with the exact permissions the application needs, and sets a TTL on them. When the TTL expires, Vault automatically revokes the credentials.

This approach has profound security implications:

• Reduced Risk of Leaks: The credentials are short-lived. Even if they are leaked, their window of usability is minimal.
• No More Credential Sprawl: You don’t have static credentials scattered across your infrastructure.
• Automated Rotation: Rotation is built-in. There’s no need for manual password changes.
• Fine-Grained Auditing: Every secret checkout is a discrete event that can be audited.

Getting Secrets into Pods: The Vault Agent Injector

So, Vault is amazing. But a crucial question remains: how does an application running in a Kubernetes Pod get a secret from Vault? You don’t want to bundle a Vault token with your application image. That would just be trading one secret for another.

This is where the Vault Agent Injector comes in. It’s a Kubernetes admission controller that intercepts Pod creation events and, based on a set of annotations, automatically injects a Vault Agent container into the Pod. This agent is a lightweight client that authenticates with Vault and manages the lifecycle of secrets for the application.

Here’s the workflow:

1. Annotate your Deployment: You add specific annotations to your Kubernetes Deployment, StatefulSet, or Pod manifest. These annotations tell the injector to act on this Pod, which Vault role to use, and which secrets to retrieve.

   annotations:
     vault.hashicorp.com/agent-inject: 'true'
     vault.hashicorp.com/role-name: 'myapp-role'
     vault.hashicorp.com/agent-inject-secret-database-config.json: 'database/creds/myapp-role'

2. Pod Creation is Intercepted: When you kubectl apply your manifest, the Vault Agent Injector (which is a MutatingAdmissionWebhook) intercepts the request to the Kubernetes API server.

3. The Pod is Mutated: The injector modifies the Pod specification, adding two containers:

   • An init container that authenticates with Vault using the Pod’s Service Account Token and fetches the initial set of secrets.
   • A sidecar container that runs alongside your application. This sidecar keeps the secrets fresh (renewing leases) and can even re-fetch them if they change.

4. Secrets are Shared: The agent containers share a memory volume with your application container. The secrets are written to files on this volume.

5. Application Consumes Secrets: Your application simply reads the secrets from the well-known file path (e.g., /vault/secrets/database-config.json). It doesn’t need any Vault-specific logic. The application remains completely unaware of Vault’s existence.

This elegant pattern decouples your application from the secret management system. Your developers can focus on writing code, not on integrating with Vault. The platform team, led by Alex, can manage the entire secret lifecycle centrally and securely.

The GitOps Conundrum: Sealed Secrets

Vault is a powerful solution for centralized secret management and dynamic credentials. But it doesn’t directly solve the original problem that triggered Alex’s security incident: how to safely store Kubernetes Secret manifests in a Git repository.

This is the exact problem that Sealed Secrets was created to solve. It’s a beautifully simple and effective tool for “one-way” encryption of secrets.

The Core Idea: Encrypt for the Cluster

Sealed Secrets works on a simple but powerful premise: anyone can encrypt a secret, but only the controller running in the target cluster can decrypt it.

It consists of two components:

1. A Controller: This runs in your Kubernetes cluster. On startup, it generates a public/private key pair. The private key stays securely within the controller in the cluster, while the public key is made available to developers.

2. A CLI tool (kubeseal): Developers use this tool, along with the controller’s public key, to encrypt a standard Kubernetes Secret into a new custom resource called a SealedSecret.

Here’s the workflow:

1. Author a standard Secret: A developer starts by creating a regular Kubernetes Secret manifest on their local machine. This file contains the sensitive data in the usual Base64-encoded format.

   # Create a secret locally
   kubectl create secret generic my-db-password --from-literal=password='S3cr3tP@ssw0rd!' --dry-run=client -o yaml > my-secret.yaml

2. Seal it with kubeseal: The developer then pipes this local Secret manifest into the kubeseal command. kubeseal fetches the public key from the controller running in the cluster and uses it to encrypt the data fields of the secret.

   # Seal the secret
   cat my-secret.yaml | kubeseal > sealed-my-secret.yaml

3. Commit the SealedSecret: The output is a SealedSecret manifest. This file is safe to commit to Git. The sensitive data is now encrypted with the public key, and the original Secret file can be discarded.

   apiVersion: bitnami.com/v1alpha1
   kind: SealedSecret
   metadata:
     name: my-db-password
     namespace: default
   spec:
     encryptedData:
       password: AgBy3i4OJSWK+PiTySYZZA9rO43cGDEq... # Encrypted value

4. Deploy with GitOps: Your GitOps tool (like ArgoCD or Flux) picks up the SealedSecret from the repository and applies it to the cluster.

5. The Controller Unseals: The Sealed Secrets controller, running in the cluster, is the only entity with the private key. It detects the new SealedSecret resource, decrypts the encryptedData, and creates a standard Kubernetes Secret in the cluster.

This workflow is perfect for GitOps. The source of truth in Git contains no plaintext secrets, yet the final, necessary Secret is available to your applications in the cluster. It solves the “secret in Git” problem elegantly and securely.

The Bridge to a Wider World: External Secrets Operator

Both Vault and Sealed Secrets are powerful, but they represent two distinct philosophies. Vault is a full-blown, centralized secret management system. Sealed Secrets is a focused tool for a specific GitOps workflow. What if you already have secrets in an external provider, like AWS Secrets Manager, Google Secret Manager, or Azure Key Vault, and you just want to sync them into your Kubernetes cluster?

This is where the External Secrets Operator (ESO) shines. It acts as a bridge, synchronizing secrets from an external source into Kubernetes Secret objects.

How It Works: A Syncing Mechanism

The External Secrets Operator extends the Kubernetes API with its own set of Custom Resource Definitions (CRDs), primarily:

• SecretStore: This resource defines how to connect to an external secret management system. It contains the connection details and authentication credentials for a specific provider (e.g., AWS region, Vault address).
• ExternalSecret: This resource specifies which secrets to fetch from a SecretStore and how to transform them into a Kubernetes Secret.

The workflow is straightforward:

1. Define a SecretStore: You create a SecretStore (or ClusterSecretStore for a cluster-wide scope) that points to your external provider.

   apiVersion: external-secrets.io/v1beta1
   kind: SecretStore
   metadata:
     name: aws-secrets-manager
   spec:
     provider:
       aws:
         service: SecretsManager
         region: us-east-1
         auth:
           jwt:
             serviceAccountRef:
               name: my-app-sa

2. Create an ExternalSecret: You define an ExternalSecret that references the SecretStore and specifies the remote secret to fetch.

   apiVersion: external-secrets.io/v1beta1
   kind: ExternalSecret
   metadata:
     name: my-db-credentials
   spec:
     refreshInterval: "1h"
     secretStoreRef:
       name: aws-secrets-manager
       kind: SecretStore
     target:
       name: my-db-credentials-secret # Name of the K8s Secret to create
     data:
     - secretKey: username
       remoteRef:
         key: my-app/db/credentials
         property: username
     - secretKey: password
       remoteRef:
         key: my-app/db/credentials
         property: password

3. The Operator Syncs: The External Secrets Operator detects the ExternalSecret resource. It uses the information in the referenced SecretStore to connect to the external provider, fetches the specified secret (my-app/db/credentials in this case), and creates (or updates) a native Kubernetes Secret named my-db-credentials-secret with the retrieved data.

4. Automatic Refresh: The operator will periodically re-fetch the secret based on the refreshInterval, ensuring that the Kubernetes Secret stays in sync with the external source.

ESO is the perfect tool when you have an established secret management system outside of Kubernetes and need a simple, reliable way to get those secrets into your cluster for your applications to consume.

Comparing the Approaches: Vault vs. Sealed Secrets vs. External Secrets

Alex now had three powerful tools in his arsenal. But which one was right for which job? He created a table to clarify their strengths and use cases.

Alex realized they didn’t have to choose just one. They could use them together:

• Vault as the central, authoritative source of truth for all critical secrets and for generating dynamic database credentials.
• External Secrets Operator to sync secrets from Vault into their Kubernetes clusters.
• Sealed Secrets for the few, simple secrets that were managed directly by development teams and needed to be stored in Git for specific applications.

This hybrid approach offered the best of all worlds: the power and security of Vault, the GitOps-native workflow of Sealed Secrets, and the bridging capability of the External Secrets Operator.

Hands-On: Building a Secure Secrets Workflow

Theory is great, but there’s no substitute for getting your hands dirty. In this section, we’ll walk through a complete, end-to-end example of setting up Vault and Sealed Secrets on a local minikube cluster. Follow along to see how these pieces fit together in a real-world scenario.

Prerequisites

Before we begin, make sure you have the following tools installed on your macOS machine:

• minikube
• kubectl
• Helm
• Vault CLI

Start your minikube cluster:

minikube start --driver=docker

Step 1: Deploy HashiCorp Vault

We’ll use the official HashiCorp Helm chart to deploy Vault. This will set up Vault in standalone mode, which is perfect for our local testing.

1. Add the HashiCorp Helm repository:

   helm repo add hashicorp https://helm.releases.hashicorp.com
   helm repo update

2. Install the Vault chart:

   We’ll install Vault with a few custom values. We’ll enable the UI and set the service type to NodePort so we can access it from our local machine.

   helm install vault hashicorp/vault --set "server.dev.enabled=true" --set "ui.enabled=true" --set "server.service.type=NodePort"

   This will deploy Vault in “dev” mode, which is an in-memory, pre-unsealed, single-node instance. This is great for development but should never be used in production.

3. Access the Vault UI:

   Once the Vault pod is running, you can access the UI. First, get the NodePort of the Vault service:

   export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services vault-ui)
   export NODE_IP=$(minikube ip)
   echo "http://$NODE_IP:$NODE_PORT"

   Open the printed URL in your browser. You should see the Vault UI. The root token for dev mode is root.

Step 2: Store and Retrieve a Secret with Vault Agent

Now that Vault is running, let’s configure it to work with Kubernetes and inject a secret into a sample application.

1. Exec into the Vault pod:

   kubectl exec -it vault-0 -- /bin/sh

2. Enable Kubernetes authentication:

   Inside the Vault pod’s shell, run the following commands. This tells Vault to trust the Kubernetes cluster’s service account tokens for authentication.

   # Enable the Kubernetes auth method
   vault auth enable kubernetes
   
   # Configure the Kubernetes auth method
   vault write auth/kubernetes/config \
       token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
       kubernetes_host="https://kubernetes.default.svc" \
       kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

3. Create a policy and a role:

   We need to create a policy that grants read access to a specific secret path, and a role that binds this policy to a Kubernetes service account.

   # Create a policy that allows reading a secret
   vault policy write myapp-policy - <<EOF
   path "secret/data/myapp/config" {
     capabilities = ["read"]
   }
   EOF
   
   # Create a role that binds the policy to a service account
   vault write auth/kubernetes/role/myapp-role \
       bound_service_account_names=myapp-sa \
       bound_service_account_namespaces=default \
       policies=myapp-policy \
       ttl=24h

4. Create a secret:

   Let’s put a secret in Vault for our application to consume.

   vault kv put secret/myapp/config username="my-app-user" password="my-app-password"

   Exit the Vault pod’s shell by typing exit.

5. Deploy the application:

   Now, we’ll deploy a simple application with the necessary annotations for the Vault Agent Injector. We’ll also create the ServiceAccount that we bound our Vault role to.

   Create a file named myapp.yaml:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: myapp-sa
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: myapp
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: myapp
     template:
       metadata:
         labels:
           app: myapp
         annotations:
           vault.hashicorp.com/agent-inject: "true"
           vault.hashicorp.com/role-name: "myapp-role"
           vault.hashicorp.com/agent-inject-secret-config: "secret/data/myapp/config"
       spec:
         serviceAccountName: myapp-sa
         containers:
         - name: myapp
           image: busybox
           command: ["sh", "-c", "while true; do sleep 5; done"]

   Apply the manifest:

   kubectl apply -f myapp.yaml

6. Verify the secret injection:

   Once the myapp pod is running, exec into it and check for the secret.

   kubectl exec -it $(kubectl get pods -l app=myapp -o jsonpath='{.items[0].metadata.name}') -- sh

   Inside the pod, you’ll find the secret in /vault/secrets/config:

   cat /vault/secrets/config

   You should see the username and password we stored in Vault!

   {
     "data": {
       "password": "my-app-password",
       "username": "my-app-user"
     },
     "metadata": {
       ...
     }
   }

Step 3: Set Up Sealed Secrets for GitOps

Now, let’s tackle the GitOps workflow with Sealed Secrets.

1. Install the Sealed Secrets controller:

   We’ll use the official Helm chart for this as well.

   helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
   helm repo update
   helm install sealed-secrets sealed-secrets/sealed-secrets -n kube-system

2. Install the kubeseal CLI:

   On macOS, you can use Homebrew:

   brew install kubeseal

3. Create a secret and seal it:

   Let’s create a new secret and seal it.

   # Create a secret locally
   kubectl create secret generic my-sealed-secret --from-literal=super-secret=so-secret --dry-run=client -o yaml > my-secret.yaml
   
   # Seal the secret
   cat my-secret.yaml | kubeseal > my-sealed-secret.yaml

   Take a look at my-sealed-secret.yaml. The super-secret value is now encrypted.

4. Deploy the sealed secret:

   Apply the SealedSecret manifest to your cluster:

   kubectl apply -f my-sealed-secret.yaml

5. Verify the unsealing:

   The Sealed Secrets controller will decrypt the SealedSecret and create a regular Kubernetes Secret. Check for it:

   kubectl get secret my-sealed-secret -o yaml

   You’ll see the data field contains the Base64-encoded version of our original secret. You can decode it to verify:

   echo 'c28tc2VjcmV0' | base64 --decode

   It should print so-secret.

Debugging and Troubleshooting

• Vault Agent Injector not working: Check the logs of the vault-agent-injector pod. It’s usually a problem with the webhook configuration or RBAC permissions.
• Sealed Secret not unsealing: Check the logs of the sealed-secrets-controller pod in the kube-system namespace. It might be a permissions issue or a problem with the key.
• kubeseal can’t fetch the certificate: Make sure your kubectl is configured correctly and you have access to the cluster. You can also fetch the certificate manually and use the --cert flag.

Key Takeaways

• Kubernetes Secrets alone are not enough for a secure, scalable secret management solution.
• HashiCorp Vault provides a centralized, secure, and auditable way to manage secrets, with powerful features like dynamic secrets.
• The Vault Agent Injector is an elegant way to get secrets from Vault into your Kubernetes pods without modifying your application code.
• Sealed Secrets is a simple and effective tool for managing secrets in a GitOps workflow, allowing you to store encrypted secrets safely in your Git repositories.
• A hybrid approach, using Vault as the central source of truth and Sealed Secrets for GitOps workflows, can provide a robust and flexible secret management strategy.

The Path Forward

Alex leaned back in his chair, a sense of accomplishment washing over him. The incident had been a wake-up call, but it had also been a catalyst for building a much more secure and robust platform. He had a clear path forward for managing secrets at NovaCraft, one that combined the strengths of Vault, Sealed Secrets, and the External Secrets Operator.

But as he closed his laptop for the day, a new thought crept into his mind. Now that the platform was more secure, how could he make it more observable? How could he get deep insights into the performance of the applications running on the cluster? His next challenge was already taking shape: a deep dive into the world of observability with Prometheus and Grafana.

This is Part 5 of Season 2 of The Container Odyssey.