Alex, a seasoned backend engineer with a decade of experience building and scaling monolithic applications, stared at the screen. The warm glow of the monitor illuminated a face etched with a mixture of pride and a nagging sense of unease. He had just successfully deployed his first microservice at NovaCraft, a fast-growing startup that was all-in on Kubernetes. The service was simple, a small API that returned a list of products. But it was a start. A small victory in his personal odyssey to master this new world of containers and orchestration.

His moment of triumph was short-lived. A message from Sarah, the lead architect, popped up on his screen. “Hey Alex, great job on getting the product service up. I was just looking at the code. We need to talk about the database connection string.”

Alex’s heart sank. He knew exactly what she was talking about. In his haste to get the service running, he had hardcoded the database URL directly into the application code. It was a rookie mistake, a shortcut he would never have taken in his previous role. But in the whirlwind of learning Docker, Kubernetes, and a dozen other new technologies, he had let it slip.

“I know, I know,” he typed back, a flush of embarrassment creeping up his neck. “It’s on my list to fix. I was just focused on getting it deployed first.”

“No worries,” Sarah replied, her tone reassuring. “We’ve all been there. But now is the perfect time to learn about the Kubernetes way of managing configuration. Let’s talk about ConfigMaps and Secrets. It’s time to solve the config puzzle.”

Alex leaned back in his chair, a sense of relief washing over him. This was what he had been missing. Not just the ‘what’ of Kubernetes, but the ‘how’. The best practices, the patterns, the idiomatic way of building and deploying applications. He was ready to learn. He was ready to solve the puzzle.

The Config Puzzle: Externalizing Configuration

At the heart of the issue Alex is facing is a fundamental principle of building robust and scalable applications: never store configuration in your application code. Hardcoding values like database URLs, API keys, or feature flags makes your application brittle and difficult to manage. Every time a value changes, you have to rebuild and redeploy your application. This is not just inefficient; it’s a recipe for disaster in a dynamic environment like Kubernetes.

Think of your application as a chef in a busy kitchen. The chef knows how to cook the dishes (the application logic), but they don’t store the recipes in their head. Instead, they have a recipe book. When they need to cook a dish, they refer to the recipe book for the ingredients and instructions. If a recipe changes, you don’t need to retrain the chef. you just update the recipe book.

In Kubernetes, ConfigMaps are that recipe book. They are a dedicated object for storing non-sensitive configuration data as key-value pairs. By externalizing your configuration into ConfigMaps, you decouple your application from its environment. This makes your application more portable, easier to manage, and more scalable.

The Secret Ingredient: Managing Sensitive Data

While ConfigMaps are great for storing non-sensitive configuration data, what about sensitive information like passwords, API tokens, and TLS certificates? This is where Secrets come in. Secrets are another Kubernetes object, similar to ConfigMaps, but they are specifically designed to hold sensitive data.

So, what’s the difference between a ConfigMap and a Secret? The primary difference is that the data in a Secret is stored in a base64-encoded format. This is not encryption, and it’s crucial to understand this distinction. Base64 is an encoding scheme that represents binary data in an ASCII string format. It’s easily reversible. Anyone with access to the Secret can easily decode the data.

Think of it like writing a message in a simple substitution cipher, like A=1, B=2, etc. It might look like gibberish to a casual observer, but anyone who knows the key can easily decipher it. Base64 is even simpler than that, as the “key” is public knowledge.

So, if Secrets are not encrypted, what’s the point of using them? The primary purpose of Secrets is to separate sensitive data from your application code and to control who has access to it. By using Kubernetes’ Role-Based Access Control (RBAC) system, you can restrict which users and service accounts can read or write to a particular Secret. This is a significant improvement over storing sensitive data in a ConfigMap, which is typically more widely accessible.

Furthermore, Kubernetes provides several mechanisms for consuming Secrets in your Pods, such as mounting them as files or exposing them as environment variables. When a Secret is mounted as a file, it is stored in an in-memory filesystem (tmpfs) on the node, which means it is never written to disk. This provides an additional layer of security.

Consuming Configuration: Files vs. Environment Variables

Now that we understand what ConfigMaps and Secrets are, the next question is: how do we get that data into our application? Kubernetes provides two primary mechanisms for this: mounting them as files or exposing them as environment variables.

The Environment Variable Approach

Exposing configuration data as environment variables is a common and straightforward approach. Most programming languages and frameworks are designed to read configuration from environment variables, so it often requires minimal code changes to adopt this pattern.

In your Pod definition, you can use the envFrom field to populate environment variables from a ConfigMap or a Secret. For example, to consume all the key-value pairs from a ConfigMap named my-config, you would add the following to your container definition:

envFrom:
- configMapRef:
    name: my-config

Similarly, to consume data from a Secret named my-secret, you would use:

envFrom:
- secretRef:
    name: my-secret

While this approach is simple, it has a significant drawback: environment variables are not updated automatically. If you update the ConfigMap or Secret, the environment variables in your running Pods will not be updated. You will need to restart the Pods to pick up the new values. This can be a major limitation in dynamic environments where you need to change configuration on the fly.

The Volume Mount Approach

A more flexible and powerful approach is to mount ConfigMaps and Secrets as volumes. When you mount a ConfigMap or a Secret as a volume, the data is exposed as files in a directory inside your container. Each key in the ConfigMap or Secret becomes a file in the directory, and the value becomes the content of the file.

For example, to mount a ConfigMap named my-config to a directory named /etc/config, you would add the following to your Pod definition:

volumes:
- name: config-volume
  configMap:
    name: my-config
... // in your container definition
volumeMounts:
- name: config-volume
  mountPath: /etc/config

The real power of this approach lies in the fact that mounted ConfigMaps and Secrets are updated automatically. When you update the ConfigMap or Secret, the files in the mounted volume are updated within a short period. Your application can then detect the change and reload its configuration without requiring a restart. This is a huge advantage for building dynamic and resilient applications.

However, this approach requires your application to be able to read configuration from files and to be able to detect changes and reload its configuration. This might require some additional code, but the benefits are well worth the effort.

Immutable ConfigMaps and Secrets

In a dynamic environment like Kubernetes, it’s often desirable to prevent accidental or unwanted changes to your configuration. This is where immutable ConfigMaps and Secrets come in. By marking a ConfigMap or a Secret as immutable, you can prevent any changes to its data. This can be a powerful tool for building more predictable and stable systems.

To create an immutable ConfigMap or Secret, you simply set the immutable field to true in the object’s definition. For example:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  my-key: my-value
immutable: true

Once an immutable ConfigMap or Secret is created, you cannot change its data. If you need to update the configuration, you will need to create a new ConfigMap or Secret and update your Pods to use the new object. This might seem like a limitation, but it can be a desirable feature in many scenarios. For example, if you are using a GitOps workflow to manage your configuration, you can create a new ConfigMap or Secret for each change and then update your application’s deployment to use the new object. This provides a clear audit trail of all configuration changes and makes it easy to roll back to a previous version if something goes wrong.

Hands-On: Refactoring the API

Now it’s time to put our knowledge into practice. We’re going to refactor Alex’s product service to use a ConfigMap for the database URL and a Secret for the API key. We’ll start by creating a simple Flask application that reads its configuration from a file. Then, we’ll create a ConfigMap and a Secret, and we’ll mount them as a volume in our Pod. Finally, we’ll verify that our application is using the configuration from the ConfigMap and the Secret.

The Application

First, let’s create a simple Flask application. Create a directory for our project and inside it, a file named app.py with the following content:

from flask import Flask, jsonify
import os


app = Flask(__name__)


CONFIG_FILE = '/etc/config/config.properties'


def get_config():
    config = {}
    if os.path.exists(CONFIG_FILE):
        with open(CONFIG_FILE, 'r') as f:
            for line in f:
                if '=' in line:
                    key, value = line.strip().split('=', 1)
                    config[key] = value
    return config


@app.route('/')
def index():
    config = get_config()
    db_url = config.get('DB_URL', 'No DB URL configured')
    api_key = config.get('API_KEY', 'No API Key configured')
    return jsonify(message='Hello from the Product Service!', db_url=db_url, api_key=api_key)


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

This application reads its configuration from a file located at /etc/config/config.properties. It then exposes a single endpoint that returns a JSON object containing a welcome message, the database URL, and the API key.

Next, create a requirements.txt file:

Flask

And a Dockerfile:

FROM python:3.9-slim


WORKDIR /app


COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt


COPY . .


CMD ["python", "app.py"]

The ConfigMap

Now, let’s create a ConfigMap to store our database URL. Create a file named configmap.yaml with the following content:

apiVersion: v1
kind: ConfigMap
metadata:
  name: product-service-config
data:
  config.properties: |
    DB_URL=postgres://user:password@host:port/db

This ConfigMap contains a single key, config.properties, which contains the database URL.

Create the ConfigMap in your Kubernetes cluster:

Terminal window
kubectl apply -f configmap.yaml

The Secret

Next, let’s create a Secret to store our API key. Create a file named secret.yaml with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: product-service-secret
type: Opaque
data:
  API_KEY: c2VjcmV0X2FwaV9rZXk=

This Secret contains a single key, API_KEY, which contains the base64-encoded value of secret_api_key.

Create the Secret in your Kubernetes cluster:

Terminal window
kubectl apply -f secret.yaml

The Deployment

Now, let’s create a Deployment that uses our ConfigMap and Secret. Create a file named deployment.yaml with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: product-service
  labels:
    app: product-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: product-service
  template:
    metadata:
      labels:
        app: product-service
    spec:
      containers:
      - name: product-service
        image: <your-docker-hub-username>/product-service:v1
        ports:
        - containerPort: 8080
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
      volumes:
      - name: config-volume
        projected:
          sources:
          - configMap:
              name: product-service-config
          - secret:
              name: product-service-secret

This Deployment creates a single Pod running our product service. It also creates a projected volume that combines our ConfigMap and Secret into a single directory. This directory is then mounted at /etc/config in our container.

Before you apply this, make sure to build and push the docker image to your docker hub account.

Terminal window
# Build the image
docker build -t <your-docker-hub-username>/product-service:v1 .


# Push the image
docker push <your-docker-hub-username>/product-service:v1

Now, create the Deployment:

Terminal window
kubectl apply -f deployment.yaml

Verifying the Configuration

Now that our application is running, let’s verify that it’s using the configuration from our ConfigMap and Secret. First, let’s get the name of our Pod:

Terminal window
kubectl get pods

Then, let’s port-forward to our Pod:

Terminal window
kubectl port-forward <pod-name> 8080:8080

Now, in a new terminal, let’s curl our application:

Terminal window
curl http://localhost:8080

You should see the following output:

{
  "api_key": "c2VjcmV0X2FwaV9rZXk=",
  "db_url": "postgres://user:password@host:port/db",
  "message": "Hello from the Product Service!"
}

As you can see, our application is now using the database URL from our ConfigMap and the API key from our Secret. We have successfully decoupled our configuration from our application code!

Rotating a Secret

One of the benefits of using volumes to mount Secrets is that they are updated automatically. Let’s see this in action. First, let’s update our Secret with a new API key. Let’s say our new API key is new_secret_api_key. First we need to encode it to base64.

Terminal window
echo -n 'new_secret_api_key' | base64

This will output bmV3X3NlY3JldF9hcGlfa2V5. Now, let’s update our secret.yaml file:

apiVersion: v1
kind: Secret
metadata:
  name: product-service-secret
type: Opaque
data:
  API_KEY: bmV3X3NlY3JldF9hcGlfa2V5

Now, apply the change:

Terminal window
kubectl apply -f secret.yaml

Now, if you wait a few seconds and curl your application again, you should see the new API key:

{
  "api_key": "bmV3X3NlY3JldF9hcGlfa2V5",
  "db_url": "postgres://user:password@host:port/db",
  "message": "Hello from the Product Service!"
}

As you can see, our application picked up the new API key without a restart. This is a powerful feature that can help you build more dynamic and resilient applications.

Debugging and Troubleshooting

As with any new technology, you’re bound to run into a few bumps in the road. When working with ConfigMaps and Secrets, a common issue is a Pod failing to start with a CreateContainerConfigError. This error often indicates that the ConfigMap or Secret you are referencing in your Pod definition does not exist. Double-check that you have created the ConfigMap and Secret, and that the names match what you have specified in your Pod definition. Another frequent problem is the application not picking up the configuration. If you are mounting the ConfigMap or Secret as a volume, make sure that the mountPath in your Pod definition is correct, and that your application is reading the files from the correct directory. If you are using environment variables, remember that they are not updated automatically. You will need to restart your Pods to pick up the new values. Sometimes, you might have updated your ConfigMap or Secret, but your application is not seeing the changes. If you are mounting the ConfigMap or Secret as a volume, it can take a few moments for the changes to be propagated to the Pod. If you are still not seeing the changes after a minute or two, you can try deleting and recreating the Pod to force a refresh. Lastly, a permission denied error when your application tries to read the configuration files can happen if the user that your application is running as does not have permission to read the files in the mounted volume. You can use a securityContext in your Pod definition to specify the user and group that your application should run as.

Key Takeaways

In this chapter, we’ve explored the Kubernetes way of managing configuration. The most important principle to remember is to externalize your configuration. Never store configuration in your application code. Instead, use ConfigMaps and Secrets to decouple your application from its environment. For non-sensitive data, ConfigMaps are the standard choice. For sensitive data like passwords and API keys, use Secrets. While the data is only base64 encoded, not encrypted, Secrets provide a way to control access to sensitive data using RBAC. When it comes to consuming configuration, prefer mounting as volumes over environment variables. Mounting ConfigMaps and Secrets as volumes is more flexible and powerful than using environment variables because mounted volumes are updated automatically. This allows you to build more dynamic and resilient applications. Finally, for increased stability, use immutable ConfigMaps and Secrets. Immutable ConfigMaps and Secrets can help you build more predictable and stable systems by preventing accidental or unwanted changes to your configuration.

The Puzzle Solved, A New Challenge Awaits

Alex leaned back, a genuine smile on his face this time. The product service was humming along, its configuration neatly tucked away in a ConfigMap and a Secret. He had not only fixed his initial mistake but had also gained a much deeper understanding of how to manage configuration in Kubernetes. He felt a sense of accomplishment, of having solved a complex puzzle and emerged victorious.

His thoughts were interrupted by another message from Sarah. “Nice work, Alex. The service is looking much better now. Ready for your next challenge?”

Alex grinned. “Bring it on.”

“Great,” Sarah replied. “Now that you have the configuration sorted, let’s talk about how to handle persistent data. Your service is stateless right now, but what happens when you need to store data that needs to survive a pod restart? Let’s dive into the world of persistent volumes and stateful sets.”

Alex’s grin widened. A new puzzle. A new challenge. He couldn’t wait.