Kubernetes Secrets: why Base64 is not encryption

echo cGFzc3dvcmQ= | base64 -d yields password. That’s all the protection a Kubernetes Secret offers on its own. Base64 is an encoding that lets arbitrary bytes fit into YAML, not encryption. That isn’t a flaw in Kubernetes but intentional: Secrets are a separate object type so you can treat them differently from a ConfigMap in terms of permissions. They’re only as well protected as those permissions.

Who can read a Kubernetes Secret?

• Anyone with `get` or `list` on Secrets in the namespace. That permission is included in many roles nobody looks at closely, such as edit.
• Anyone allowed to start a pod in the namespace. They simply mount the Secret into their pod and read it. Whoever may create pods can effectively read Secrets.
• Operators and controllers with cluster-wide permissions. A monitoring agent, a backup tool, a CI with an admin token: any of them can read all Secrets, and a leak at one of them is a leak of everything.
• Anyone who gets to etcd. Whether the data there is stored encrypted is up to whoever operates the control plane. With a managed offering, it’s worth asking.

Where do secrets leak?

The Secret object is rarely the actual gap. The values end up in other places along the way:

In the Git repository. A secret.yaml with Base64 values is checked in as plain text, just more awkward to read. Once it’s in the history, it stays there.

In Helm values. Helm stores the values of every revision in a release Secret in the namespace. A password that was in values.yaml sits there in every revision, even long after it was changed.

Directly in the pod template. A value that sits in the Deployment as env: value: instead of via a reference to a Secret is visible to anyone allowed to read Deployments, which is far more people than those allowed to read Secrets.

In CI logs and chat histories. An echo $DATABASE_URL for debugging, a connection string pasted into the team chat. Both outlive the key’s intended lifetime.

What actually protects secrets?

Keep permissions tight

Nobody needs read access to Secrets in their day-to-day work. People deploy; they don’t read passwords. Roles without get secrets, no cluster-wide admin tokens for tools, and whoever may create pods is chosen deliberately.

A secret manager as the source of truth

A secret manager such as Scaleway’s keeps values encrypted and versioned, with access controls and an audit trail. The application receives the value at startup, either because the deployment fetches it during rollout or because the External Secrets Operator syncs it into a Kubernetes Secret. To be honest: even then, the value ends up in a Secret in the namespace, because the pod needs it. What you gain is a single source that isn’t in Git, isn’t in Helm values and isn’t on laptops.

Not readable after saving

A secret that is never shown again after saving can’t be copied by accident. Whoever needs the value needs it in the application, not on the screen.

Rotation that actually lands

A key is only rotated once the application uses it. A process reads environment variables at startup. If you create a new version in the secret manager and don’t restart, you’ve only done half the job. Rotation therefore takes two steps: a new version, then a restart, plus an indication of which instance is still running with the old one.

A checklist

• No secret in the Git repository, not even Base64-encoded.
• No passwords in Helm values or in plain text in the pod template.
• People have no read access to Secrets in production.
• Every value has one source, ideally a secret manager.
• Every rotation ends with a restart, and someone checks that it happened.
• The audit log shows who changed a secret, never the value.

An example: rotating the Stripe key

Here’s what a rotation that actually lands looks like, using a payment provider’s key as an example:

1. Generate a new key. Create a new key with the provider. Many providers, Stripe included, keep the old one valid for a period you choose. That’s your window for the switch.
2. New version in the secret manager. Save the value there as a new version, not in the chat, not in a file.
3. Restart. Restart the application so it reads the new value. Check that all instances have restarted, not just one.
4. Verify. A test payment, a look at the logs: no authentication errors.
5. Revoke the old key. Only now, and then for real.

The most common mistake happens between steps 2 and 3: the new version is saved, but nobody restarts. Two weeks later the old key expires, and payments collapse on a Saturday evening.

External Secrets Operator: when it’s worth it

The External Secrets Operator runs in the cluster and syncs values from a secret manager into Kubernetes Secrets. It’s worth it when many applications need the same values, when values change frequently or when your security policy requires values to be resolved only inside the cluster. The price: another controller with permissions on the secret manager that needs to be maintained, updated and monitored. For a few applications with infrequent changes, fetching the values from the secret manager during rollout is enough.

Small steps for this week

If you can’t rebuild everything at once, start like this: search the Git repository for Base64 blocks and passwords, and rotate whatever you find. Check who is allowed to get secrets in the cluster, and shorten the list. And for the next new key, use the secret manager right away instead of an environment variable in the manifest.

How Clusterward handles this

In Clusterward, a secret can no longer be read after saving, not even by administrators, and is stored AES-256-GCM-encrypted or in Scaleway Secret Manager, in readable folders per application, environment and service. Helm charts receive secrets via placeholders instead of having them in plain text in the maintained values. Every ten minutes, Clusterward checks whether a secret has received a new version externally, shows which instance is still using the old one and, if you want, restarts automatically. Delivery via the External Secrets Operator can be chosen per cluster. The audit log only lists names, never values. More in Secrets; what of this an auditor sees as evidence in Audit & NIS2 and in the post NIS2 in Kubernetes operations.

Conclusion

Base64 protects nothing; permissions do. Keep values out of Git, Helm values and pod templates, give people no read access and make a secret manager the single source. And rotate in a way that the new version actually reaches the application.

Where are your secrets today? Tell us how your applications get their passwords and keys. We’ll show you where they can be read by others and what a move to a secret manager looks like. Ask a question →

Sources and further reading

• Kubernetes: Secrets
• Helm: Storage backends – release data in Secrets
• Scaleway: Secret Manager
• Stripe: Rotate an API key