Synthetic Industry

Troubleshooting guide · updated 2026-10-11

Secrets in an env file? Protected alternatives on Compose, systemd and Kubernetes, and the rotation order

Why environment variables are a weak place for secrets, how Compose secrets, systemd credentials and Kubernetes Secrets differ, and a rotation order that does not take the app offline.

Why an env file is the weak option

An env file is convenient and that is its problem. It is copied between machines, pasted into chats, committed by accident and read by whoever can read the directory. Docker's documentation notes that secrets passed as environment variables risk accidental exposure: the variables are often visible to every process in the container, and they can be printed in logs when debugging an error, without anyone intending it. A file with restricted permissions, delivered to only the service that needs it, narrows who can read the value. It does not make the secret safe from someone with root on the host; it reduces how easily it leaks.

  • Env var: inherited by child processes and easy to print.
  • File with permissions: readable by one user or service.
  • Neither protects against a compromised host.

Compose, systemd and Kubernetes each have a mechanism

Compose secrets come from a file or an environment variable on the host and are mounted as files under a secrets directory inside the container, but only for services that list them. Some official images accept variables whose names end in _FILE and read the secret from the path they point to. systemd credentials, available from systemd 247, are loaded by the unit and appear as files readable only by the service user; on an older systemd you would use a file with tight permissions instead, and the documentation notes that embedding data in the unit file is for non-sensitive data only, because unit files are world-readable. Kubernetes Secrets are objects consumed as mounted files or environment variables. By default they are stored unencrypted in the cluster's data store, and base64 only obscures them, so encryption at rest and tight access rules are the cluster administrator's job.

  • Compose: top-level secrets plus per-service grants.
  • systemd: LoadCredential reads a protected file for the unit.
  • Kubernetes: mounted volume preferred over environment variables for updates.

The application has to read from a file

The platform can deliver a file, but the program still has to read it. Check whether the app or image supports a file-based setting, usually a path variable. If it does not, the options are a small wrapper that reads the file and exports the value just before starting the process, or a code change. Both are modest but real work, and they must not end up printing the value. Test with throwaway values: the app should start and pass its health check using them, and fail when the file is missing. A test that passes with an empty file proves nothing.

  • Look for a *_FILE style setting before writing a wrapper.
  • A health check that needs the secret proves it was read.
  • Test the missing-file case too.

A rotation order that does not break the app

Rotation means replacing a value because it may have leaked or because time has passed. For a planned rotation the safe order is: find every system that uses the key, create the new value at the issuer, place it where the app reads it, restart or reload the app, confirm it works with the new value, and only then revoke the old one. Revoking first takes the app offline. A key you know or suspect is exposed is different: keep the overlap as short as you can, and revoke the old value as soon as the new one is confirmed, accepting a brief interruption if that is the price of closing the exposure. If it was committed to a repository, treat it as exposed and rotate it first; removing it from history is a separate job. A Kubernetes detail: a mounted Secret volume is eventually updated after the Secret changes, but a subPath mount does not receive updates, so you may need to restart the pods. You generate and hold the keys; nobody else needs to see them.

  • List every consumer of each key first.
  • Planned rotation: keep the old value valid until the new one is confirmed.
  • Exposed key: shorten the overlap and revoke promptly once the new value works.
  • Restart where the platform does not reload on change.

What the fixed job does and does not do

The secrets job moves one app's settings to the platform's file-based mechanism using throwaway values, removes the real values from the env file and the build context, searches the delivered files for any value, checks file permissions and hands over the rotation checklist. It never receives your real keys, and it does not rewrite a repository's history or decide which exposed keys must be revoked. If a key has already been committed, treat it as exposed and rotate it first.

Sources and limits

  • Docker: Compose secrets Checked 2026-10-11.
    • Compose secrets are mounted as files in the container under /run/secrets, and each service must be granted access.
    • Environment variables used for secrets risk exposure to every process and appearing in logs when debugging.
    • Some images accept variables ending in _FILE that point to the mounted path.
  • systemd: service credentials Checked 2026-10-11.
    • Credentials are files delivered to one service, readable only by the service user, and are intended to replace environment variables and plain files for secrets.
    • SetCredential embeds data in the unit file, which is world-readable, so it is only for non-sensitive data.
  • Kubernetes: Secrets Checked 2026-10-11.
    • Secrets are stored unencrypted in the API server's data store by default; anyone who can create a pod in a namespace can read the Secrets there.
    • Base64 obscures values but is not confidentiality.
    • Mounted Secret volumes are updated eventually; a subPath mount does not receive updates.
  • systemd 247 release notes Checked 2026-10-11.
    • systemd 247 added the credentials logic and the SetCredential= and LoadCredential= unit settings.