Secrets management

A database password baked into source code is a password everyone with the repo already knows.

The idea

A hardcoded secret — an API key, a database password, a token — written into code, config, or a committed .env file feels convenient. But it leaks everywhere the code travels: into git history (forever, plus every fork and clone), into build images, into log lines and crash dumps. Worse, you can't rotate it without editing code and redeploying every service that holds a copy.

Secrets management flips this. A secrets manager — Vault, a cloud KMS, AWS Secrets Manager — holds the real secret centrally. At boot the app proves its own identity, gets a short-lived credential over an authenticated channel, scoped to just what it needs, audited on the way out, and rotatable from one place. The secret never touches the repo or the image.

Press play to watch a hardcoded secret spread, then the same app fetch a short-lived one from a vault.

How it works

The failure mode is a secret written down where the code lives. Once it's in a commit it's in history forever — deleting the line later doesn't remove it, and every clone and fork keeps a copy. The fix is to never write the secret down: fetch it at runtime from a vault, scoped to this workload's identity, with a short time-to-live so a leaked copy expires on its own.

import hvac  # vault client

# WRONG — baked into the repo, can't rotate, lives in git history forever:
# DB_PASSWORD = "hunter2-prod"

# RIGHT — fetch at runtime over an authenticated channel; never commit it.
client = hvac.Client(url="https://vault.internal")
client.auth.approle.login(role_id=WORKLOAD_ROLE)   # prove this app's identity

lease = client.secrets.database.generate_credentials(name="app-db")
db_password = lease["data"]["password"]            # short-lived, scoped, audited
ttl = lease["lease_duration"]                       # e.g. 3600s — auto-rotates on expiry

# Use it in memory only. Don't log it, don't write it to disk, refresh before TTL.

Trade-offs

Concern	Hardcoded secret	Secrets manager
Rotation	Edit code + redeploy every service	Central, automatic on TTL expiry
Blast radius if leaked	Valid until someone notices and redeploys	Dead within the lease window
Audit trail	None — who read the repo?	Every issue logged by identity
Runtime dependency	None — but that's the problem	App needs the vault reachable at boot
Secret-zero bootstrap	No bootstrap — secret is just there	Still need one credential to authenticate

Watch out for

Secret-zero / bootstrap problem. The app needs some credential to authenticate to the vault. Anchor it to platform identity (cloud IAM role, Kubernetes service account) so there's no static secret to bootstrap, rather than another hardcoded token.
Secrets leaking into logs. A vault fetch is wasted if the value lands in a log line, an error message, or an environment dump on a crash page. Mark it sensitive and never print it.
Long-lived static creds even with a vault. Storing a secret in the manager but never expiring it just moves the problem. Use short TTLs and dynamic, per-lease credentials so a leak self-destructs.
.env committed to git or baked into a Docker layer. A deleted .env still lives in history and forks; a secret added in one image layer persists in that layer even if a later layer removes it. Keep secrets out of the build entirely.
Over-broad policies. One identity that can read every secret turns any single compromise into a full breach. Scope each policy to exactly the secrets that workload needs.

Worked example

A developer commits a Postgres password into a public repo for a quick fix. They notice an hour later and delete the line — but it's already in git history, and three people have forked the project. The password is now permanently retrievable by anyone, and it stays valid. Rotating it means generating a new password and redeploying the 8 services that hold the old one, coordinated so nothing breaks mid-rollout. That's hours of work, and the old credential is live the whole time.

Now run the same leak with a vault. The app holds no password — it fetches a lease with a 60-minute TTL. If that leased credential leaks, it's dead within the hour with no human in the loop, because the vault rotates the underlying secret on expiry. Same mistake, but the blast radius shrinks from "valid forever across forks" to "valid for under an hour, audited, scoped to one service."

Check yourself

You find a prod API key hardcoded in a commit that already merged to main. What's the first move?

Coach note: once a secret has been pushed, treat it as already public — cleaning up the repo is good hygiene, but it's the rotation that actually closes the exposure.