Secret logged at scale

The idea

A secret is anything that grants access: an API key, a database password, a session token. Logs are meant for plain, replayable facts — not for credentials. The trouble is that a log line is not a single object you can quietly delete. It is a fact that fans out: to the app log on disk, to a central index you can search, to a security pipeline (SIEM), and to cold storage kept for months.

So a single line like log.info("request", req) — where req happens to carry an Authorization header — becomes a credential sitting in four places, readable by everyone with log access. Incident response is then a sequence: detect it, contain it (rotate the secret, scrub the lines, stop the bleed), and find the root cause (the statement that dumped a whole object).

See it work

How it works

The fix is structural, not a one-time scrub. You make secrets impossible to log by default, and you only ever log fields from a known allow-list. Never hand a whole request, header set, or arbitrary object to the logger — that is how secrets sneak in. Mark sensitive fields so any formatter redacts them, and run a secret scanner in CI so a leaking line never ships.

# A redaction filter applied at the logging boundary.
# Every record passes through it before it reaches any sink.

import logging, re

SENSITIVE_KEYS = {"authorization", "password", "api_key", "token", "set-cookie"}
TOKEN_RE = re.compile(r"(tok|sk|ghp)_[A-Za-z0-9]{6,}")

def mask(value: str) -> str:
    # keep a short prefix for correlation, drop the rest
    return value[:4] + "_****" if len(value) > 4 else "****"

class RedactFilter(logging.Filter):
    def filter(self, record: logging.LogRecord) -> bool:
        # 1) structured fields: redact by key name (allow-list everything else)
        for key in list(getattr(record, "fields", {})):
            if key.lower() in SENSITIVE_KEYS:
                record.fields[key] = "****"
        # 2) free-text safety net: mask anything shaped like a token
        if isinstance(record.msg, str):
            record.msg = TOKEN_RE.sub(lambda m: mask(m.group()), record.msg)
        return True  # never drop the line, only sanitise it

logging.getLogger().addFilter(RedactFilter())

# Good: only named, non-sensitive fields are logged.
log.info("request", fields={"method": "POST", "path": "/pay", "status": 200})

# Bad: dumps the whole request, headers and all -> Authorization leaks.
# log.info("request", req)

Two layers matter: the key allow-list handles structured logs (you log status, never headers), and the token regex is a safety net for free text and stack traces. Neither replaces rotation — once a value reaches a sink, treat it as compromised.

Signals

Watch out for

• Lines you can see are already replicated. By the time you notice, the log probably reached the central index, the SIEM, and cold storage. Scrubbing one sink leaves the others holding it.
• Rotate even after a clean scrub. Deleting the lines does not un-leak the value — it was readable while it lived in those sinks. The only safe assumption is that the secret is burned; issue a new one and revoke the old.
• Debug logging left on in production. DEBUG often dumps full payloads and headers. A flag flipped for one investigation can quietly log every credential for days.
• Logging full exception objects. Stack traces and repr(obj) happily print the connection string, the bearer token, or the request body that triggered the error.
• Confusing PII with a secret. Both deserve care, but a secret grants access and must be rotated; PII must be scrubbed and access-controlled. Rotation does nothing for a leaked email, and scrubbing alone does nothing for a leaked key.

Worked example

A payments engineer adds log.info("incoming", req) while chasing a flaky webhook, and ships it. req carries an Authorization: Bearer tok_9f3a…b2 header, so every webhook now writes the live partner token into the app log.

Detect. Twelve hours later the CI secret scanner flags tok_ in a sampled log export, and the key vendor sends an automated “exposed credential” alert. Two independent signals — this is real.

Contain. The on-call rotates the partner token first: the leaked value is revoked and a fresh tok_4c81… is issued, so anything copied out is now dead. They deploy the redaction filter so future lines show tok_****, then scrub the matched lines from the live log index. They confirm the value also sits in last night’s cold-storage snapshot — which is exactly why rotation, not scrubbing, was the move that mattered.

Root cause. The offending statement passed a whole request object to the logger. The lasting fix is a logging boundary that only emits allow-listed fields, plus a CI scanner that would have blocked the change. The postmortem notes that the value was exposed for twelve hours across four sinks, so rotation was non-negotiable regardless of how thorough the scrub looked.

Check yourself