Making a write safe to retry

The idea

When a write times out, two very different worlds look identical from the client: the request never arrived, or it arrived, worked, and the response got lost on the way home. The caller cannot tell them apart, so it retries — and if the server has no memory of the first attempt, it happily charges the card again.

An idempotency key is the caller's way of saying “this is the same intent as before, not a new one.” The server records the key the first time, along with what it decided, and on any repeat it replays that stored answer instead of doing the work twice.

That is the whole trick. Everything else is the fine print of the contract: what a matching key owes you, what a changed body under the same key means, and how long the memory lasts.

Watch a retry with and without a key

How it works

1. The caller mints the key once, at the moment of intent. One key per business action — one per tap of the pay button — not one per HTTP attempt. A UUID is fine; store it with the cart so every retry finds the same one.
2. It rides in a header: Idempotency-Key: idem_9f2c. Keys are scoped per account or API key, never global.
3. The server claims the key atomically before doing work — a unique index on (account_id, key) and an insert that either wins or tells you someone else got there first. This is what makes two simultaneous retries safe.
4. It stores a fingerprint of the request next to the key: a hash of method, path and body. That is what lets it later tell “same intent” from “different intent, recycled key”.
5. It does the work, then writes the status code and response body into that same row — ideally committing the effect and the record in one transaction, so a crash can never leave a charge with no memory of it.
6. On a repeat, four cases and four answers. Match and finished → replay the stored response. Match but still running → 409, come back shortly. Key found, fingerprint differs → 422, and change nothing. No row → treat as brand new.
7. Rows expire. 24 hours is the common window. After that the key is meaningless and the same request is a new one — which is why the retry loop must live inside the window.

# first attempt
POST /v1/charges
Idempotency-Key: idem_9f2c
{ "amount": 4200, "currency": "usd", "source": "tok_x1" }

-> 201 Created  { "id": "ch_a1", "amount": 4200, "status": "captured" }
   …response lost in transit. ledger: 1 charge, $42.00

# retry, same key, same body  (seconds later)
-> 200 OK       { "id": "ch_a1", ... }   Idempotent-Replay: true
   ledger: 1 charge, $42.00      no card touched

# retry, same key, body changed to 5200
-> 422          { "error": "key idem_9f2c was used with a different payload" }
   ledger: 1 charge, $42.00      nothing changed

# retry, same key, 25 h later — past the 24 h ttl
-> 201 Created  { "id": "ch_a2", "amount": 4200 }
   ledger: 2 charges, $84.00     the memory was already swept

# two retries land at the same instant
-> 201 for the insert that won the unique index
-> 409 for the loser: "a request with this key is in progress"

Which verbs are idempotent for free. GET and HEAD are safe — they change nothing. PUT is idempotent because it replaces the whole resource with the body you sent: send it ten times, the final state is the same. POST and PATCH are not: POST means “make another one”, and a PATCH like {"balance": "+10"} compounds.

DELETE is the interesting argument. It is idempotent in effect — after one call or ten, the resource is gone — but not in response: you get 204 the first time and 404 after. That is fine, because idempotency is a property of server state, not of the status code. The real trap is a DELETE that does something on the way out (issues a refund, cancels a subscription, fires webhooks). Those side effects are ordinary writes and need the same protection as any POST.

When to use it

• POST that moves money or creates a record do require a caller-supplied key; persist the response for the TTL and replay it verbatim. trade-off a key store to write, index and expire, plus one extra lookup on every write.
• PUT of a full representation do nothing extra — retries converge on the same state. trade-off you must send every field; “partial PUTs” quietly lose the guarantee.
• DELETE of a resource do retry freely, and treat 404 on a retry of your own delete as success. trade-off the status code differs between calls; cascades and refunds still need a key.
• PATCH and relative updates do add a key, or restate the change absolutely (balance = 110, not +10). trade-off absolute writes need a version or ETag to avoid clobbering a concurrent edit.
• At-least-once consumers: queues, webhooks do dedupe on the producer's event id — the same contract, different name. trade-off a dedupe table with its own retention, and a decision about what to do past it.
• High-volume, low-stakes writes (analytics pings) do consider skipping it; a duplicate row may cost less than a key store on the hot path. trade-off your counts are approximate, so say so downstream.

Watch out for

• Minting the key inside the retry loop. If the helper calls randomUUID() per attempt, every retry gets a fresh identity and the server correctly creates a fresh charge. The key must be created once with the intent and survive the retries — including an app restart, if the retry does.
• Recording the key but not the response. Answering the retry with a bare 200, an empty body, or “duplicate request” leaves the caller unable to learn the charge id. The contract is same key returns the original result, so store the status and body and hand them back.
• No body fingerprint. Without one, a client that retries with a changed amount either gets silently told $42.00 succeeded when it asked for $52.00, or gets charged twice. Hash method, path and body; on mismatch return 422 and change nothing.
• Check-then-act under concurrency. Two retries arriving 3 ms apart both read “no row”, both charge. Claim the key with a unique constraint or a conditional insert, and return 409 to whoever loses — a duplicate charge is much worse than a caller waiting 200 ms.
• Forgetting that keys expire, and that side effects don't. A retry after the 24-hour window is genuinely a new request — document the window and keep the retry budget well inside it. And if the first attempt committed the charge but died before its email and webhook, the replay must not re-fire them blindly; give those their own dedupe or reconcile them from the stored row.

Worked example

Support escalates three charges of $42.00 on one customer, 8 seconds apart, all 201 Created — ch_a1, ch_a2, ch_a3, $126.00 in total. The mobile client has an 8-second timeout; the acquirer call sits at a p99 of 11 seconds. So the first attempt succeeded, the client gave up before the response landed, and the retry helper fired twice more.

The logs show three distinct Idempotency-Key values, which is the bug in one line: the key was generated inside withRetries(), so each attempt announced itself as a new order. The fix is two-sided. On the client, mint the key when the user taps pay, persist it beside the cart, and reuse it for every attempt of that intent. On the server, add a unique index on (account_id, idempotency_key), store the request fingerprint plus status code, response body and expires_at = now() + 24h, commit that row in the same transaction as the charge, and answer repeats with replay, 409 or 422 as the case demands.

Worth saying out loud in an interview: this does not make retries free. It makes the window safe. A user who taps pay again tomorrow is a new intent with a new key, and charging them again is then the correct answer.

Check yourself