Bank API timeout duplicate transfer

On-call incidentHard40 minStripe style

Your task

A payouts service double-sends bank transfers intermittently, maybe 1 in 500. Dashboards: the worker calls the bank API, then writes a payout_completed row to Postgres; under load, the DB write occasionally times out AFTER the bank API already succeeded; the worker's retry logic then re-runs the whole task from the top, calling the bank API a second time. The bank API call has no idempotency key. CPU and DB connections are saturated during peak payout runs, which is when the timeouts cluster. How do you triage, stop the double-payouts immediately, and reconcile transfers already sent twice?

It covers

  • Retry after success
  • Non atomic side effect
  • Exactly once
  • Payments