Question
A payouts service double-sends bank transfers intermittently — maybe 1 in 500. Dashboards: the worker calls the bank API, then writes a `payout_completed` row to Postgres; under load, the DB write occasionally times out AFTER the bank API already succeeded; the worker's retry logic then re-runs the whole task from the top, calling the bank API a second time. The bank API call has no idempotency key. CPU and DB connections are saturated during peak payout runs, which is when the timeouts cluster. How do you triage, stop the double-payouts immediately, and reconcile transfers already sent twice?
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.