Transfer endpoint deadlocks after deploy

On-call incidentMedium30 minStripe style

Your task

Since this morning's deploy, a small but steady stream of requests on the 'transfer between accounts' endpoint fail with ERROR: deadlock detected, and the error rate scales with concurrency, it's worst at peak. The transfer logic debits one account and credits another within a single transaction, updating two accounts rows. Before the deploy there were essentially no deadlocks. The DB is otherwise healthy. Triage and mitigate.

It covers

  • Deadlock
  • Ordering
  • Retry