Lock contention on hot merchant

On-call incidentMedium30 minStripe style

Your task

Postgres. An order-processing service uses a per-merchant SELECT ... FOR UPDATE on a single merchant_balances row to serialize balance updates. After onboarding one very high-volume merchant, that merchant's order p99 hit 8 s and its workers spend most time waiting; other merchants are fine. pg_stat_activity shows many sessions on Lock / transactionid all targeting that one merchant's balance row. Throughput per worker is low. Triage and redesign so a single hot merchant doesn't serialize itself into the ground.

It covers

  • Advisory lock leak
  • Serialization
  • Single row hotspot