Order latency spikes on sale

On-call incidentMedium30 minShopify style

Your task

During a flash on-sale at 10:00 your ticketing service (Java + Hibernate on Postgres) starts throwing intermittent 'deadlock detected' and 'could not serialize access' errors, and order success rate drops to 70%. The hottest SKU is a single popular event row whose available_count column every purchase decrements with SELECT ... FOR UPDATE then UPDATE. Throughput is fine for unpopular events. CPU and connections are healthy. How do you triage and mitigate during the live event, and fix it afterward?

It covers

  • Lock contention
  • Deadlock
  • Hot row