Lock contention on hot row
Your task
A real-time ad-bidding service runs fine at 8k req/s but at a marketing event traffic ramps to 11k req/s and latency doesn't degrade gracefully, it falls off a cliff: median goes from 12ms to 900ms over about 90 seconds, then stays there even as you add pods. The DB shows enormous lock_wait time concentrated on UPDATEs to a single campaign_budgets row for the headline campaign that's getting 70% of the event traffic. CPU per pod actually drops as you scale out. Adding capacity made it worse. Triage and explain the dynamics.
It covers
- Hot row
- Latency cliff
- Serialization