Lock contention limits throughput

On-call incidentHard30 minShopify and Amazon style

Your task

Your inventory-reservation service scaled its stateless workers from 10 to 50 to handle a sale, but throughput on the hot SKUs plateaued and latency rose. Every reservation takes a per-SKU distributed lock (Redis Redlock-style) to prevent overselling. Tracing shows workers spending most of their time waiting to acquire the lock for a handful of popular SKUs, and lock wait time grew roughly linearly as you added workers. CPU is low everywhere. How do you triage and what's the durable fix?

It covers

  • Distributed lock
  • Lock contention
  • Serialization