Global mutex lock contention

On-call incidentHard40 min

Your task

A Go in-memory rate-limiter service's CPU jumps from 25% to 95% at 13:30 and p99 doubles, yet useful throughput (allowed+denied decisions/sec) actually drops. perf/pprof shows most on-CPU time in runtime mutex/lock and futex, not in the business logic, and the OS shows a huge spike in context switches and run-queue length. A deploy at 13:25 added a single global sync.Mutex around a counter map that every request now contends on; before, sharding kept contention low. Request rate is only up ~5%. Triage and mitigate.

It covers

  • Lock contention
  • Context switching
  • False sharing