Mutex contention plateaus throughput

On-call incidentHard40 min

Your task

A Go in-memory metrics-aggregation service can't keep up at 16 cores: throughput plateaus around the same number whether you give it 4 cores or 16, and per-request latency rises with load. A CPU profile shows a large fraction of time in sync.(*Mutex).Lock/runtime futex; the mutex protects a single global map[string]*counter that every request reads-modify-writes under one lock. GOMAXPROCS is set correctly and the box isn't CPU-starved overall. Adding cores didn't help throughput. Triage and explain why more cores don't help.

It covers

  • Hot mutex
  • Global lock
  • Serialization bottleneck