False sharing cache coherence

On-call incidentHard40 minCitadel style

Your task

A low-latency market-data fan-out service regressed after a refactor that replaced a single global counter with a small array of per-shard counters 'to reduce contention.' Counterintuitively, throughput dropped ~30% and p99 got worse under high core counts, even though each thread now writes only to its own array slot and there's no lock. CPU is high but instructions-per-cycle dropped sharply; perf counters show a large rise in L2/L3 cache coherence traffic and 'HITM' (modified-cache-line) events. No allocation, no GC, no lock contention. How do you triage and fix?

It covers

  • False sharing
  • Cache line
  • Concurrency