Compaction throughput throttled

On-call incidentHard35 min

Your task

A Cassandra cluster serving a write-heavy time-series workload starts rejecting and stalling writes. Symptom: client p99 write latency spikes from 8ms to 9s, with intermittent WriteTimeoutExceptions. Dashboards: pending compactions on three nodes have climbed from near-zero to 4,800 and are still rising; SSTable count per table is in the tens of thousands; disk is only 61% full so it's not a space problem; the compaction throughput throttle (compaction_throughput_mb_per_sec) is pinned at its configured ceiling; one node shows a single STCS compaction that's been running for 9 hours on a 600 GB SSTable. A bulk backfill job was started yesterday afternoon. How do you triage and mitigate?

It covers

  • Compaction
  • Write stall
  • Backpressure