Write amplification spikes

On-call incidentHard35 minMeta style

Your task

You run a RocksDB-backed metadata service (LSM-tree storage engine) across 12 nodes. After a config change last night that raised target_file_size_base and lowered level0_file_num_compaction_trigger, p99 write latency tripled and NVMe write bandwidth per node climbed from ~150 MB/s to ~1.1 GB/s even though the application's logical write rate is flat. Dashboards: bytes-written-to-disk / bytes-written-by-app (write amplification factor) jumped from ~9x to ~34x; pending-compaction-bytes is steady (not backing up); SSD wear (TBW) is now projected to exhaust endurance in 14 months instead of 6 years. CPU is up on compaction threads. No errors, no rejected writes, just brutal amplification. How do you triage, find root cause, and fix?

It covers

  • Write amplification
  • Lsm tree
  • Compaction