Write amplification spikes
You run a RocksDB-backed metadata service (LSM-tree storage engine) across 12 nodes. After a config change last night that raised `target_file_size_base` and lowered `level0_file_num_compaction_trigger`, p99 write latency tripled and NVMe write bandwidth per node climbed from ~150 MB/s to ~1.1 GB/s even though the application's logical write rate is flat. Dashboards: bytes-written-to-disk / bytes-written-by-app (write amplification factor) jumped from ~9x to ~34x; pending-compaction-bytes is steady (not backing up); SSD wear (TBW) is now projected to exhaust endurance in 14 months instead of 6 years. CPU is up on compaction threads. No errors, no rejected writes — just brutal amplification. How do you triage, find root cause, and fix?
Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.