Disk saturation from debug logging

On-call incidentMedium30 min

Your task

A single-host analytics ingestion service (Java) suddenly slows to a crawl at 03:00. Throughput drops 80%, GC logs show frequent long pauses, and free -m shows almost no free memory and a large amount in 'buff/cache'. iostat shows the data disk at 100% util with high await. CPU is mostly in iowait and %sy. No deploy happened, but a config change yesterday set DEBUG-level logging on a hot path, and log files have grown to hundreds of GB. The host also runs the embedded storage engine on the same disk. Triage and mitigate.

It covers

  • Disk io
  • Memory pressure
  • Log volume