Disk saturation from debug logging
Your task
A single-host analytics ingestion service (Java) suddenly slows to a crawl at 03:00. Throughput drops 80%, GC logs show frequent long pauses, and free -m shows almost no free memory and a large amount in 'buff/cache'. iostat shows the data disk at 100% util with high await. CPU is mostly in iowait and %sy. No deploy happened, but a config change yesterday set DEBUG-level logging on a hot path, and log files have grown to hundreds of GB. The host also runs the embedded storage engine on the same disk. Triage and mitigate.
It covers
- Disk io
- Memory pressure
- Log volume