On-callMediumoc-g105

Subject Capacity incidentsLevel Mid–Senior~30 minCommon in Storage & CDN · Reliability & on-call interviewsIndustries Technology

Question

A subset of your service instances start failing writes with `No space left on device`, return 500s, and some crash on startup. The host dashboards show root-volume disk usage hit 100% on the affected nodes; CPU and memory are normal. A change yesterday bumped the application log level to DEBUG to chase a bug, and log rotation hasn't kept up — the local disk filled with verbose logs and a few large core dumps. How do you triage and recover, and how do you keep disk from being a silent killer?

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

Learn the concepts

Diagram & narrate the incident

Loading whiteboard…

Run or narrate your approach, then ask the coach.