Deleted file still growing

On-call incidentMedium30 min

Your task

At 02:50 a fleet of app hosts starts returning 500s; df -h shows / at 100% and writes fail with ENOSPC. du -sh /var/log/* finds a single app.log at 180GB. You rm app.log and df still shows the disk full, and the file keeps growing in du even though it's gone from ls. A config push three days ago changed the log level to DEBUG on a hot request path, and logrotate exists but its postrotate sends SIGHUP, which this app ignores. Triage and fix without taking an outage.

It covers

  • Log rotation
  • Open file handle
  • Enospc