Writes fail despite disk space
Your task
An upload-processing service starts failing with 'No space left on device' when writing temp files, but df -h shows the data volume is only 47% full. Writes are erroring across the whole fleet on the same mount. df -i shows inode usage at 100%. The directory that holds per-job scratch files has tens of millions of tiny files, and a cleanup cron that used to delete finished jobs' scratch dirs was disabled in a refactor two weeks ago. How do you triage this confusing 'disk full but not full' symptom and recover?
It covers
- Inodes
- Small files
- Filesystem