Inode exhaustion from PHP session files

On-call incidentMedium30 min

Your task

A PHP application server starts throwing 'No space left on device' on file creation at 10:00, yet df -h shows / at only 55% used, there's plenty of free space in bytes. New session creation fails so users can't log in. The app stores PHP session files as one tiny file per session under /var/lib/php/sessions, and the session garbage-collector probability was effectively set to 0 in a config change last month. A marketing push tripled new (anonymous) sessions today. Walk through triage and remediation.

It covers

  • Small files
  • Session files
  • Df i