Disk Inode Exhaustion

When your server says "Disk Full" but you have 500GB of free space.

The idea

A hard drive doesn't just store data (file contents); it also stores metadata (file names, permissions, and locations on the disk). In Linux filesystems (like ext4), this metadata is stored in structures called inodes. When you format a disk, the OS creates a fixed number of inodes. Every single file or directory uses exactly 1 inode, regardless of its size. If an application creates millions of tiny 1-byte files (like session tokens or cache files), you can run out of inodes completely while the disk is still 99% empty. Once you are out of inodes, you cannot create new files, and the system crashes.

Step 1: A newly formatted disk. It has a bucket for Data, and a bucket for Inodes.

How it works (Debugging)

When an application fails to write to disk, developers usually run df -h to check disk space. If it says 10% used, they get confused. The secret command is df -i, which checks inode usage. Finding the culprit means finding the directory with the most files.

# Standard disk usage (Data) - Looks fine!
$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       100G   10G   90G  10% /

# Inode usage - 100% full!
$ df -i /
Filesystem       Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1       6553600 6553600       0  100% /

# Find the directory hiding millions of tiny files:
$ find /var/log -xdev -type d -exec \
    sh -c 'echo "$(ls -1 "{}" | wc -l) {}"' \; | sort -rn | head

Cost

Inodes are allocated when the file system is created (formatted). On ext4, the default ratio is 1 inode per 16KB of space. You can change this ratio when you format the drive (mkfs.ext4 -i 4096 /dev/sda1 to get more inodes), but you cannot change it easily after the fact without wiping the drive. Some newer filesystems (like XFS or Btrfs) allocate inodes dynamically, solving this problem entirely.

Watch out for