Docker storage exhaustion from build cache

On-call incidentMedium30 min

Your task

A long-running Docker host (used as a self-hosted runner) starts failing container starts with 'no space left on device' on /var/lib/docker, and df -h confirms that filesystem is full. The host has been up for months running thousands of CI jobs. docker system df shows large 'RECLAIMABLE' figures: dozens of GB in dangling images, stopped containers, anonymous volumes, and build cache. Image churn rose after the team switched to multi-stage builds last month. No application bug. Triage and remediate.

It covers

  • Docker
  • Overlay2
  • Dangling images