Disk full WAL accumulation

On-call incidentHard35 min

Your task

Your primary Postgres just went read-only / started rejecting writes with ERROR: could not extend file ... No space left on device. The data volume is 100% full. It filled fast over the last few hours. The pg_wal directory is enormous, far bigger than usual. You recently spun up a new read replica that's been having trouble staying connected, and there's a logical replication slot feeding a downstream analytics pipeline that's been broken for two days. Triage and mitigate without losing data.

It covers

  • Disk full
  • Wal accumulation
  • Replication slot