Postgres WAL directory fills disk

On-call incidentHard35 min

Your task

At 01:10 your Postgres primary stops accepting writes: the app logs 'could not extend file ... No space left on device' and the database goes effectively read-only. df shows the data volume at 100%. The pg_wal directory has ballooned to hundreds of GB. You recently set up a new logical replication subscriber for an analytics pipeline, and that subscriber has been down for maintenance since yesterday afternoon. How do you triage, recover service, and prevent recurrence?

It covers

  • Wal
  • Log growth
  • Replication