Schema migration locks writes

On-call incidentHard40 min

Your task

A migration job kicks off at 02:30 to add a NOT NULL column with a default to the 'orders' table (Postgres 11, ~400M rows). At 02:31 writes to orders start timing out, the API returns 503s for anything touching orders, and pg_stat_activity shows a long-running ALTER TABLE holding an AccessExclusiveLock with a growing queue of blocked queries behind it. Replica lag begins climbing. The migration was tested on a staging copy with 1M rows and ran in 2 seconds. Triage and mitigate.

It covers

  • Lock contention
  • Online ddl
  • Postgres