Migration holding exclusive lock

On-call incidentHard40 min

Your task

A deploy at 11:00 ran a schema migration that's now blocking the whole app. The migration does ALTER TABLE orders ADD COLUMN status_v2 text NOT NULL DEFAULT 'pending' followed by a CREATE INDEX on a 150M-row Postgres table, and it's running inside the deploy's migration step. Since it started, all reads and writes to orders hang; pg_stat_activity shows the migration holding an AccessExclusiveLock on orders, and behind it a growing queue of normal app queries all "waiting for lock," including simple SELECTs. The app is effectively down for anything touching orders. The migration has been running 6 minutes with no end in sight. Triage and decide.

It covers

  • Schema migration
  • Ddl locks