Metadata lock blocks checkout

On-call incidentHard35 minShopify style

Your task

A routine schema migration that adds a NOT NULL column with a default to a 200M-row orders table on MySQL 8 (InnoDB) was deployed at 16:00. By 16:02 the entire checkout path is timing out. SHOW PROCESSLIST is full of queries in 'Waiting for table metadata lock'. The migration itself shows as running. APM shows write throughput dropped to near zero while reads continue. The on-call DBA says the table is 'too big to lock'. How do you triage and what's your mitigation plan?

It covers

  • Lock contention
  • Migrations