Index migration stalls read replicas
Your task
A migration to add an index on a large 'events' table (MySQL 8, primary + 2 read replicas) runs at 03:00 using a standard online DDL. The primary stays healthy and the migration completes there in 6 minutes. But starting at 03:00, replica lag on both read replicas climbs to 40 minutes, and the app, which reads recent events from replicas, starts showing users stale/missing data and the analytics dashboards go blank. After the index finishes replicating, lag drains and everything recovers. Triage and mitigate.
It covers
- Index build
- Replication lag
- Mysql