Code RoomIndex migration stalls read replicas
MediumPrep Room Coding #2437

Index migration stalls read replicas

On-callDatabases & SQLDistributed systemsMid–Senior~35 min

A migration to add an index on a large 'events' table (MySQL 8, primary + 2 read replicas) runs at 03:00 using a standard online DDL. The primary stays healthy and the migration completes there in 6 minutes. But starting at 03:00, replica lag on both read replicas climbs to 40 minutes, and the app — which reads recent events from replicas — starts showing users stale/missing data and the analytics dashboards go blank. After the index finishes replicating, lag drains and everything recovers. Triage and mitigate.

What a strong answer looks like

Stop the bleeding first (mitigate), then form hypotheses from real signals. Separate root cause from symptom, communicate status as you go, and close with what prevents a repeat.

0:00 of about 35 min
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.