Read replica lag climbing
Your task
A migration creates an index on a large hot Postgres table to speed up a new query. The author used CREATE INDEX CONCURRENTLY to avoid blocking writes, good. The primary stays healthy. But minutes after the migration starts, every read replica falls far behind: replica lag climbs to minutes, and the read-heavy app (which routes reads to replicas) starts serving very stale data and breaching freshness SLOs; some read endpoints time out. The primary's write traffic is normal. How do you triage and mitigate, and what's the subtlety the author missed?
It covers
- Index creation
- Concurrently
- Replica lag