Read replica lag climbing

On-call incidentHard35 min

Your task

A migration creates an index on a large hot Postgres table to speed up a new query. The author used CREATE INDEX CONCURRENTLY to avoid blocking writes, good. The primary stays healthy. But minutes after the migration starts, every read replica falls far behind: replica lag climbs to minutes, and the read-heavy app (which routes reads to replicas) starts serving very stale data and breaching freshness SLOs; some read endpoints time out. The primary's write traffic is normal. How do you triage and mitigate, and what's the subtlety the author missed?

It covers

  • Index creation
  • Concurrently
  • Replica lag