Replica lag stale reads

On-call incidentMedium30 min

Your task

Users report that after saving a profile change, refreshing sometimes shows the OLD value, then it 'fixes itself' a few seconds later. Started 30 minutes ago. Dashboards: write traffic is normal; one of three Postgres read replicas shows replication lag oscillating 3–15s (the other two are <500ms); the load balancer routes reads round-robin across all three; the lagging replica's WALReceiver is fine but its disk write latency doubled after a noisy co-tenant landed on that host. No app deploy. How do you triage, stop the bad user experience now, and prevent stale reads going forward?

It covers

  • Read your writes
  • Replication lag
  • Stale reads
  • Postgres