Profile updates show stale reads

On-call incidentMedium30 min

Your task

After a database failover at 15:20, users intermittently see stale data: they update a profile, the save succeeds, but a reload sometimes shows the old value for ~10s. Dashboards: a MySQL primary failover happened at 15:20 (old primary had a hardware fault); your app reads from a pool of replicas and writes to the primary; replication lag on two replicas spiked to 8-12s right after the failover; reads are randomly load-balanced across replicas including the lagging ones. No app deploy. How do you triage and mitigate?

It covers

  • Read from replica
  • Replication lag
  • Stale read