Redis snapshot data loss

On-call incidentHard40 min

Your task

After a sudden power loss took down a Redis host at 03:00, the on-call promotes a replica and service resumes. By morning, users report lost data: session carts, recently-posted draft comments, and counters reset. Dashboards: the Redis cluster is healthy now; the promoted replica was using RDB snapshots every 5 minutes (AOF disabled); master_repl_offset after promotion is ~4 minutes behind the last write the app logs show being acked; the app treated Redis as the system of record for these features. How do you triage what was lost, mitigate ongoing impact, and recover what you can?

It covers

  • Truncated write
  • Fsync
  • Redis
  • Cache vs source