Cache serialization format collision

On-call incidentHard40 min

Your task

During a 30-minute rolling deploy of the profile service, users intermittently see corrupted or stale profile data, sometimes another user's field, sometimes a missing avatar, at roughly the rate matching the fraction of pods on the new version, and it persists for a while after the deploy completes. Dashboards: shared Redis cache hit-rate is normal, no errors, no latency change. The new version changed the cached value's serialization from JSON to a packed binary format and kept the same cache keys; old pods write/read JSON under those keys, new pods write/read binary under the same keys, against a shared Redis. How do you triage and mitigate, and why does it persist after the deploy?

It covers

  • Cache poisoning
  • Serialization
  • Shared cache