Hot shard in Redis Cluster

On-call incidentHard35 minMeta style

Your task

A celebrity posted at 20:00 and one piece of content is going viral. Your Redis Cluster spreads keys across 12 shards, but one shard's CPU is at 100% while the other 11 sit at 20%, that shard's network egress is saturated, and reads of the viral post's cached payload are timing out even though the data is cached and small. Overall cluster memory is fine. How do you triage and mitigate this in real time?

It covers

  • Hot key
  • Sharding