Single tenant shard hot

On-call incidentHard35 min

Your task

Your write path uses a horizontally sharded datastore (data routed by tenant_id hash to 64 shards). During a product launch, one tenant goes viral. Shard 23's CPU and write latency spike to 10x the fleet median, its queue depth climbs, and a slice of traffic, all for that one big tenant, sees timeouts, while the other 63 shards sit nearly idle. Adding more shards in the past didn't help similar events. No deploy; pure traffic. Triage and mitigate.

It covers

  • Sharding
  • Skew
  • Celebrity key