Hot shard blocks cluster

On-call incidentHard30 minAmazon style

Your task

Your time-series ingestion service writes to a sharded datastore keyed by metric_name. After onboarding a huge customer, write latency and throttling errors spiked, even though you added shards and aggregate cluster utilization is only ~40%. Dashboards show one shard at 100% while the rest idle: that customer emits one extremely high-volume metric, so nearly all its writes hash to a single partition. Adding shards didn't help, the hot key still lands on one. How do you triage and mitigate, and what's the durable fix?

It covers

  • Partition key hotspot
  • Shard imbalance
  • Horizontal cap