Kinesis partition skew from unbalanced keys

On-call incidentMedium35 minAmazon style

Your task

An IoT ingestion pipeline writes device telemetry to a Kinesis stream (8 shards), using region_code as the partition key. At 12:00 producers start getting ProvisionedThroughputExceededException (write throttling) on some records, but aggregate stream IncomingBytes is well under the 8-shard write budget (16 MB/s). The CloudWatch shard-level WriteProvisionedThroughputExceeded metric is high on exactly 2 of the 8 shards; the other 6 are nearly idle. Recent context: most traffic comes from two region_code values (us-east, eu-west) covering ~80% of devices. Triage and mitigate the write throttling despite spare aggregate capacity.

It covers

  • Kinesis
  • Hot shard
  • Partition key