Kinesis iterator age from dual consumer

On-call incidentMedium30 minAmazon style

Your task

A Kinesis-backed analytics pipeline alerts at 13:00: GetRecords.IteratorAgeMilliseconds has climbed to 25 minutes and is rising on the events stream (6 shards). The KCL consumer fleet (6 workers) is at ~20% CPU. CloudWatch shows ReadProvisionedThroughputExceeded is non-zero and climbing on the stream, and GetRecords.Latency is elevated. Recent context: a new internal dashboard team subscribed a second consumer application (a second KCL app, not enhanced fan-out) to the same stream this morning. Triage and mitigate the growing iterator age.

It covers

  • Kinesis
  • Iterator age
  • Throughput exceeded