Poison message blocks shard

On-call incidentMedium30 minAmazon style

Your task

An events-ingestion consumer reading from an ordered partition (Kinesis shard) stops making progress at 11:40. Dashboards: IteratorAge for one shard climbs linearly to 45 minutes while the other shards are healthy; the consumer logs a tight loop of the same JsonParseException: Unexpected character ('\x00') every few hundred ms; CPU on that worker is at 100%; no records are being checkpointed for that shard. A new upstream producer SDK version rolled out an hour ago. How do you triage, unblock the pipeline, and avoid losing the records behind the bad one?

It covers

  • Dlq
  • Blocked partition
  • Deserialization
  • Sqs