Single partition consumer falls behind
Your task
An events pipeline writes to a Kafka topic audit-log (16 partitions) keyed by org_id, consumed by a 16-instance group. At 11:00 lag alerts fire but ONLY on partition 4: its lag is 6M and climbing while the other 15 partitions are at ~0. The instance owning partition 4 is pegged at 100% CPU; the rest idle. No errors, no poison message, the records on partition 4 are all valid and from a single org_id. Recent context: a large enterprise customer (org_id=ACME) turned on verbose audit logging this morning, and they generate ~70% of all audit events. How do you triage and mitigate?
It covers
- Low cardinality key
- Hot key
- Fan out