Question

At 02:30 the `account-activity` Kafka consumer group's lag suddenly jumps from ~0 to the *entire retention* of the topic (≈2 billion records across 48 partitions) and the consumers start churning at full throttle, re-emitting events downstream. Support pages in: users are getting old notifications resent and a few derived counters look doubled. Dashboards: no broker errors, produce rate normal; the group's committed offsets dropped to the log-start offset on all partitions at 02:30. Recent context: an ops engineer ran an incident-recovery script that recreated the consumer group's `__consumer_offsets` after a separate tooling problem; the consumers run with `auto.offset.reset=earliest`. How do you triage and mitigate?