Question

Design a high-throughput pub/sub system for clickstream/telemetry: producers publish ~2M events/sec across thousands of topics, and many independent consumer groups (analytics, billing, ML feature pipelines) each read the full stream at their own pace without affecting one another. Requirements: a slow consumer must not slow down producers or other consumers, consumers can be added/removed and the load rebalances, and on consumer crash, processing resumes without dropping events. Walk through the design.