Exactly-once billing aggregates
Your task
Design a stateful stream-processing job that maintains exactly-once rolling 1-hour aggregates (count, sum, distinct-users) over a 200k events/sec firehose, where the aggregates are served to customers and bill them, so an over-count is a refund and an under-count is lost revenue. The job runs on a cluster that experiences periodic node failures, autoscaling-driven rebalances, and weekly deploys. Walk through how state, checkpointing, and the output sink combine to guarantee no event is counted twice or zero times across all those disruptions.
It covers
- Streaming aggregation
- Checkpointing
- Idempotent output