Design stateful windowed stream aggregation

DesignHard50 minDatabricks and Google style

Your task

Design a stateful stream-processing pipeline that computes per-key windowed aggregations (e.g. 5-minute rolling revenue per merchant) over an event stream where events arrive out of order and late (mobile clients buffer offline for minutes/hours). 300K events/sec, results must be correct despite late data, and the operator state (per-key windows) can be large and must survive crashes. Emit results with bounded latency.

It covers

  • Windowed aggregation
  • Watermarks
  • Late data