Exactly-once transaction ETL
Your task
Design an exactly-once streaming ETL that consumes ~120k events/sec of financial transactions from Kafka, enriches each with reference data and FX rates, and writes aggregated per-merchant hourly revenue into both an OLAP warehouse (for analytics) and a low-latency KV store (for a live merchant dashboard). A consumer crash, a rebalance, or a redeploy must never double-count or drop revenue, and the dashboard must converge to the same totals the warehouse reports. Reads are heavy on the KV store (10k QPS); the warehouse is append-mostly.
It covers
- Exactly once
- Streaming
- Idempotent sink