Exactly-once transaction ETL

DesignHard45 minDatabricks and Snowflake style

Your task

Design an exactly-once streaming ETL that consumes ~120k events/sec of financial transactions from Kafka, enriches each with reference data and FX rates, and writes aggregated per-merchant hourly revenue into both an OLAP warehouse (for analytics) and a low-latency KV store (for a live merchant dashboard). A consumer crash, a rebalance, or a redeploy must never double-count or drop revenue, and the dashboard must converge to the same totals the warehouse reports. Reads are heavy on the KV store (10k QPS); the warehouse is append-mostly.

It covers

  • Exactly once
  • Streaming
  • Idempotent sink