CDC pipeline with ordering guarantees

DesignHard40 minDatabricks style

Your task

Design a change-data-capture (CDC) pipeline that streams every row change from a 5 TB MySQL OLTP cluster (40k writes/sec) to a data lake, a search index, and a cache invalidator. Consumers need changes within 5 seconds, in commit order per row, with no lost or duplicated effects after a crash. How do you capture changes, preserve ordering, and guarantee delivery semantics?

It covers

  • Cdc
  • Log based
  • Exactly once