CDC pipeline with ordering guarantees
Your task
Design a change-data-capture (CDC) pipeline that streams every row change from a 5 TB MySQL OLTP cluster (40k writes/sec) to a data lake, a search index, and a cache invalidator. Consumers need changes within 5 seconds, in commit order per row, with no lost or duplicated effects after a crash. How do you capture changes, preserve ordering, and guarantee delivery semantics?
It covers
- Cdc
- Log based
- Exactly once