Search index sync from Postgres
Your task
Design a change-data-capture pipeline that keeps a search index (Elasticsearch) and a read-cache continuously in sync with a primary Postgres database for an e-commerce catalog of 50M products updated thousands of times/sec (price changes, inventory, edits). The search results can be slightly stale but must never be permanently wrong, must reflect deletes, and must survive consumer crashes and re-indexes. How do you capture changes from Postgres and propagate them reliably without dual-writes from the app?
It covers
- Cdc
- Search index
- Consistency
- Outbox