Search index sync from Postgres

DesignHard50 minShopify and Amazon style

Your task

Design a change-data-capture pipeline that keeps a search index (Elasticsearch) and a read-cache continuously in sync with a primary Postgres database for an e-commerce catalog of 50M products updated thousands of times/sec (price changes, inventory, edits). The search results can be slightly stale but must never be permanently wrong, must reflect deletes, and must survive consumer crashes and re-indexes. How do you capture changes from Postgres and propagate them reliably without dual-writes from the app?

It covers

  • Cdc
  • Search index
  • Consistency
  • Outbox