Near-real-time search indexing

DesignHard45 minGoogle style

Your task

Design near-real-time indexing for a search system where new and updated documents must become searchable within ~5 seconds of the write, while the cluster serves 50k queries/sec over 2B documents. Writes come at 40k docs/sec including frequent updates and deletes (not just appends). The naive 'rebuild the index' approach is far too slow. How do you make a write visible in seconds without tanking query latency or memory, and how do you handle updates/deletes when inverted-index postings are append-only?

It covers

  • Near real time indexing
  • Segment merge
  • Freshness