Near-real-time search indexing
Your task
Design near-real-time indexing for a search system where new and updated documents must become searchable within ~5 seconds of the write, while the cluster serves 50k queries/sec over 2B documents. Writes come at 40k docs/sec including frequent updates and deletes (not just appends). The naive 'rebuild the index' approach is far too slow. How do you make a write visible in seconds without tanking query latency or memory, and how do you handle updates/deletes when inverted-index postings are append-only?
It covers
- Near real time indexing
- Segment merge
- Freshness