Ranking experimentation platform
Your task
Design the relevance-evaluation and experimentation platform for a large search team, the system that decides whether a ranking change actually improves results before it ships to all users. It must support offline metrics (from judged data + logs), online A/B tests, guard against metric gaming, and detect when an experiment is harming a user segment.
It covers
- Evaluation
- Ab testing
- Feedback loops