Ranking experimentation platform
Design the relevance-evaluation and experimentation platform for a large search team — the system that decides whether a ranking change actually improves results before it ships to all users. It must support offline metrics (from judged data + logs), online A/B tests, guard against metric gaming, and detect when an experiment is harming a user segment.
What a strong answer looks like
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts (data model, bottlenecks, consistency, failure modes) and name the trade-offs you are making.
Clarify5:30 left
Estimate5:30 planned
Design16:30 planned
Deep dive13:30 planned
Failure9:00 planned
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.