Ranking model rollout safety
Design the experimentation platform a search team uses to safely roll out new ranking models. They ship a new ranker every week, and a bad ranker hurts revenue and user trust immediately. You need to (a) detect within hours whether a new ranker is better or worse than the current one, (b) protect against a catastrophically bad model reaching all users, and (c) handle the fact that ranking quality is hard to measure with a single conversion metric. Design the rollout + evaluation system.
What a strong answer looks like
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts (data model, bottlenecks, consistency, failure modes) and name the trade-offs you are making.
Clarify4:30 left
Estimate4:30 planned
Design13:30 planned
Deep dive10:30 planned
Failure7:00 planned
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.