Design offline ranker evaluation
Design the logging and offline-evaluation infrastructure that lets a ranking team estimate 'how would a new ranker have performed?' *without* shipping it to live traffic for every candidate model. The hard problem: your logs only contain feedback (clicks/conversions) on items the *current* ranker chose to show — you have no labels for items it didn't show, and clicks are heavily position-biased (top results get clicked regardless of relevance). How do you log and evaluate so offline estimates predict online results?
What a strong answer looks like
Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts (data model, bottlenecks, consistency, failure modes) and name the trade-offs you are making.
Clarify5:30 left
Estimate5:30 planned
Design16:30 planned
Deep dive13:30 planned
Failure9:00 planned
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.
Run or narrate your approach, then ask the coach.