There is more of this guide.
Practice
20 questions- ML system designYou're building a system that summarizes news articles. Walk me through your evaluation strategy beyond a few automatic metrics.
- ML system designYour offline model evaluation looks great but online metrics show degraded performance. Walk me through the most common reasons.
- ML system designDesign an evaluation framework for a chatbot that handles customer support. What does success look like and how do you measure it?
- ML system designDesign an A/B test for replacing a heuristic ranker with an ML model. What guardrails do you set?
- ML system designDesign a system where you continuously evaluate a model against a champion baseline and promote new versions automatically.
- ML system designDesign an evaluation harness for a code-completion model. How do you measure usefulness beyond exact-match accuracy?