ML system design questions.

67 ml system designquestions from the bank — open to read. Pick one and practice it out loud; a coach note comes back in seconds.

Learn the ideas first

All 67 questions

Design an ML system to recommend videos to logged-in users on a streaming service with 200 million users.Training data pipelineSenior–staff+Design the monitoring story for a credit-decisioning model that's been in production for six months. What do you alert on?Drift detectionSenior–staff+Design a fraud-detection ML system that has to decide every transaction in under 100ms at 10K transactions per second.Serving architectureSenior–staff+You're building a system that summarizes news articles. Walk me through your evaluation strategy beyond a few automatic metrics.Eval designSenior–staff+Your offline model evaluation looks great but online metrics show degraded performance. Walk me through the most common reasons.Online/offline gapSenior–staff+Design a system that decides when to retrain a customer-churn model. What signals would trigger a retrain?Retraining cadenceSenior–staff+Design a feature store for a company with a hundred ML use cases. What's non-negotiable in the design?Training data pipelineStaff+–leadershipDesign a system where your ML model serves predictions for an e-commerce ranking page. What happens when the model is unavailable?Fallback behaviorSenior–staff+Design a system for serving large-language-model inference to a million daily users, balancing cost and latency.Serving architectureSenior–staff+Design an evaluation framework for a chatbot that handles customer support. What does success look like and how do you measure it?Eval designSenior–staff+Design the labeling pipeline for a system that classifies user-uploaded images into 200 categories. Labels are noisy.Training data pipelineSenior–staff+Walk me through how you'd build a drift-detection system for a fleet of fifty production ML models.Drift detectionStaff+–leadershipDesign a search-ranking system that personalizes results per user using behavioral features. Latency budget is 50ms.Serving architectureSenior–staff+How would you instrument a deployed model so that you can detect training-serving skew quickly?Online/offline gapSenior–staff+Design the data pipeline for training a delivery-time prediction model that needs to update daily.Training data pipelineSenior–staff+An LLM-based product feature occasionally produces unsafe outputs. Design the safety system around it.Fallback behaviorSenior–staff+Design an A/B test for replacing a heuristic ranker with an ML model. What guardrails do you set?Eval designSenior–staff+Design a feature for showing AI-generated product descriptions on an e-commerce site. Quality and cost both matter.Serving architectureSenior–staff+Design a system where you continuously evaluate a model against a champion baseline and promote new versions automatically.Retraining cadenceStaff+–leadershipHow would you architect a system that lets data scientists run dozens of experiments per day against the same offline dataset, reproducibly?Training data pipelineStaff+–leadershipWalk me through how you'd detect that a deployed image-classification model is being shown a new class of input it wasn't trained on.Drift detectionSenior–staff+Design an evaluation harness for a code-completion model. How do you measure usefulness beyond exact-match accuracy?Eval designSenior–staff+Design the serving infrastructure for a system that runs many small models — say, one per customer — economically.Serving architectureStaff+–leadershipYou discover that the feature store generates one set of values offline and a slightly different one online. How do you design around that?Online/offline gapStaff+–leadershipDesign a system where your ML model assists a human reviewer in approving loan applications. What does the integration look like?Fallback behaviorSenior–staff+Design a system to train a model on user data while respecting deletion requests. What does compliance with right-to-be-forgotten look like?Training data pipelineStaff+–leadershipYour model retrains nightly but stakeholders want it to react to events within minutes. How do you architect a system that supports both?Retraining cadenceSenior–staff+Design an evaluation strategy for a retrieval-augmented generation system. What's actually being measured?Eval designSenior–staff+Design a system that uses ML to suggest replies in a messaging product, with privacy as a hard constraint.Serving architectureSenior–staff+Your team's pricing model occasionally produces obviously wrong prices. What system around the model prevents customer harm?Fallback behaviorSenior–staff+You're building a spam classifier for email. Walk me through how you'd collect and prepare training data, including how you'd handle the class imbalance between spam and legitimate emails.Training data pipelineEntry–midDesign a simple drift detection check for a model that predicts house prices. What metrics would you track on incoming prediction requests to know if the model is seeing different data than it was trained on?Drift detectionEntry–midYou're deploying a sentiment analysis model that scores product reviews. Should you serve it via a REST API, batch processing, or both? Explain your reasoning based on the use case.Serving architectureEntry–midYou've built a model that tags support tickets by topic. Beyond overall accuracy, what other metrics would you track to ensure it's working well across all ticket categories?Eval designEntry–midYour movie recommendation model has 85% accuracy offline but users aren't clicking the recommendations in production. What are three possible reasons for this gap?Online/offline gapEntry–midYou've deployed a model that predicts restaurant delivery times. How would you decide how often to retrain it—daily, weekly, monthly? What factors influence this decision?Retraining cadenceEntry–midYour product search ranking model goes down during a deployment. What should the search page show to users while the model is unavailable?Fallback behaviorEntry–midYou're training a click-through rate model on web traffic data. How would you split your data into train/validation/test sets, and what time period should each cover?Training data pipelineEntry–midYour job title classification model was trained on 2023 data. It's now 2025. What simple checks would you implement to detect if the distribution of job titles has changed significantly?Drift detectionEntry–midYou need to serve predictions from a lightweight model that scores user profiles. Would you precompute all scores and cache them, or compute on-demand? What trade-offs matter here?Serving architectureEntry–midYou built a model to detect toxic comments. How would you evaluate whether it works equally well across different types of toxicity (hate speech, harassment, threats)?Eval designEntry–midYour model predicts customer lifetime value and is retrained monthly. Suddenly, a competitor launches and customer behavior changes overnight. How would you adjust your retraining approach to respond faster?Retraining cadenceEntry–midDesign a training data pipeline for a recommendation model that needs to incorporate user interactions from the last 24 hours. How do you handle late-arriving events and ensure reproducibility?Training data pipelineMid–seniorYou're serving a personalized ranking model that takes 300ms to score items, but the page needs to load in under 200ms. Walk me through your architecture options and trade-offs.Serving architectureSenior–staff+Design a drift detection system for a loan-approval model where ground truth labels arrive 6-12 months after predictions. What proxy metrics do you monitor in the meantime?Drift detectionSenior–staff+Your medical imaging model was trained on data from five hospitals but will deploy to hundreds. Design your evaluation strategy to catch distribution shift before it impacts patient care.Eval designSenior–leadershipDesign the fallback behavior for a dynamic pricing model in a ride-sharing app. The model goes down during Friday evening rush hour—what happens?Fallback behaviorMid–seniorYou're building a content moderation system that must review 50K user posts per minute. Design the ML serving architecture, including how you handle spikes to 200K per minute.Serving architectureSenior–staff+Design a retraining strategy for a product search ranking model in an e-commerce platform. How do you decide between daily, weekly, or event-triggered retraining?Retraining cadenceMid–seniorYour A/B test shows the new model has better offline metrics but users are engaging less. Walk me through your debugging process to identify what's wrong.Online/offline gapMid–seniorDesign a training data pipeline for a fraud detection model where only 0.1% of transactions are fraudulent. How do you handle class imbalance and ensure the model sees recent fraud patterns?Training data pipelineMid–seniorYou're deploying a recommendation model across 20 countries with different languages and cultures. Design your evaluation framework to ensure quality across all markets.Eval designSenior–staff+Design a system to detect when your ad-click prediction model is being gamed by adversarial actors. What metrics and patterns do you monitor?Drift detectionSenior–staff+Your search ranking model serves results in under 50ms, but you want to add a large language model reranker. Design the serving architecture to maintain latency SLAs.Serving architectureSenior–staff+Design the fallback strategy for a real-time inventory allocation model in a warehouse management system. The model fails—how do you ensure orders still get fulfilled?Fallback behaviorMid–seniorYou're building a training pipeline for a model that uses features from ten different upstream data sources with varying SLAs. How do you design for reliability and debuggability?Training data pipelineSenior–staff+Design a retraining system for a conversational AI model. What signals indicate the model needs updating versus just prompt engineering or retrieval improvements?Retraining cadenceSenior–staff+Your offline metrics use last-click attribution but your online system uses multi-touch attribution. Design an evaluation strategy that bridges this gap.Online/offline gapSenior–staff+Design a drift detection system for a fleet of region-specific models (50+ regions). How do you distinguish between legitimate regional differences and actual model degradation?Drift detectionStaff+–leadershipYou're serving embeddings for a semantic search system to 100M users. Design the caching and serving architecture to minimize cost while maintaining relevance.Serving architectureSenior–staff+Design an evaluation framework for a code-generation model used by developers. How do you measure quality beyond pass-at-k and what does production success look like?Eval designSenior–staff+Your supply chain demand forecasting model performs well on average but badly during promotional periods. Walk me through how you'd diagnose and address this online-offline gap.Online/offline gapMid–seniorDesign a training data pipeline for a document understanding model where documents arrive as scanned PDFs with varying quality. How do you ensure consistent training data quality?Training data pipelineMid–seniorDesign the fallback behavior for an ML-powered drug interaction checker in a pharmacy system. What happens when the model is unavailable or returns low-confidence predictions?Fallback behaviorSenior–staff+You're designing a retraining pipeline for a time-series forecasting model. How do you decide the lookback window for training data and when historical data becomes obsolete?Retraining cadenceMid–seniorDesign a serving architecture for a multi-modal model (text + image) that powers product search. How do you optimize for cost when image encoding is 10x more expensive than text?Serving architectureSenior–staff+Your credit risk model shows concept drift: default rates are rising but feature distributions look stable. Design your investigation process and mitigation strategy.Drift detectionSenior–staff+