Prep RoomSign inStart free

Experiment design questions.

81 experiment design questions from the bank, ready to practice out loud.

Practice experiment design

Learn the ideas first · free with an account

A/B test design: hypothesis, randomization, primary metricSTAR and CARL: structuring your storyExperiment pitfalls: peeking, novelty, multiple comparisonsGuardrail metrics and healthy growthStatistical power and sample sizeInterference and choosing the randomization unitWhen you can't A/B test: quasi-experiments and holdouts

All 81 questions

Framing onboarding flow hypothesisYour PM wants to test a new onboarding flow. How would you frame the hypothesis before running anything?Hypothesis framingMid–senior
Deciding sufficient A/B test dataA stakeholder wants to ship a change after one week of A/B data. How do you decide if you have enough?Sample sizing and powerMid–senior
Randomization unit for collaboration featureYou want to test a feature that changes how teams collaborate inside an organization. What's your randomization unit and why?Randomization unitSenior–staff+
Interpreting engagement decay over timeA new feed feature shows huge engagement gains in week one and flat results in week three. How would you interpret that?Novelty/network effectsMid–senior
Primary metric moves wrong directionYour test moves the primary metric in the wrong direction but every secondary metric improves. What do you do?Interpreting unexpected resultsSenior–staff+
Testing critical CTA copy changeYour team wants to A/B-test a copy change on a critical CTA. The change is two words. Walk me through whether and how you'd run it.Hypothesis framingMid–senior
Detecting small baseline improvementsYour stakeholder wants to detect a 0.5% improvement on a metric with a current baseline of 12%. Walk me through what you'd need.Sample sizing and powerSenior–staff+
Randomization in two-sided marketplacesYou're testing a marketplace feature that affects both buyers and sellers. What randomization scheme avoids contamination?Randomization unitSenior–staff+
Revenue lift, no other movementYour A/B test shows a 4% lift in revenue but no movement in any other metric. How suspicious should you be and why?Interpreting unexpected resultsSenior–staff+
Network-dependent feature testA teammate wants to A/B-test virality features that depend on other users in the test. What's the risk and how do you handle it?Novelty/network effectsSenior–staff+
Turning vague goals into experimentsYour PM wants to test 'making the homepage better'. How do you turn that into a runnable experiment?Hypothesis framingMid–senior
Designing experiments with limited trafficYou only have access to 30% of traffic for this experiment. How does that change your design?Sample sizing and powerMid–senior
Randomization unit for multi-session usersYou want to A/B-test an algorithm change that affects sessions, but users have multiple sessions. What's your randomization unit?Randomization unitMid–senior
Communicating flat results to leadershipYour experiment showed a flat result, but a leadership decision depends on shipping it. Walk me through your conversation with the decision-maker.Interpreting unexpected resultsSenior–leadership
Validating long-running positive effectsA long-running feature test still shows a positive effect after three months. Should you trust that and how would you check?Novelty/network effectsSenior–staff+
Ad-format guardrail metricsWalk me through the guardrail metrics you'd set up before A/B-testing a new ad-format on the home feed.Hypothesis framingSenior–staff+
Interpreting significant A/A test resultsYour peer ran an A/A test and saw a 'significant' difference. What does that tell you?Sample sizing and powerMid–senior
Causal inference without randomizationYou can't randomly assign users for ethical or product reasons, but you need to estimate the effect of a treatment. Walk me through your options.Randomization unitSenior–staff+
Handling geographic heterogeneity in resultsA/B test results split sharply by country — positive in one, negative in another. Walk me through how you'd interpret that.Interpreting unexpected resultsSenior–staff+
Conversion up, what shouldn't moveYour test is meant to improve free-to-paid conversion. What metrics would you not want to see move, even if conversion goes up?Hypothesis framingMid–senior
Managing overlapping concurrent experimentsYou're running multiple experiments simultaneously and they touch overlapping user segments. What do you do about it?Sample sizing and powerSenior–staff+
Measuring long-term promotion valueHow would you measure the long-term value of a one-time promotion, beyond the immediate revenue lift?Novelty/network effectsSenior–staff+
Power-user negative effectYour A/B test shows a positive effect overall, but power-users see a clearly negative effect. How do you make the ship decision?Interpreting unexpected resultsSenior–staff+
Running experiments for committed decisionsYour team agrees the feature should ship regardless of the result. Should you still run the experiment?Hypothesis framingSenior–leadership
Noisy null versus real nullWalk me through the difference between a noisy null result and a real null. How do you tell them apart?Interpreting unexpected resultsSenior–staff+
Detecting heterogeneous treatment effectsYou suspect your treatment is heterogeneous. How would you design the experiment to capture that?Sample sizing and powerSenior–staff+
Rideshare randomization spilloverYou want to test a feature affecting drivers in a rideshare service. What randomization scheme prevents network spillover?Randomization unitSenior–staff+
Testing features with compounding effectsHow would you design an experiment to test a feature where the effect is expected to compound over time, like learning the product?Novelty/network effectsSenior–staff+
Reconciling contradictory experimentsTwo of your experiments contradict each other on the same metric. Walk me through how you'd reconcile them.Interpreting unexpected resultsSenior–staff+
Running intentionally underpowered testsWhen would you intentionally run an underpowered experiment, knowing you can't detect a typical effect?Hypothesis framingSenior–leadership
Null and alternative hypothesesYou're testing a new notification feature to increase daily active users. Write out the null and alternative hypotheses you'd use for this experiment.Hypothesis framingEntry–mid
Sample size for conversion upliftYour baseline conversion rate is 8% and you want to detect a 10% relative improvement. Roughly how many users do you need per variant?Sample sizing and powerEntry–mid
Randomization unit for search resultsYou're testing a change to the search results page. Should you randomize by user, by search query, or by session? Explain your choice.Randomization unitEntry–mid
Three-day lift then baselineYour experiment shows strong gains in the first three days, then returns to baseline. What are two possible explanations?Novelty/network effectsEntry–mid
Significant but small effect sizesYour A/B test result is statistically significant but the effect size is smaller than expected. How would you communicate this to your PM?Interpreting unexpected resultsEntry–mid
Metrics beyond the primary metricBefore launching an experiment on button color, what should you define beyond just the primary metric?Hypothesis framingEntry–mid
Power calculation for baseline metricYou have 100,000 daily active users and want to run a two-week test. Is this enough to detect a 5% change in a 20% baseline metric?Sample sizing and powerEntry–mid
Network effects and randomizationYou're testing a feature where users can invite friends. Why might user-level randomization cause problems here?Randomization unitEntry–mid
Adoption rising while engagement fallsYour new feature test shows declining engagement week-over-week, but adoption is still increasing. What might explain this pattern?Novelty/network effectsEntry–mid
Wide confidence interval, no significanceYour test shows no significant difference, but the confidence interval is very wide. What does this tell you and what would you do?Interpreting unexpected resultsEntry–mid
Reframing subjective preference testsA PM asks you to test whether 'users like the new design better.' How would you reframe this into a testable hypothesis?Hypothesis framingEntry–mid
Early stopping at significanceYour experiment reached statistical significance on day 2 of a planned 14-day test. Should you stop early? Why or why not?Sample sizing and powerEntry–mid
Offline evals versus online testingYour offline evals say a new prompt for your LLM summarization feature beats the current one. The PM asks why you'd bother with an online A/B test at all. What's your answer?Evaluating AI featuresMid–senior
Non-deterministic LLM outputsYou're A/B testing two LLM-backed variants whose outputs differ run to run, even on identical inputs. What does that non-determinism do to your experiment design and your read of the results?Evaluating AI featuresSenior–staff+
Validating LLM-as-judge metricsYour team proposes an LLM-as-judge quality score as the primary metric for an assistant A/B test. What would you validate before letting it decide the ship call?Evaluating AI featuresSenior–staff+
Variance reduction with CUPEDExplain to your PM what CUPED-style variance reduction buys an experiment — and describe a situation where it buys you almost nothing.Sample sizing and powerSenior–staff+
Imbalanced randomization mid-testHalfway through a test you notice the treatment arm has 4% fewer users than the 50/50 split should produce. Your PM says that's close enough — keep going. What's your call?Interpreting unexpected resultsMid–senior
Long-term holdouts for cumulative impactYour org ships a dozen small AI features a quarter and every individual A/B test wins. How would you use a long-term holdout to check the cumulative story, and what makes holdouts hard to keep honest?Novelty/network effectsSenior–staff+
Interleaved versus A/B for rankersYou need to compare two search rankers but your product only gets a few thousand queries a day. How would you decide between an interleaved design and a classic A/B?Sample sizing and powerSenior–staff+
Shipping without experimentationYour PM wants to ship a checkout redesign without an experiment — 'the old version is obviously worse.' What's your call, and what evidence would change it?Hypothesis framingMid–senior
Multiple comparisons problemYour primary metric is flat, but three of twenty secondary metrics are significant at p < 0.05 — all positive. The team wants to ship on the strength of those three. How do you respond?Interpreting unexpected resultsMid–senior
Skewed metrics with outlier usersYour test's primary metric is revenue per user, and a handful of whales dominate it. What does that skew do to your experiment, and what are your options?Sample sizing and powerSenior–staff+
Black Friday confoundYour company launched a feature to all users the week of Black Friday and revenue rose 30%. Why can't you credit the feature, and what setup would have let you?Hypothesis framingEntry–mid
Control group exposure during testA feature-flag bug showed the treatment to your control group for one full day of a 14-day test. What are your options for handling the analysis?Randomization unitEntry–mid
Model upgrade mid-experimentYou're A/B testing a new prompt for an AI feature, and the model provider upgrades the underlying model halfway through the test. Why is that a problem for your results?Evaluating AI featuresEntry–mid
Cost-quality tradeoff in AI featuresA more capable model wins your quality A/B but costs five times more per request. What else has to be true before you ship, and how do you put cost into the experiment readout?Evaluating AI featuresSenior–staff+
Learning models in A/B testsYou're testing an AI-powered recommendation engine that learns from user behavior over time. How does the model's learning affect your experiment duration and when you lock the model?Evaluating AI featuresSenior–leadership
Power calculation with limited trafficYour baseline conversion rate is 2.3% and you need to detect a 10% relative lift. Traffic is limited to 50K users per week. Walk me through your power calculation and timeline.Sample sizing and powerMid–senior
Randomization in multi-user accountsYou're launching a new chat feature in a B2B product where conversations span multiple users across the same company account. What's your randomization strategy?Randomization unitSenior–leadership
Non-deterministic AI outputsAn LLM-powered feature shows different results every time a user retries the same query. How does this non-determinism affect your experiment design and analysis?Evaluating AI featuresSenior–leadership
Early stopping with significanceYour experiment shows statistically significant results on day 3, but your pre-registered plan said to run for two weeks. What factors determine whether you stop early?Interpreting unexpected resultsMid–senior
Price information leakage between groupsYou're designing an experiment for a pricing change in a two-sided marketplace. How do you prevent price information from leaking between treatment and control groups?Randomization unitSenior–leadership
Multi-arm versus sequential testingYour stakeholder wants to test five different headline variations simultaneously. What are the trade-offs of a multi-arm test versus sequential A/B tests here?Sample sizing and powerMid–senior
Fraud detection tradeoff metricsA fintech fraud detection model is tested and shows 15% fewer false positives but 3% more false negatives. How do you frame the success criteria before running this experiment?Hypothesis framingSenior–leadership
Notification contamination risksYou're testing a notification feature that could trigger user invites to non-users. What contamination risks exist and how would you mitigate them?Novelty/network effectsSenior–leadership
Validating AI moderation accuracyAn AI content moderation tool is 10x faster than human review but has unknown accuracy. What experiment would you design to validate it before broader rollout?Evaluating AI featuresSenior–leadership
Enterprise experiments with few companiesYour experiment on enterprise customers requires company-level randomization, but you only have 200 companies. Is this experiment feasible and what adjustments would you make?Sample sizing and powerSenior–leadership
Unexpected guardrail metric movementYou see a 20% lift in your guardrail metric that should never move. What are the most likely causes and how do you investigate?Interpreting unexpected resultsMid–senior
Defining search relevance hypothesesA product manager wants to test 'improving search relevance.' What specific hypotheses and metrics would you propose before writing any experiment plan?Hypothesis framingMid–senior
Segmented user responses to featureYour test shows that new users love the feature but existing power users hate it. The overall metric is flat. How do you interpret this and what do you recommend?Interpreting unexpected resultsSenior–leadership
Randomization across fulfillment centersYou're testing a same-day delivery option in a logistics network where fulfillment centers serve overlapping geographic areas. What's your randomization approach?Randomization unitSenior–leadership
Conflicting metrics in summarizationAn AI summarization feature reduces time-on-page by 40% but increases return visits by 25%. How would you have framed the success metric upfront?Hypothesis framingSenior–leadership
Control contamination mid-testYour control group is accidentally exposed to the treatment for 6 hours on day 4 of a 14-day test. How does this affect your analysis and what are your options?Interpreting unexpected resultsMid–senior
Seasonality in experiment timingYou need to test a feature during the holiday shopping season when traffic patterns are anomalous. How does seasonality change your experimental approach?Sample sizing and powerMid–senior
Network effects in social featuresA social feature creates network effects where a user's experience depends on how many of their friends are also in treatment. How do you design around this?Novelty/network effectsSenior–leadership
Feedback loops in AI performanceYour AI model's performance degrades when user behavior shifts post-launch. How would you design an experiment that accounts for this feedback loop?Evaluating AI featuresSenior–leadership
Testing rare user actionsYou're asked to test a feature that only affects 2% of users who complete a rare action. What sample size challenges do you face and how do you address them?Sample sizing and powerSenior–leadership
Declining lift across significant weeksWeek one shows a 5% revenue lift, week two shows 2%, week three shows -1%. All results are statistically significant. What's your interpretation?Interpreting unexpected resultsSenior–leadership
Randomization across multiple platformsYour mobile app experiment needs to account for users who switch between the app and web. What's your randomization strategy to maintain consistency?Randomization unitMid–senior
Statistical power in pharmaceutical trialsA pharmaceutical trial protocol requires 80% power to detect a 15% reduction in adverse events at p<0.01. Walk me through what this means for trial design.Sample sizing and powerMid–senior
Baseline stability with learning modelsYou're testing an AI code completion tool for developers. The model improves with usage data, but you need a stable baseline. How do you structure the experiment?Evaluating AI featuresSenior–leadership