Experiment design questions.
81 experiment designquestions from the bank — open to read. Pick one and practice it out loud; a coach note comes back in seconds.
Learn the ideas first
All 81 questions
Your PM wants to test a new onboarding flow. How would you frame the hypothesis before running anything?A stakeholder wants to ship a change after one week of A/B data. How do you decide if you have enough?You want to test a feature that changes how teams collaborate inside an organization. What's your randomization unit and why?A new feed feature shows huge engagement gains in week one and flat results in week three. How would you interpret that?Your test moves the primary metric in the wrong direction but every secondary metric improves. What do you do?Your team wants to A/B-test a copy change on a critical CTA. The change is two words. Walk me through whether and how you'd run it.Your stakeholder wants to detect a 0.5% improvement on a metric with a current baseline of 12%. Walk me through what you'd need.You're testing a marketplace feature that affects both buyers and sellers. What randomization scheme avoids contamination?Your A/B test shows a 4% lift in revenue but no movement in any other metric. How suspicious should you be and why?A teammate wants to A/B-test virality features that depend on other users in the test. What's the risk and how do you handle it?Your PM wants to test 'making the homepage better'. How do you turn that into a runnable experiment?You only have access to 30% of traffic for this experiment. How does that change your design?You want to A/B-test an algorithm change that affects sessions, but users have multiple sessions. What's your randomization unit?Your experiment showed a flat result, but a leadership decision depends on shipping it. Walk me through your conversation with the decision-maker.A long-running feature test still shows a positive effect after three months. Should you trust that and how would you check?Walk me through the guardrail metrics you'd set up before A/B-testing a new ad-format on the home feed.Your peer ran an A/A test and saw a 'significant' difference. What does that tell you?You can't randomly assign users for ethical or product reasons, but you need to estimate the effect of a treatment. Walk me through your options.A/B test results split sharply by country — positive in one, negative in another. Walk me through how you'd interpret that.Your test is meant to improve free-to-paid conversion. What metrics would you not want to see move, even if conversion goes up?You're running multiple experiments simultaneously and they touch overlapping user segments. What do you do about it?How would you measure the long-term value of a one-time promotion, beyond the immediate revenue lift?Your A/B test shows a positive effect overall, but power-users see a clearly negative effect. How do you make the ship decision?Your team agrees the feature should ship regardless of the result. Should you still run the experiment?Walk me through the difference between a noisy null result and a real null. How do you tell them apart?You suspect your treatment is heterogeneous. How would you design the experiment to capture that?You want to test a feature affecting drivers in a rideshare service. What randomization scheme prevents network spillover?How would you design an experiment to test a feature where the effect is expected to compound over time, like learning the product?Two of your experiments contradict each other on the same metric. Walk me through how you'd reconcile them.When would you intentionally run an underpowered experiment, knowing you can't detect a typical effect?You're testing a new notification feature to increase daily active users. Write out the null and alternative hypotheses you'd use for this experiment.Your baseline conversion rate is 8% and you want to detect a 10% relative improvement. Roughly how many users do you need per variant?You're testing a change to the search results page. Should you randomize by user, by search query, or by session? Explain your choice.Your experiment shows strong gains in the first three days, then returns to baseline. What are two possible explanations?Your A/B test result is statistically significant but the effect size is smaller than expected. How would you communicate this to your PM?Before launching an experiment on button color, what should you define beyond just the primary metric?You have 100,000 daily active users and want to run a two-week test. Is this enough to detect a 5% change in a 20% baseline metric?You're testing a feature where users can invite friends. Why might user-level randomization cause problems here?Your new feature test shows declining engagement week-over-week, but adoption is still increasing. What might explain this pattern?Your test shows no significant difference, but the confidence interval is very wide. What does this tell you and what would you do?A PM asks you to test whether 'users like the new design better.' How would you reframe this into a testable hypothesis?Your experiment reached statistical significance on day 2 of a planned 14-day test. Should you stop early? Why or why not?Your offline evals say a new prompt for your LLM summarization feature beats the current one. The PM asks why you'd bother with an online A/B test at all. What's your answer?You're A/B testing two LLM-backed variants whose outputs differ run to run, even on identical inputs. What does that non-determinism do to your experiment design and your read of the results?Your team proposes an LLM-as-judge quality score as the primary metric for an assistant A/B test. What would you validate before letting it decide the ship call?Explain to your PM what CUPED-style variance reduction buys an experiment — and describe a situation where it buys you almost nothing.Halfway through a test you notice the treatment arm has 4% fewer users than the 50/50 split should produce. Your PM says that's close enough — keep going. What's your call?Your org ships a dozen small AI features a quarter and every individual A/B test wins. How would you use a long-term holdout to check the cumulative story, and what makes holdouts hard to keep honest?You need to compare two search rankers but your product only gets a few thousand queries a day. How would you decide between an interleaved design and a classic A/B?Your PM wants to ship a checkout redesign without an experiment — 'the old version is obviously worse.' What's your call, and what evidence would change it?Your primary metric is flat, but three of twenty secondary metrics are significant at p < 0.05 — all positive. The team wants to ship on the strength of those three. How do you respond?Your test's primary metric is revenue per user, and a handful of whales dominate it. What does that skew do to your experiment, and what are your options?Your company launched a feature to all users the week of Black Friday and revenue rose 30%. Why can't you credit the feature, and what setup would have let you?A feature-flag bug showed the treatment to your control group for one full day of a 14-day test. What are your options for handling the analysis?You're A/B testing a new prompt for an AI feature, and the model provider upgrades the underlying model halfway through the test. Why is that a problem for your results?A more capable model wins your quality A/B but costs five times more per request. What else has to be true before you ship, and how do you put cost into the experiment readout?You're testing an AI-powered recommendation engine that learns from user behavior over time. How does the model's learning affect your experiment duration and when you lock the model?Your baseline conversion rate is 2.3% and you need to detect a 10% relative lift. Traffic is limited to 50K users per week. Walk me through your power calculation and timeline.You're launching a new chat feature in a B2B product where conversations span multiple users across the same company account. What's your randomization strategy?An LLM-powered feature shows different results every time a user retries the same query. How does this non-determinism affect your experiment design and analysis?Your experiment shows statistically significant results on day 3, but your pre-registered plan said to run for two weeks. What factors determine whether you stop early?You're designing an experiment for a pricing change in a two-sided marketplace. How do you prevent price information from leaking between treatment and control groups?Your stakeholder wants to test five different headline variations simultaneously. What are the trade-offs of a multi-arm test versus sequential A/B tests here?A fintech fraud detection model is tested and shows 15% fewer false positives but 3% more false negatives. How do you frame the success criteria before running this experiment?You're testing a notification feature that could trigger user invites to non-users. What contamination risks exist and how would you mitigate them?An AI content moderation tool is 10x faster than human review but has unknown accuracy. What experiment would you design to validate it before broader rollout?Your experiment on enterprise customers requires company-level randomization, but you only have 200 companies. Is this experiment feasible and what adjustments would you make?You see a 20% lift in your guardrail metric that should never move. What are the most likely causes and how do you investigate?A product manager wants to test 'improving search relevance.' What specific hypotheses and metrics would you propose before writing any experiment plan?Your test shows that new users love the feature but existing power users hate it. The overall metric is flat. How do you interpret this and what do you recommend?You're testing a same-day delivery option in a logistics network where fulfillment centers serve overlapping geographic areas. What's your randomization approach?An AI summarization feature reduces time-on-page by 40% but increases return visits by 25%. How would you have framed the success metric upfront?Your control group is accidentally exposed to the treatment for 6 hours on day 4 of a 14-day test. How does this affect your analysis and what are your options?You need to test a feature during the holiday shopping season when traffic patterns are anomalous. How does seasonality change your experimental approach?A social feature creates network effects where a user's experience depends on how many of their friends are also in treatment. How do you design around this?Your AI model's performance degrades when user behavior shifts post-launch. How would you design an experiment that accounts for this feedback loop?You're asked to test a feature that only affects 2% of users who complete a rare action. What sample size challenges do you face and how do you address them?Week one shows a 5% revenue lift, week two shows 2%, week three shows -1%. All results are statistically significant. What's your interpretation?Your mobile app experiment needs to account for users who switch between the app and web. What's your randomization strategy to maintain consistency?A pharmaceutical trial protocol requires 80% power to detect a 15% reduction in adverse events at p<0.01. Walk me through what this means for trial design.You're testing an AI code completion tool for developers. The model improves with usage data, but you need a stable baseline. How do you structure the experiment?