There is more of this guide.
Practice
24 questions- Experiment designA stakeholder wants to ship a change after one week of A/B data. How do you decide if you have enough?
- Experiment designYour stakeholder wants to detect a 0.5% improvement on a metric with a current baseline of 12%. Walk me through what you'd need.
- Experiment designYou only have access to 30% of traffic for this experiment. How does that change your design?
- Experiment designYour peer ran an A/A test and saw a 'significant' difference. What does that tell you?
- Experiment designYou're running multiple experiments simultaneously and they touch overlapping user segments. What do you do about it?
- Experiment designYou suspect your treatment is heterogeneous. How would you design the experiment to capture that?