Experiment report for a pricing page test

DesignEasy30 min

Your task

Design the data flow for testing two versions of a software company's pricing page. Visitors are split 50/50 between the current page and a new one, each visitor must keep seeing the same version on every visit, and the team wants a daily report of how many visitors in each version started a free trial and how many became paying customers within 14 days of starting it. About 40,000 visitors reach the pricing page a day, most of them not logged in, and trials and payments are recorded in the company's main database. Describe how visitors are assigned, what is logged, how the report is computed, and how you would notice that the split itself is broken.

It covers

  • Assignment hashing
  • Exposure logging
  • Sample ratio check