Test whether a new agent version really regressed

Agent buildMedium25 min

Your task

You changed your agent's prompt and reran the same eval tasks. Each task gets a score from 0 to 10, and the new version's total came out lower. Before you roll back, you want to know how often a drop this big turns up by chance when the change did nothing.

Write regression_p_value(before, after) that returns that chance.

Input and output:

  • before, after: the old and the new version's scores, as two lists of the same length, up to 300 whole numbers from 0 to 10. Index i in both lists is the same task.
  • Return a float. The tests accept any value within 0.000001 of the exact one.

Rules:

  • A task's difference is its after score minus its before score. The observed total is the sum of the differences.
  • If the change did nothing, each difference was as likely to come out with the opposite sign. So take every way of keeping or flipping the sign of each difference: 2 to the power n ways for n tasks, all equally likely. A difference of 0 stays 0 either way, and both ways still count.
  • Return the share of those ways whose total is at most the observed total. With no tasks, return 1.0.

Examples

regression_p_value(before: list[int], after: list[int]) -> float
Example 1
Inputregression_p_value([7, 8, 6, 9, 7], [5, 8, 6, 7, 6])
Output0.125
Example 2
Inputregression_p_value([6, 5, 7, 8], [7, 4, 7, 9])
Output0.875
Example 3
Inputregression_p_value([], [])
Output1