Test whether a new agent version really regressed
Your task
You changed your agent's prompt and reran the same eval tasks. Each task gets a score from 0 to 10, and the new version's total came out lower. Before you roll back, you want to know how often a drop this big turns up by chance when the change did nothing.
Write regression_p_value(before, after) that returns that chance.
Input and output:
before,after: the old and the new version's scores, as two lists of the same length, up to 300 whole numbers from 0 to 10. Indexiin both lists is the same task.- Return a float. The tests accept any value within 0.000001 of the exact one.
Rules:
- A task's difference is its
afterscore minus itsbeforescore. The observed total is the sum of the differences. - If the change did nothing, each difference was as likely to come out with the opposite sign. So take every way of keeping or flipping the sign of each difference: 2 to the power
nways forntasks, all equally likely. A difference of 0 stays 0 either way, and both ways still count. - Return the share of those ways whose total is at most the observed total. With no tasks, return 1.0.
Examples
regression_p_value(before: list[int], after: list[int]) -> floatExample 1
Input
regression_p_value([7, 8, 6, 9, 7], [5, 8, 6, 7, 6])Output
0.125Example 2
Input
regression_p_value([6, 5, 7, 8], [7, 4, 7, 9])Output
0.875Example 3
Input
regression_p_value([], [])Output
1