Split a run budget to steady an eval score
Your task
Your eval suite's score is the average of its tasks' scores, and a task's score moves from run to run because the agent samples its answers. Averaging several runs of a task steadies its score, and a jumpy task needs more runs than a steady one. Each run is graded by a script, so runs are cheap, but the budget is fixed.
Write least_noise(spreads, budget) that returns the smallest noise any split of the budget reaches.
Input and output:
spreads: each task's spread from past runs, 1 to 10,000 whole numbers from 0 to 100. A spread is the standard deviation of one run's score, in score points: how far a run typically lands from the task's average.budget: the total number of runs, a whole number from the number of tasks to 1,000,000,000.- Return a float. The tests accept any value within 0.000001 of the exact one.
Rules:
- Each task runs a whole number of times, at least once, and the runs add up to at most
budget. - A task with spread
sthat runsrtimes addss * s / rto the noise. That is the variance of its averaged score. - A split's noise is the sum over all tasks. Return the smallest noise any split reaches.
Examples
least_noise(spreads: list[int], budget: int) -> floatExample 1
Input
least_noise([20, 10], 4)Output
233.33333333333334Example 2
Input
least_noise([30, 10, 0], 7)Output
275Example 3
Input
least_noise([40, 40, 40], 3)Output
4800