Split a run budget to steady an eval score

Agent buildHard40 min

Your task

Your eval suite's score is the average of its tasks' scores, and a task's score moves from run to run because the agent samples its answers. Averaging several runs of a task steadies its score, and a jumpy task needs more runs than a steady one. Each run is graded by a script, so runs are cheap, but the budget is fixed.

Write least_noise(spreads, budget) that returns the smallest noise any split of the budget reaches.

Input and output:

  • spreads: each task's spread from past runs, 1 to 10,000 whole numbers from 0 to 100. A spread is the standard deviation of one run's score, in score points: how far a run typically lands from the task's average.
  • budget: the total number of runs, a whole number from the number of tasks to 1,000,000,000.
  • Return a float. The tests accept any value within 0.000001 of the exact one.

Rules:

  • Each task runs a whole number of times, at least once, and the runs add up to at most budget.
  • A task with spread s that runs r times adds s * s / r to the noise. That is the variance of its averaged score.
  • A split's noise is the sum over all tasks. Return the smallest noise any split reaches.

Examples

least_noise(spreads: list[int], budget: int) -> float
Example 1
Inputleast_noise([20, 10], 4)
Output233.33333333333334
Example 2
Inputleast_noise([30, 10, 0], 7)
Output275
Example 3
Inputleast_noise([40, 40, 40], 3)
Output4800