Split an agent's eval score across its parts
Your task
Your agent has several parts you can switch on or off, such as a planner, a memory and a few tools. You ran the eval once for every combination of parts and recorded each run's score. Now you want to credit the full agent's score to its parts fairly, so each part gets its average gain over every order in which the parts could be switched on.
Write part_credit(scores) that returns each part's credit.
Input and output:
scores: the eval score for every combination ofmparts, formfrom 1 to 12, so the list holds 2 to the powermwhole numbers from 0 to 10,000.scores[mask]is the score with exactly the parts whose bits are set inmaskswitched on: partiis the bit1 << i.- Return a list of
mfloats, part 0 first. The tests accept any value within 0.000001 of the exact one.
Rules:
- Picture switching the parts on one at a time, in some order, starting with none on. A part's gain in that order is the score just after it is switched on minus the score just before.
- A part's credit is its gain averaged over all the orders of the
mparts, each order counted once.
Examples
part_credit(scores: list[int]) -> list[float]Example 1
Input
part_credit([40, 70])Output
[30]Example 2
Input
part_credit([40, 55, 60, 90])Output
[22.5, 27.5]Example 3
Input
part_credit([20, 20, 20, 80])Output
[30, 30]