Score an agent's majority vote over k sampled answers
Your task
Your agent answers each task by sampling several answers and returning the most common one, picking at random among the most common when they tie. You logged a set of sampled answers for every task. You want the exact chance that a vote over k of them gets the task right, without running the agent again.
Write majority_at_k(tasks, k) that returns the mean of that chance over all tasks.
tasks: a list of[gold, answers].goldis the right answer andanswersis the list of logged answers for the task. Answers are strings, compared exactly. A task has at most 30 answers and at leastk.k: a whole number, at least 1.- Return a float. The tests accept any value within 0.000001 of the exact one.
Rules:
- A vote draws
kdifferent entries from the task'sanswerslist, and every set ofkentries is equally likely. Different entries can hold the same string. - The vote returns the answer that appears most often in the draw. When several answers tie for the most, it returns one of them, each equally likely.
- A task's chance is the probability that its vote returns
gold. Return the mean over all tasks, or0.0when there are no tasks.
Examples
majority_at_k(tasks: list[list], k: int) -> floatExample 1
Input
majority_at_k([["42", ["42", "42", "41"]]], 1)Output
0.6666666666666666Example 2
Input
majority_at_k([["42", ["42", "42", "41"]]], 3)Output
1Example 3
Input
majority_at_k([["yes", ["yes", "no", "yes", "no"]]], 2)Output
0.5