Score an agent's majority vote over k sampled answers

Agent buildHard40 min

Your task

Your agent answers each task by sampling several answers and returning the most common one, picking at random among the most common when they tie. You logged a set of sampled answers for every task. You want the exact chance that a vote over k of them gets the task right, without running the agent again.

Write majority_at_k(tasks, k) that returns the mean of that chance over all tasks.

  • tasks: a list of [gold, answers]. gold is the right answer and answers is the list of logged answers for the task. Answers are strings, compared exactly. A task has at most 30 answers and at least k.
  • k: a whole number, at least 1.
  • Return a float. The tests accept any value within 0.000001 of the exact one.

Rules:

  1. A vote draws k different entries from the task's answers list, and every set of k entries is equally likely. Different entries can hold the same string.
  2. The vote returns the answer that appears most often in the draw. When several answers tie for the most, it returns one of them, each equally likely.
  3. A task's chance is the probability that its vote returns gold. Return the mean over all tasks, or 0.0 when there are no tasks.

Examples

majority_at_k(tasks: list[list], k: int) -> float
Example 1
Inputmajority_at_k([["42", ["42", "42", "41"]]], 1)
Output0.6666666666666666
Example 2
Inputmajority_at_k([["42", ["42", "42", "41"]]], 3)
Output1
Example 3
Inputmajority_at_k([["yes", ["yes", "no", "yes", "no"]]], 2)
Output0.5