Rank agent versions against the fewest judge verdicts

Agent buildHard40 min

Your task

An LLM judge compares agent versions in pairs on the same tasks and picks a winner each time. Its verdicts are not always consistent: A can beat B, B beat C, and C beat A. The release review still needs one ranking.

Write rank_versions(versions, verdicts) that returns the versions in ranked order, best first.

  • versions: the version names, at most 10, all different. Names use only letters, digits, . and -.
  • verdicts: a list of [winner, loser] pairs, one per judge comparison. The same pair can appear many times, in either order.
  • Return a list that holds every version once.

Rules:

  1. A ranking goes against a verdict when it puts the loser above the winner.
  2. Return a ranking that goes against as few verdicts as possible.
  3. If several rankings tie, return the first one. Compare rankings name by name from the top, in plain string order: by character code, so B sorts before a.
  4. Ignore a verdict that names a version not in versions, or names the same version twice.

Examples

rank_versions(versions: list[str], verdicts: list[list[str]]) -> list[str]
Example 1
Inputrank_versions(["v1", "v2", "v3"], [["v1", "v2"], ["v2", "v3"], ["v1", "v3"]])
Output["v1", "v2", "v3"]
Example 2
Inputrank_versions(["a", "b", "c"], [["a", "b"], ["b", "c"], ["c", "a"]])
Output["a", "b", "c"]
Example 3
Inputrank_versions(["x", "y"], [["y", "x"], ["x", "y"], ["y", "x"]])
Output["y", "x"]