Rank agents on a rubric with partial credit and ties

An eval harness runs several versions of a support agent on the same task suite and ranks them from their transcripts. The ranking decides which version ships, so two versions that did equally well must tie, and the arithmetic must not invent a winner. Write rank_agents(suite, transcripts, weights).

• suite: the tasks, each {"task": id, "tools": [...], "keywords": [...]}. tools lists the tool calls a good run makes, in order; keywords lists what the final answer must mention. Each task has at most 4 tools and at most 4 keywords (possibly none), keywords are distinct ignoring case, and task ids are unique. The suite has at least one task.
• transcripts: each {"agent": name, "task": id, "calls": [...], "answer": str, "unsafe": bool}: the tool names the agent called, in order, its final answer, and whether a safety check flagged the run.
• weights: [w_tools, w_answer], positive integers of at most 10.

Score of one transcript, a fraction from 0 to 1:

• Tool credit: the largest k such that the task's first k tools appear in calls in that order, with any other calls allowed in between, divided by the number of tools. A task with no tools gives credit 1.
• Answer credit: how many keywords appear in the answer as a substring, ignoring case, divided by the number of keywords. A task with no keywords gives credit 1.
• Score = (w_tools * tool_credit + w_answer * answer_credit) / (w_tools + w_answer).
• An unsafe transcript scores 0, whatever else it did.

Score of an agent: the mean of its scores over every task in the suite. A task it has no transcript for scores 0. If it has several transcripts for one task, the last one in the list counts. Transcripts for tasks that are not in the suite are ignored, but their agent is still ranked.

Ranking: higher score first. Agents with exactly equal scores share a rank and are listed by name in plain string order (by character code, so uppercase before lowercase); the rank after a tie skips (1, 1, 3). Scores are exact fractions, and the tie test must be exact.

Return [[name, rank, score], ...] with score as a float.

Example, with weights [1, 1] and this suite:

{"task": "t1", "tools": ["search", "read"], "keywords": ["refund", "30 days"]}
{"task": "t2", "tools": ["lookup"], "keywords": ["shipped"]}

rank_agents(suite, transcripts, [1, 1]) returns [["A", 1, 1.0], ["B", 2, 0.5], ["C", 2, 0.5], ["D", 4, 0.0]]. For B on t1, search appears in the calls but read does not appear after it, so the tool credit is 1/2.