AI-assisted engineering/Agent evaluation

Calibrate an LLM judge's scores against human labels

Agent buildHard40 min

Your task

An LLM judge scores your agent's replies from 0 to 1,000, and people have labelled a sample of the same replies as good or bad. You want a calibration table that turns a judge score into the chance a person calls the reply good. A higher judge score must never get a lower chance.

Write calibrate(scores, labels) that returns the table.

Input and output:

scores: the judge's score for each labelled reply, up to 100,000 whole numbers from 0 to 1,000.
labels: the person's label for the same replies, in the same order: 1 for good, 0 for bad.
Return a list of [score, chance] pairs, one for each different score in scores, in increasing order of score. Each chance is a float, and the tests accept any value within 0.000001 of the exact one.

Rules:

Replies with the same judge score get the same chance.
The chances never go down as the score goes up. They may stay level.
Of all the tables that follow those rules, return the one with the smallest total squared error. That error is the sum, over every labelled reply, of the square of its label minus the chance for its score. Exactly one table has the smallest error.
With no replies, return an empty list.

Examples

calibrate(scores: list[int], labels: list[int]) -> list[list]

Example 1

Inputcalibrate([55, 55, 90, 20, 55, 90, 20, 20, 20, 90, 90, 55], [1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 1, 0])

Output[[20, 0.25], [55, 0.5], [90, 0.75]]

Example 2

Inputcalibrate([60, 30, 60, 80, 30, 80], [0, 0, 0, 1, 1, 1])

Output[[30, 0.25], [60, 0.25], [80, 1]]

Example 3

Inputcalibrate([700, 300, 700, 300, 500], [1, 0, 0, 1, 1])

Output[[300, 0.5], [500, 0.6666666666666666], [700, 0.6666666666666666]]