Reference-free answer grading

Your team runs the scoring service behind an internal evaluation platform. The inputs are long free form technical answers: design memos, incident write ups, research summaries. There is no reference answer for any of them, and for many there is no single correct one. The answers now routinely come from systems stronger in the subject area than anything you would use to grade them, so grading up rather than down is the normal case and not the edge case.

Off the table: reference answers, a second model you already trust to be more capable in the domain, human labels at scoring time, and any reward model trained on in domain preference data, because that data does not exist and the budget to collect it does not either. You do get a small audit budget of expert hours, spent once, to test the grader rather than to train it.

Design the scoring service. Say what it computes, what each component contributes, and where the score comes from when no component can check the claim. Then make the argument that matters: why the score moves with correctness and judgment rather than with length, confidence, structure, vocabulary or house style. A grader that reproduces a fixed ranking with high agreement is not evidence, because a fluency ranking is also perfectly self consistent.

Hand in the failure mode you accept, the population of answers on which your grader is knowingly wrong, and the experiment that would refute your identifiability argument. Say how the expert audit hours are spent to run it, and what result would make you turn the service off.

Ties break toward the design whose refutation experiment could actually come back negative, and toward the one that says plainly which part of the reported score is measurement and which part is preference.