Reasoning faithfulness verification

A model emits a chain of reasoning and then an answer. You need a per instance verdict on whether the stated chain is the computation that produced the answer, or a plausible account assembled alongside it. This goes into an audit trail, so a claim that the reasoning is load bearing will be relied on by people who cannot read the weights.

Off the table: fine tuning the model toward faithfulness, treating a second model's judgment that the chain looks reasonable as ground truth, and asking the model whether it used its own reasoning. All three are circular in the same way, they certify the artifact under test using the artifact under test. You may run the model as often as you like, you may edit prompts and chains, and if your design needs activations then say so and pay for that access in your cost account.

The trap is that the naive intervention, corrupt the chain and see whether the answer changes, cannot distinguish dependence on the content of the reasoning from the model reacting to a prompt that now looks damaged. Your design has to address that, not route around it.

Hand in an operational definition of faithfulness that does not bottom out in another judgment call, a test procedure with a null hypothesis and a stated false positive rate, and the control that separates content dependence from off distribution reaction. Then hand in the experiment that would refute your own test, including a case you expect to be faithful and a case you expect to be unfaithful, and say how you obtained those expectations without begging the question.

Ties break toward the design that gives per instance verdicts rather than population rates, and toward the one whose definition survives a model that reaches a right answer for a reason it cannot verbalize.