Code RoomDetect targeted model behavior
FrontierPrep Room Coding #4982

Detect targeted model behavior

System designDistributed systemsSenior–Staff~75 min

You are accepting a checkpoint from outside your organization into a product path. The provenance half is already handled and is not the exercise: the artifact is signed, its hash is pinned, the serving stack builds reproducibly, and you hold a bill of materials. All of that establishes you received exactly what the publisher sent. None of it says anything about what the publisher's training run put into the weights.

The concern is a targeted behavior: on a narrow condition chosen by whoever influenced the training data, the model does something different. The condition can be semantic rather than lexical, for instance a property of the code under review or an attribute of the user carried in context, so there is no string to grep for.

Design the acceptance program that decides whether this artifact ships, and be honest about what a pass means. You are not asked to construct such a behavior or to describe how one would be implanted. Keep adversary capability abstract and spend the session on detection, on what the detector cannot see, and on what you do at serving time because of that.

Off the table: retraining, because you have neither the data nor the compute. Reproducing the publisher's run, for the same reason. A labelled set of known bad checkpoints to fit a detector on, because that is the assumption behind most published detectors and it fixes in advance which families you can catch. Assuming the trigger is a fixed token string.

Hand in the acceptance procedure and its cost, the assertion a pass actually makes, written so a reviewer cannot read more into it than you meant, the false negative class in shape, the compensating controls at serving time that assume detection already failed, the condition that would make you withdraw an artifact already deployed, and the experiment showing your detector responds to implanted behavior rather than to the ordinary quirks any fine tune leaves behind. Ties break toward the program that admits low power and compensates over the one claiming a clean bill.

What a strong answer looks like

Clarify scale and constraints first. Propose a clean component breakdown, then go deep on the hard parts (data model, bottlenecks, consistency, failure modes) and name the trade-offs you are making.

Clarify8:30 left
Estimate8:30 planned
Design25:00 planned
Deep dive20:00 planned
Failure13:00 planned
0:00
Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.