Jailbreak escape rate estimation

Your deployment refuses a category of request. The refusal boundary is enforced by a model you did not train plus filters you did. The safety evaluation suite passes, meaning everything the team knew to test is handled. You now owe a release review a number: what fraction of adversarial traffic gets through in production, and why anyone should believe it.

Design the estimator.

The available exits are closed. You may not retrain the model or fine tune the filters, since you integrate rather than train. You may not make a red team the source of the number. You may treat past mitigations and their eventual bypasses as historical data, but you may not treat the coverage of any search procedure as a measurement of the boundary, and this exercise does not involve producing new bypasses. You may not use a stronger trusted model as an adjudicator. You may not answer with a categorical claim, because a release review that hears zero stops listening.

Hand in a statistic with a defensible interpretation: what it estimates, over what population, and what a bound from it does and does not assert. State the distributional assumption it rests on and why that assumption is the weak point, given that the traffic you care about is chosen by someone who reads your mitigation notes. Then give the experiment. Describe how you would check the estimator's calibration when its whole purpose is predicting failures on classes absent from the data you calibrated on, and say what a calibration result would look like if the estimator were tracking similarity to the evaluation suite rather than tracking the boundary. Ties break toward the estimator that reports a usefully wide interval it can defend over a tight number it cannot.