Fix the threshold picker for a rare defect model
Your task
An AI trained a defect detector on data with 1 defect in 200, using an unweighted loss and a fixed 0.5 threshold. The loss fell smoothly, and the model almost always predicted "no defect". Part of the fix is to choose the decision threshold on validation data by F1, a score that balances precision and recall and ignores the easy true negatives.
An assistant wrote select_f1_threshold(scores, labels, thresholds). It is in the editor. It has a bug. Find it and fix it.
Input and output:
scores: the model's score for each validation row.labels: each row's label, at the same index: 1 for a defect, anything else for none. The two lists have the same length.thresholds: the thresholds to try, in order.- Return the best threshold, or -1 if there are no candidates.
Rules the tests check:
- At a threshold, a row is predicted positive when its score is at least the threshold.
- Count the true positives (predicted positive, label 1), the false positives (predicted positive, label not 1) and the false negatives (not predicted positive, label 1).
- F1 is 2 times the true positives, divided by 2 times the true positives plus the false positives plus the false negatives. If that denominator is 0, F1 is 0.
- Return the threshold with the highest F1. On a tie, return the one that comes first in
thresholds.
Examples
select_f1_threshold(scores: list[float], labels: list[int], thresholds: list[float]) -> floatExample 1
Input
select_f1_threshold([0.1, 0.2, 0.05, 0.4], [0, 1, 0, 1], [0.5, 0.3, 0.15])Output
0.15Example 2
Input
select_f1_threshold([0.2, 0.7], [0, 0], [0.9, 0.5])Output
0.9Example 3
Input
select_f1_threshold([0.42, 0.38, 0.44, 0.05, 0.02], [1, 0, 1, 0, 0], [0.5, 0.4])Output
0.4