Fix the metrics for a rare disease classifier

Review the AI's codeEasy10 min

Your task

An AI evaluated a rare disease classifier, with about 1.5% positive cases, using accuracy alone. It printed 98.6% and called the model "highly performant", but a model that says "negative" to everyone scores 98.5%. The fix reports precision, recall and the majority baseline next to accuracy.

An assistant wrote evaluate_rare_class(y_true, y_pred). It is in the editor. It has a bug. Find it and fix it.

Input and output:

  • y_true: the real labels. y_pred: the model's predictions, at the same index.
  • Return a list of four numbers: accuracy, precision, recall and baseline.

Rules the tests check:

  • If the lists differ in length, use only the first n pairs, where n is the shorter length. If n is 0, return [0, 0, 0, 0].
  • A label is positive only when it is exactly 1. Every other value is negative.
  • Accuracy: the share of pairs where the prediction and the real label are both positive or both negative.
  • Precision: of the pairs predicted positive, the share that are really positive. 0 if nothing is predicted positive.
  • Recall: of the pairs that are really positive, the share predicted positive. 0 if there are no real positives.
  • Baseline: the accuracy of always predicting the larger class, which is the larger of the positive and negative counts, divided by n.
  • Answers within 0.000001 of the expected values pass.

Examples

evaluate_rare_class(y_true: list[int], y_pred: list[int]) -> list[float]
Example 1
Inputevaluate_rare_class([1, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
Output[0.9, 0, 0, 0.9]
Example 2
Inputevaluate_rare_class([1, 1, 0, 0, 0, 0, 0, 0, 0, 0], [1, 0, 0, 0, 0, 0, 0, 0, 0, 1])
Output[0.8, 0.5, 0.5, 0.8]
Example 3
Inputevaluate_rare_class([1, 0, 1, 0], [1, 0, 1, 0])
Output[1, 1, 1, 0.5]