Lead calibration drifts over-confident

On-call incidentMedium30 min

Your task

Your lead-scoring model outputs a calibrated probability that sales uses with a fixed 0.7 threshold to decide which leads to call. A 'calibration drift' alarm fires: a reliability-diagram check shows predicted probabilities are now systematically over-confident, leads the model says are 0.8 convert at only ~0.55 in reality, and this has worsened over the past three weeks. AUC/ranking quality, however, is essentially unchanged (the model still orders leads well). No model deploy in 50 days. Dashboards: the overall positive base rate has slowly fallen (market softened) while feature distributions are roughly stable. How do you triage and respond?

It covers

  • Calibration
  • Probability drift
  • Threshold
  • Recalibration