When the world changes faster than your AI's underlying math.
You train a model in 2019 to predict airline ticket prices. It works perfectly. In 2020, a global pandemic hits, and flights drop to $10. Your model, however, confidently predicts $500. This is Concept Drift—the fundamental relationship between the inputs (date, destination) and the output (price) has shifted. Even worse, the model is uncalibrated: it tells you it is "99% confident" in its $500 prediction. A well-calibrated model would recognize the inputs are weird and output a confidence of 10% instead.
Machine Learning models in production decay over time. To handle drift, you need monitoring pipelines that trigger automatic retraining when statistical metrics drop. To handle overconfidence, you use Calibration (like Platt Scaling or Isotonic Regression) which maps the raw, overconfident model scores to true probabilities.
from sklearn.calibration import CalibratedClassifierCV
from sklearn.ensemble import RandomForestClassifier
base_model = RandomForestClassifier()
# BAD: The raw model will often output "0.99" confidence
# even when it's just guessing on new data.
base_model.fit(X_train, y_train)
# GOOD: Wrap the model in a Calibrator.
# It holds back some validation data to learn how "overconfident"
# the base model is, and scales the outputs to match reality.
# If it says 0.80, there is genuinely an 80% chance it's true.
calibrated_model = CalibratedClassifierCV(base_model, method='sigmoid')
calibrated_model.fit(X_train, y_train)
real_probability = calibrated_model.predict_proba(X_new)Calibration requires setting aside a portion of your training data purely for the calibrator to learn on, leaving less data for the actual model to learn patterns. Fixing drift requires constant, expensive retraining jobs and humans-in-the-loop to verify that the new data isn't just a temporary glitch (like a 1-day system outage).