Model monitoring without ground truth

DesignHard40 minCapital One and American Express style

Your task

Design a production monitoring system for a credit-risk model that decides loan approvals. The catch: the true label (did the borrower default?) only arrives 6-18 months after the prediction, so you cannot measure accuracy in real time. The business needs early warning if the model starts degrading or behaving unfairly. Design what you monitor in the absence of ground truth, how you detect drift, and how you'd catch a silent failure (e.g. an upstream feature pipeline breaks) within minutes, not months.

It covers

  • Drift
  • Delayed labels
  • Alerting