Model approval rate doubles overnight

On-call incidentHard35 minCapital One style

Your task

An ops alert fires: your loan-approval model's approval rate jumped from 22% to 61% in two hours with no model change. Dashboards: the model serves 200s, latency normal. Digging in, the feature 'annual_income' is now arriving as 0 or null for ~70% of applicants since 13:00, and the model treats low/zero income with a feature default that happens to push scores toward approval. An upstream data provider changed a JSON field name ('income' to 'incomeAnnual') in a 13:00 release, so your ingestion silently maps the old key to null. Triage and respond.

It covers

  • Data quality
  • Null poisoning
  • Upstream schema
  • Validation