Production ML for Risk: Monitoring, Drift, and Retraining

Everything in this track so far happens before deployment. This module is about the longer, less glamorous life that starts after: a risk model in production, scoring real applicants, while the world that generated its training data quietly stops existing. The core fact of production ML in credit is brutal and simple — you will know the inputs have changed long before you know the outputs are wrong — and a monitoring stack is how you act on the early signal instead of waiting for the expensive one.

How models die

Three distinct decay modes, worth separating because they demand different responses:

• Covariate drift — the population changes; the relationship doesn’t. A marketing push brings in younger applicants; a competitor exits and their rejects flood your funnel; a new channel skews thin-file. The model may still be correct on each applicant while its portfolio-level behavior (approval rates, score mix, calibration at the book level) shifts.
• Concept drift — the relationship itself changes. Payment-deferral programs make “missed payment” mean something new; a rate shock changes what 60% utilization implies; fintech credit builders alter what a two-year history is worth. Same inputs, different outcomes: the model is now wrong per applicant, and no reweighting fixes it.
• Label and pipeline drift — the boring killers. A bureau changes a field’s definition; an upstream system starts writing nulls; the “bad” definition gets amended by a policy team that didn’t tell you. In postmortems of production model failures, plumbing beats mathematics with embarrassing regularity.

PSI: the workhorse early-warning metric

Credit’s standard drift detector is the Population Stability Index. Freeze the score distribution from development, bin it into deciles, and compare today’s population against it:

Same construction as Module 2’s Information Value — it is IV, repurposed: instead of separating goods from bads, it separates then from now. The conventions: below 0.10 stable, 0.10–0.25 watch closely, above 0.25 investigate before trusting the model. Feel what moves it:

Three experiments. A pure mean shift of 0.3σ — barely visible in the bars — already pushes PSI toward the warning zone; the metric is sensitive by design, because small average shifts often hide larger segment shifts. Widening dispersion alone trips it too — same average applicant, different mix. And the third slider is the realistic scenario: a new segment flowing in at the risky end. Notice how modest inflow trips the alarm and the worst-bin readout points exactly at where to look — PSI’s real value isn’t the number, it’s the decomposition by bin telling you which part of the population is new.

Run the same statistic per feature (CSI, characteristic stability) and you get triage: score PSI says whether something changed; feature-level CSIs say what. Score stable but three features drifting = compensating changes worth understanding anyway. Score drifting with all features stable = check the pipeline, someone changed a mapping.

The monitoring stack, by latency

The defining constraint of credit monitoring is the outcome lag: today’s approvals earn their 12-month labels a year from now (Module 5’s performance window, now working against you). So the stack organizes by how fast each signal becomes available:

Two rows deserve commentary. Override rates — how often humans reverse the model — are an underrated drift detector: underwriters collectively noticing something the model can’t see shows up here first, and a rising override rate is your staff filing a bug report through their behavior. And early delinquency is the compromise between waiting for real labels and flying blind: 30-days-past-due at three months on book correlates strongly enough with eventual default to serve as a leading indicator of model failure — track it by vintage against prior cohorts at the same age (Module 5’s vintage curves, running live).

Retraining: policy, not panic

The naive instinct — “drift detected, retrain now” — fails in credit for a specific reason: you can only retrain on labeled data, and labels lag by the performance window. Retraining “on the latest data” means training on loans booked 12–24 months ago — often further from today’s regime than your current model’s sample. Retraining during a drift episode can make the model worse.

A sane retraining policy has three parts:

1. Cadence + triggers. A scheduled annual redevelopment (the validator’s expectation anyway) plus defined triggers: sustained score PSI above threshold, calibration drifting beyond tolerance on matured vintages, Gini decay beyond its historical noise band. Triggers prompt investigation — retraining is one possible outcome, not the reflex. If drift is a pipeline bug, retraining launders the bug into the model.
2. Recalibration before redevelopment. Most decay shows up as calibration drift (Module 4’s distinction earning its keep): ranking holds while PD levels shift with the cycle. A recalibration — refitting the score-to-PD mapping on recent matured outcomes — is cheap, low-risk, and often buys a year. Full redevelopment is for when ranking degrades or the feature set itself has gone stale.
3. Shadow first. A retrained challenger scores everything in parallel, silently, for months before promotion — credit Module 2’s champion/challenger discipline is precisely a production safety mechanism. Compare stability, calibration on early outcomes, and decision swaps (which applicants flip, not just how many) before any cutover.

Governance: monitoring is a model-risk requirement

None of this is optional in a bank. SR 11-7 makes ongoing monitoring one of the three pillars of model risk management — with documented metrics, thresholds agreed before deployment, escalation paths, and evidence the process runs. The practical translation: a monitoring plan written at development time (which metrics, what frequency, what thresholds, who gets paged, what happens on breach), a monthly pack a validator can audit, and threshold breaches that trigger documented investigations rather than quiet threshold edits. Alert discipline matters as much as alert coverage — a monitoring system that cries wolf weekly trains the team to ignore it, which is worse than no system, because it looks like control.

Where this connects

• PSI is Module 2’s IV pointed at time instead of default; the regimes doing the drifting are Module 3.
• The metrics being tracked — Gini, calibration — are Module 4; the vintage machinery is Module 5.
• The governance wrapper is credit Module 12’s SR 11-7 discipline; the PDs being monitored feed IFRS 9 and IRB capital, which is why calibration drift is a CFO problem, not just a data science one.

Try it yourself

The notebook below builds a monitoring system end to end on a simulated two-year production run: PSI/CSI from scratch with decile decomposition, a drift injector (mean shift, new segment, pipeline null-bug) to see each signature, Gini and calibration tracked on maturing vintages with the outcome lag made explicit, an override-rate simulation, and the retraining trap — a model retrained mid-drift on lagged labels that underperforms both the original and a simple recalibration. The closing exercise: write the monitoring plan (metrics, thresholds, cadence, escalation) for the model you built in Module 2’s notebook, as if a validator will read it. Because one will.

Get new posts by email

One email per new article. No spam, no upsells, unsubscribe anytime.