Classification Metrics Done Right

Credit modeling has its own dialect of model evaluation — Gini and KS on this side of the office, AUC and F1 on that side — and the translation errors are a real source of bad decisions. This module sorts the zoo into what each metric actually measures, shows why the most popular metric in machine learning (accuracy) is close to meaningless on default data, and separates the two virtues people constantly conflate: ranking borrowers correctly and calibrating their probabilities correctly.

The setup: one picture, every metric

Every binary scoring model reduces to the same picture: two score distributions — goods and bads — that overlap. Every metric in this module is a different summary of that overlap. So start with the picture:

Run the experiment that motivates the whole module before reading on: set separation to 1.5σ, threshold wherever, and drag the base rate from 50% down to 2%. Watch which numbers move and which don’t — then watch accuracy climb as the model’s usefulness collapses. Everything below is an unpacking of that slider.

The ranking family: AUC, Gini, KS

These three answer the same question — does the model order bads above goods? — and none of them care about the base rate or your cutoff.

AUC (area under the ROC curve) has the cleanest probabilistic meaning: pick a random bad and a random good; AUC is the probability the bad has the higher score. 0.5 is a coin flip; 1.0 is perfect separation. It’s threshold-free — the ROC curve traces every possible cutoff, and AUC integrates over all of them.

Gini is credit’s house convention: $\text{Gini} = 2 \times \text{AUC} - 1$. Nothing more. A “Gini of 55” is an AUC of 0.775. It rescales coin-flip to 0, which is why scorecard shops like it — improvements read more honestly (AUC 0.75 → 0.80 sounds like nothing; Gini 50 → 60 sounds like what it is). When someone quotes an “accuracy ratio” from a CAP curve, that’s Gini again — same number by a different construction.

KS (Kolmogorov–Smirnov) is the maximum vertical gap between the goods’ and bads’ cumulative score distributions — the single best cutoff’s worth of separation. It answers a narrower question than AUC: not “how good is the ranking overall” but “at the best split point, how different are the populations?” That’s genuinely useful when a single cutoff decision is the point (Module 2’s approve/decline line), and it’s why KS survives in credit while barely appearing elsewhere. Its weakness is the mirror image: it ignores everything away from the optimum, so two models with equal KS can differ a lot in the tails — where your worst 5% of applicants live.

Rules of thumb for bureau-style application scorecards: Gini below ~30 is weak, 40–60 is the workhorse zone, 70+ on out-of-time data should trigger suspicion of leakage (DS Module 2’s lesson) before celebration.

Why accuracy lies

Accuracy — share of correct predictions — imports an assumption that’s false in credit: that both error types matter equally and both classes are comparably common. At a 3% default rate, the strategy “approve everyone” is 97% accurate and 0% useful. Any model chasing accuracy learns to say “good” everywhere, because the data rewards exactly that.

The honest lens under imbalance is the precision–recall pair: of the accounts you flagged, how many were really bad (precision)? Of the truly bad, how many did you catch (recall)? Precision is where the base rate bites — with 2% bads, even a strong model produces mostly false alarms at any aggressive cutoff, and the collections team feels that, not the AUC. The PR curve makes visible what the ROC curve compresses: ROC’s false-positive rate is diluted by the huge good population, so an ROC curve can look glorious while precision sits at 20%. Same model, same scores — different question. For rare-event work, look at both, but decide with PR-style quantities (or, better, expected cost).

One more honesty note: F1 is a convenience, not a law. It weights precision and recall equally, which encodes a cost assumption nobody in credit believes (a missed default costs principal; a false alarm costs a phone call). If you know the costs even roughly, optimize expected cost at the cutoff directly — that’s what the cutoff is for.

Calibration: the other virtue entirely

Ranking metrics cannot see whether your probabilities mean anything. A model that outputs 0.9 for every bad and 0.8 for every good has a perfect AUC and absurd probabilities. Calibration asks: of the loans you scored 4%, did about 4% actually default?

Credit cares about calibration more than almost any other field, because the PD’s level — not just its ranking — flows into pricing (Module 1’s EL), provisioning (IFRS 9), and capital (the IRB formula). A mis-calibrated model with a great Gini approves the right people at the wrong price.

The toolkit: reliability diagrams (bin by predicted PD, plot predicted vs. realized — the picture), the Brier score (mean squared error of probabilities — a single number blending calibration and discrimination), and recalibration maps — Platt scaling (logistic) or isotonic regression (monotone, more flexible) fitted on a holdout. Recalibration is cheap, preserves the ranking exactly (so Gini is untouched), and is mandatory whenever training used a balanced or over-sampled dataset — the base-rate distortion of Module 3 of the credit track (Zmijewski’s complaint) reincarnated in ML clothing.

Which metric for which job

The per-segment column deserves its own sentence: portfolio-level Gini can hide a model that ranks beautifully for thick-file borrowers and flips a coin on thin files. Slice before you sign.

Where this connects

• The scores being evaluated come from Module 2’s features and credit Module 2’s scorecards; champion/challenger comparisons live and die by these metrics.
• Out-of-time evaluation — the only honest kind for drifting data — is the subject of Module 5, next.
• Metric decay in production, and when it should trigger retraining, is Module 6.
• Calibrated PDs are what credit Modules 9 and 8 consume; a Gini improvement that ruins calibration is a net loss there.

Try it yourself

The notebook below builds every metric from scratch on a synthetic credit dataset — ROC and PR curves traced by hand before calling sklearn, Gini three ways (AUC transform, CAP curve, Somers’ D) to prove they agree, KS with its cutoff, reliability diagrams before and after isotonic recalibration, and the imbalance experiment: the same model evaluated at 50%, 10%, and 2% base rates, with a table of which metrics moved and which stood still. The closing exercise hands you two models — one with better Gini, one with better calibration — and a pricing rule, and asks which one makes more money. (It’s not the Gini.)

Get new posts by email

One email per new article. No spam, no upsells, unsubscribe anytime.