Backtesting & Walk-Forward Validation

The most expensive mistake in financial machine learning isn’t a bad model — it’s a good-looking evaluation of a bad model. A model that fails honestly in validation costs you a few weeks. A model that passes a broken validation goes to production, books loans or takes positions for a year, and fails with money attached. This module is about the specific ways financial validation breaks, and the schemes that fix it.

Why the textbook method trains on the future

K-fold cross-validation — shuffle the data, split into folds, rotate the test fold — is correct when observations are exchangeable: when nothing about row 500 tells you it came “after” row 200. Financial data is the opposite of exchangeable. Time order carries regimes (Module 3’s whole subject), drifting relationships, and — in credit — outcome windows that stretch months into the future.

Shuffle that, and every fold trains on the future of its own test set. The model gets to learn 2025’s patterns and then be examined on 2024. Look at what the folds actually do:

Select random k-fold and note the violation counter — then switch to expanding walk-forward and watch it go to zero. That counter is the entire argument of this module, made visual. Two distinct sins get mixed together in the k-fold case, and it’s worth separating them because they have different fixes:

1. Regime leakage — training on later periods whose patterns (rate environment, competitive conditions, borrower mix) simply didn’t exist at the test date. The fix is chronology: walk-forward schemes.
2. Label-overlap leakage — subtler and nastier. A credit label like “defaults within 12 months” (DS Module 2’s performance window) means an observation dated January contains information through the following January. A training row from November and a test row from December share most of an outcome window — same recession, same portfolio events. They are not independent, no matter how carefully you ordered them. The fix is surgical: purging (drop training observations whose label windows overlap the test set) plus an embargo (a further buffer after the test block, for serial correlation that outlives the window). Toggle it in the widget and watch the grey gap appear — that gap is the price of honesty, paid in training data.

The walk-forward family

All honest schemes share one property — training always precedes testing — and differ on one question: how much past do you keep?

• Expanding window: train on everything up to time $t$, test on the next block, roll forward. Maximum data, and the setup that most closely mimics production (you will use all available history when you deploy). Its bias: old regimes never leave the training set — 2015’s physics keeps voting on 2026’s model.
• Rolling window: train on a fixed-width recent window only. Adapts to regime change and doubles as a stress test — if performance collapses when the window drops an old crisis, you’ve learned your model’s signal was that crisis. Its price: permanently less training data, noisier fits.
• Out-of-time (OOT) holdout: the credit-industry standard, and what a validator will demand first. Fit on everything through year $Y$, evaluate untouched on year $Y{+}1$. It’s a single walk-forward step — simple, blunt, and the closest simulation of “we built this last year; how is it doing now?” Serious shops report OOT and a multi-step walk-forward, because one OOT window is a sample of size one: it tells you how the model did in that particular year.

Which to choose? For credit scoring: OOT for the headline, expanding walk-forward with purging for model selection, and a rolling variant as a regime-sensitivity probe. For anything resembling a trading signal: walk-forward only, and keep reading.

Backtest overfitting: the multiple-testing machine

There’s a second, entirely separate way to manufacture a beautiful backtest, and no split scheme prevents it: try enough things. Test 100 random, useless strategies against the same historical data and the best one will look excellent — at a 5% significance level, five of them clear the bar by construction. Every hyperparameter sweep, every feature set you tried and discarded, every “let’s just check one more variant” is a draw from this lottery, and the final reported backtest silently conditions on having won it.

This is why the field developed a dark statistic: the probability of backtest overfitting, deflated Sharpe ratios, and the rule of thumb that a backtested Sharpe should be mentally halved before believing it. You don’t need the full machinery to act on the insight:

• Count your trials — honestly, including the ones you abandoned. The more you searched, the higher the bar for the winner. (The notebook makes this visceral: the best of 200 coin-flip strategies backtests with a Sharpe above 1.5.)
• Hold out a final, untouched period — one you evaluate once, after all decisions are frozen. The moment you iterate against it, it’s spent — it has become training data for your intuition.
• Prefer fewer, theory-motivated variants over broad searches. A model that must exist for an economic reason (the credit track’s whole philosophy) starts with a better prior than the champion of a 500-model sweep.
• Report the distribution across walk-forward steps, not just the mean. A strategy that made all its money in one window is a story about that window.

Vintage analysis: credit’s native backtest

Credit has its own time-honest evaluation format worth knowing by name. A vintage is a cohort of loans booked in the same period; vintage analysis tracks each cohort’s cumulative default (or loss) curve by months-on-book, then compares curves across vintages. It answers the questions walk-forward metrics compress: Is the 2025-Q3 vintage tracking worse than 2024-Q3 at the same age? Did the new scorecard actually improve the vintages booked under it? Is deterioration coming from underwriting drift or the macro environment (all vintages bending together)? A model can hold its Gini while vintages deteriorate — ranking survives, calibration dies (Module 4’s distinction, in time-series form). Plot both.

Where this connects

• Purging exists because of the performance windows defined in Module 2; the regimes that make chronology matter are Module 3.
• The metrics computed inside each walk-forward step are Module 4’s — Gini for ranking, reliability for calibration, per step and per vintage.
• What happens after validation — monitoring the deployed model as reality drifts away from the development sample — is Module 6, next.
• The champion/challenger discipline of credit Module 2 is walk-forward validation running live, with real money as the test set.

Try it yourself

The notebook below quantifies every claim: a drifting synthetic credit panel where random k-fold overstates AUC versus purged walk-forward (the gap is the leakage, measured); the purge/embargo implementation from scratch with the label-overlap counter; the multiple-testing lottery — 200 noise strategies, the winner’s deflated glory, and the honest correction; and a vintage-curve builder that shows a stable-Gini model failing on calibration vintage by vintage. The final exercise hands you a broken published-style backtest and asks you to find all four things wrong with it. (There are five.)

Get new posts by email

One email per new article. No spam, no upsells, unsubscribe anytime.