{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "eff9d693",
   "metadata": {},
   "source": [
    "# Backtesting & Validation Lab\n",
    "\n",
    "Companion notebook to [Data Science Module 5](https://rodoslay.com/data-science/05-backtesting-and-validation) at **rodoslay.com**.\n",
    "\n",
    "Three experiments with the truth known: (1) random k-fold vs. purged walk-forward on a drifting\n",
    "credit panel — the leakage, measured in AUC; (2) the **multiple-testing lottery** — the best of 200\n",
    "coin-flip strategies backtests beautifully; (3) **vintage curves** where ranking survives and\n",
    "calibration quietly dies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f14fba1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "rng = np.random.default_rng(15)\n",
    "plt.rcParams[\"figure.figsize\"] = (10, 4.5)\n",
    "plt.rcParams[\"axes.grid\"] = True\n",
    "plt.rcParams[\"grid.alpha\"] = 0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67448108",
   "metadata": {},
   "source": [
    "## 1. K-fold vs purged walk-forward on drifting data\n",
    "\n",
    "40,000 loans over 40 months. Feature effects **drift** (one coefficient decays, another grows),\n",
    "and labels resolve over a 10-month horizon driven by a **common monthly shock** — so nearby loans\n",
    "share outcome information, exactly the label-overlap problem purging exists for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99f541d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "N, N_MONTHS, H = 40_000, 40, 10\n",
    "month = rng.integers(0, N_MONTHS, N)\n",
    "X = rng.standard_normal((N, 4))\n",
    "\n",
    "# drifting coefficients + common monthly shocks inside each loan's outcome window\n",
    "beta1 = 1.2 - 1.6 * month / N_MONTHS            # decays, then flips sign\n",
    "beta2 = 0.2 + 1.0 * month / N_MONTHS            # grows\n",
    "shocks = rng.normal(0, 0.55, N_MONTHS + H)      # macro month effects\n",
    "window_shock = np.array([shocks[m:m + H].mean() for m in month])\n",
    "\n",
    "logit = -2.6 + beta1 * X[:, 0] + beta2 * X[:, 1] + 0.5 * X[:, 2] + 1.3 * window_shock\n",
    "y = (rng.random(N) < 1 / (1 + np.exp(-logit))).astype(int)\n",
    "print(f\"default rate {y.mean():.2%}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cbafdbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval_kfold():\n",
    "    aucs = []\n",
    "    for tr, te in KFold(5, shuffle=True, random_state=0).split(X):\n",
    "        m = LogisticRegression(max_iter=1000).fit(X[tr], y[tr])\n",
    "        aucs.append(roc_auc_score(y[te], m.predict_proba(X[te])[:, 1]))\n",
    "    return np.mean(aucs)\n",
    "\n",
    "def eval_walkforward(purge=True):\n",
    "    aucs = []\n",
    "    for te_start in [20, 24, 28, 32, 36]:\n",
    "        te = (month >= te_start) & (month < te_start + 4)\n",
    "        tr_end = te_start - (H if purge else 0)         # purge: drop overlap with test labels\n",
    "        tr = month < tr_end\n",
    "        m = LogisticRegression(max_iter=1000).fit(X[tr], y[tr])\n",
    "        aucs.append(roc_auc_score(y[te], m.predict_proba(X[te])[:, 1]))\n",
    "    return np.mean(aucs)\n",
    "\n",
    "# the honest benchmark: deploy at month 30. At that moment, only loans from months < 30-H\n",
    "# have RESOLVED labels — you physically cannot train on anything more recent.\n",
    "tr = month < 30 - H; te = month >= 30\n",
    "true_next = roc_auc_score(y[te], LogisticRegression(max_iter=1000)\n",
    "                          .fit(X[tr], y[tr]).predict_proba(X[te])[:, 1])\n",
    "\n",
    "print(f\"random 5-fold CV AUC:               {eval_kfold():.4f}   <- what the naive report says\")\n",
    "print(f\"walk-forward (no purge) AUC:        {eval_walkforward(purge=False):.4f}   <- trains on labels that didn't exist yet\")\n",
    "print(f\"walk-forward (purged) AUC:          {eval_walkforward(purge=True):.4f}\")\n",
    "print(f\"deploy-and-wait AUC (months 30+):   {true_next:.4f}   <- what production will actually deliver\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da818f90",
   "metadata": {},
   "source": [
    "The ordering is the module's thesis: random CV reads highest (it trained on the future);\n",
    "the unpurged walk-forward still flatters, because its most recent training loans carry labels that\n",
    "would not have resolved yet; the **purged walk-forward is the estimate that matches what deployment\n",
    "actually delivers** — both are fitted on the same information a real deployment would have. The gap\n",
    "between the first and last lines is the performance you would have promised and not delivered.\n",
    "\n",
    "## 2. The multiple-testing lottery"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f35fbb52",
   "metadata": {},
   "outputs": [],
   "source": [
    "T = 252 * 4                                     # 4 years of daily returns\n",
    "strategies = rng.normal(0, 0.01, (200, T))      # 200 strategies of PURE NOISE\n",
    "sharpes = strategies.mean(axis=1) / strategies.std(axis=1) * np.sqrt(252)\n",
    "\n",
    "best = np.argmax(sharpes)\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))\n",
    "ax1.hist(sharpes, bins=30, color=\"#1f4d3a\", alpha=0.8)\n",
    "ax1.axvline(sharpes[best], color=\"#b3341e\", ls=\"--\", label=f\"best: {sharpes[best]:.2f}\")\n",
    "ax1.set_title(\"Sharpe ratios of 200 coin-flip strategies\"); ax1.legend()\n",
    "ax2.plot(strategies[best].cumsum(), color=\"#b3341e\")\n",
    "ax2.set_title(\"The winner's equity curve — publishable, and pure luck\")\n",
    "plt.tight_layout(); plt.show()\n",
    "\n",
    "fresh = rng.normal(0, 0.01, T)                  # the same 'strategy' out of sample = new noise\n",
    "print(f\"best in-sample Sharpe: {sharpes[best]:.2f}   expected out-of-sample: 0.00\")\n",
    "print(f\"P(best of 200 > 1.0) by simulation: high — you will 'find' a strategy every time you search\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdfa71aa",
   "metadata": {},
   "source": [
    "## 3. Vintage curves: ranking survives, calibration dies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ac6e340",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reuse the drifting panel: model trained on months 0..19, then watch vintages age\n",
    "m0 = LogisticRegression(max_iter=1000).fit(X[month < 20], y[month < 20])\n",
    "p = m0.predict_proba(X)[:, 1]\n",
    "\n",
    "rows = []\n",
    "for v0 in range(0, 36, 4):\n",
    "    vmask = (month >= v0) & (month < v0 + 4)\n",
    "    if vmask.sum() < 500: continue\n",
    "    rows.append([f\"m{v0:02d}-{v0+3:02d}\", vmask.sum(),\n",
    "                 roc_auc_score(y[vmask], p[vmask]) if 0 < y[vmask].mean() < 1 else np.nan,\n",
    "                 p[vmask].mean(), y[vmask].mean()])\n",
    "vint = pd.DataFrame(rows, columns=[\"vintage\", \"n\", \"AUC\", \"mean predicted PD\", \"realized default\"])\n",
    "vint[\"calibration gap\"] = vint[\"realized default\"] - vint[\"mean predicted PD\"]\n",
    "vint.round(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2d1d69e",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(vint[\"vintage\"], vint[\"mean predicted PD\"], \"o-\", label=\"predicted\", color=\"#1f4d3a\")\n",
    "ax.plot(vint[\"vintage\"], vint[\"realized default\"], \"o-\", label=\"realized\", color=\"#b3341e\")\n",
    "ax.set_title(\"Vintage calibration: the model's PDs quietly detach from reality\")\n",
    "ax.legend(); plt.xticks(rotation=45); plt.tight_layout(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a98a6cc",
   "metadata": {},
   "source": [
    "AUC drifts down gently — but the calibration gap opens decisively as coefficients rotate away\n",
    "from the training regime. A Gini-only monitoring pack would call this model healthy while its PDs\n",
    "misprice every vintage. (This is [Module 6's](https://rodoslay.com/data-science/06-production-ml-for-risk)\n",
    "opening argument.)\n",
    "\n",
    "## Exercise: the broken backtest\n",
    "\n",
    "A colleague reports: *\"We shuffled five years of loan data into 10-fold CV, tuned 60 XGBoost\n",
    "configurations against the folds, picked the best, then confirmed it on the same folds. CV AUC\n",
    "0.83. The 12-month default label was built with current balances. We recommend deployment.\"*\n",
    "\n",
    "Find the problems. (Count carefully — the module says four; there are five.)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}