How I killed my own backtest

The model works from 60 features. 12 of them (possession, shots per 90, save percentage, fouls — home/away/difference) come from an fbref table that holds a single row per team per season. That row, however, is the full-season aggregate — it only exists once the season is over.

row = fbref_df[(fbref_df["Squad"] == team) & (fbref_df["season"] == season)]
if len(row):
    return row.iloc[0]

So when predicting an October match, the model already “knew” how the team would perform through May. The rolling stats and squad-change features were clean — only this one join leaked, and that was enough to make the whole result false.

The measurement compared four variants on the same walk-forward setup (2018/19–2022/23, model retrained each season, 17,356 matches), with the same decision rule and three different random seeds. Switch the leak off and see what's left of the edge.

The point isn't the exact numbers — those scatter within ±1% — but the sign flip: the leaky variant sits above +6% on all three seeds, every clean variant sits around zero. That's not noise, it's a mechanism.

The measurement solved another mystery too. Live, the model performed worse than in the backtest, and I had put that down to “model shrinkage”. In reality the live system had no fbref row for the current season yet, so it used the previous season's data — that's the V2 state, at −0.86%.

So I trained the model in V1 and ran it in V2. Classic train/serve skew: the model didn't break in production — the training was lying.

The fix wasn't deleting a line; it was rules that catch the next bug of this kind too:

• xfail(strict=True) test

  A test specifically detects the leak. While the bug exists it runs as an “expected failure”; once someone fixes it, the test goes red until the marker is removed on purpose. The bug can't disappear silently.

• mandatory time window

  The ledger can only be read with a mandatory window argument — no default. You can't load the full dataset “by accident”.

• fail-closed closing price

  Only a pre-kickoff snapshot counts as the closing price. If there isn't one, the row is dropped — not back-filled with a later (in-play) price.

• locked holdout + variant counter

  A set-aside, locked dataset can only be touched in an approved round, and every variant tried is counted. Above 10 variants the threshold gets stricter; above 20 the search stops.

The measurement was run by an AI coding agent (Claude Code), within rules set in advance. This is how we worked:

1. 01It measured on a separate branch, inside the search window (before 2023-07-01); it didn't touch the locked holdout.
2. 02The task was measurement, not fixing — the report has a separate “what this does NOT claim” section.
3. 03It checked the claim with three seeds before calling it not-noise.
4. 04I reviewed and merged the branch; the fix became a separate, dated decision.
5. 05The rule stays: the agent writes code and tests; only a human touches production, money and the main branch.

• All ROI figures are at raw Pinnacle prices, on bets filtered by the decision rule — not general model accuracy.
• One hyperparameter set, three seeds, five seasons. The sign flip is robust; the ±1% differences are not.
• I'm not claiming the model is beyond saving — only that with the current feature set, without leakage, there's no detectable edge on this sample.
• This is a research project, not a tipping service.

A pretty backtest is a suspect until the leakage test is green.

That has been one of the project's ground rules ever since. The most expensive bug isn't the one that crashes — it's the one that quietly returns good numbers.