Lesson 9 of 9
Overfitting
Give a model enough freedom and it will describe any history perfectly. What that costs, how to see it, and what an honest backtest reports beside its result.
By Javier Sanz
Founder of ValueScreener. Built Ninety Nine, a securities services provider, and sold it to a tier-one global bank; ran operations and internationalization at Alpaca, the brokerage API.
Who writes thisEditorial processLinkedIn (opens in a new tab)X (opens in a new tab)
In one minute
- Every free choice in a rule is a degree of freedom, and enough of them will fit any history exactly.
- In-sample performance measures how well a rule describes the data it was built on, which is not a test.
- The only honest test is a period the rule was never allowed to see, and that period can be used once.
A rule that beat the market in a backtest has passed a test it was built to pass. Whether it would have passed a test it had never seen is a separate question, and it is the only one that matters.
What each choice in a rule costs
- 1Which indicators116 available, and any combination of them
- 2Which thresholdsEvery level each one could be cut at
- 3Which universeSize, sector, exchange, exclusions
- 4Which periodStart date, end date, whether a crisis is inside it
- 5A rule that fits the historyChosen because it fit, and reported as if it were asked first
Degrees of freedom
A degree of freedom is any choice the researcher was free to make. Choosing the indicator is one. Choosing the level to cut it at is another. Choosing the period, the universe, the rebalancing interval and the weighting are four more.
With enough of them, fitting the past is not evidence of anything. It is arithmetic. The rule is not discovering a pattern in the history; it is being shaped until it matches one.
In-sample against out-of-sample
In-sample performance is measured on the data used to build the rule. It is always at least as good as reality and usually better, and it is what a backtest reports by default.
Out-of-sample performance is measured on data the rule never touched. It is the only figure that carries information, and it is available exactly once. Look at the out-of-sample result, adjust the rule, and test again, and that period has become in-sample too.
The tells
Four signs that a reported result is describing its own construction.
A very specific threshold. A rule that cuts at 14.7 rather than 15 was tuned. The extra precision came from the data, not from a reason.
A period with unusual boundaries. A test starting in 2003 rather than 2000 has excluded something, and the something is usually a drawdown.
Many conditions on little data. Six conditions tested over fifteen annual observations has more freedom than evidence.
No statement of how many rules were tried. The most important omission, and the easiest to notice once you look for it.
What an honest backtest reports
Five things beside the return. How many rules were tried. What the universe was and whether it included companies that later delisted. Whether the figures used were the ones available at the time or the ones known now. What the transaction costs were assumed to be. And the out-of-sample result, if there is one, with the date the holdout was first examined.
A result without these is not wrong. It is unverifiable, which for a claim about money is the same problem in a politer form.
The narrower case this product runs
A retro-test of a stated condition against one company's own filed years is not a return backtest. It checks whether a written condition held or broke, year by year, against filed figures. It reports no return and claims none.
It is not immune either. Choose the condition after seeing which one would have held, and you have fitted it to that company's past exactly as surely. The defence is the same one this whole course argues for: write the condition down first.
Check yourself
A backtest reports a strong return in-sample. What has been established?
- That the rule works
- That the rule describes the data it was built on
- That the rule will decay after publication
Show the answer
Answer: B.
In-sample performance is a measure of fit, not a test. A rule shaped until it matched a history will match that history, and that is the result being reported.You test a rule on a holdout period, then adjust the rule and test again. What has happened to the holdout?
- Nothing, it is still a valid test
- It has become in-sample, because the rule was shaped by its result
- It becomes valid again after a year
Show the answer
Answer: B.
A holdout carries information only while the rule has never been influenced by it. One look and one adjustment spend it, which is why it can be used once.Which is the strongest sign that a published rule was fitted?
- It uses more than one indicator
- It does not say how many rules were tried
- It was tested on US data
Show the answer
Answer: B.
Without the number of attempts, the reported significance cannot be interpreted at all. The other tells, oddly precise thresholds and unusual period boundaries, are supporting evidence.Sources
- Bailey, Borwein, Lopez de Prado and Zhu, "Pseudo-Mathematics and Financial Charlatanism", Notices of the AMS, 2014, on reporting the best of many tried rules without stating how many were tried.
- Harvey, Liu and Zhu, "...and the Cross-Section of Expected Returns", Review of Financial Studies, 2016, on the bar a field with hundreds of tested factors requires.
- McLean and Pontiff, "Does Academic Research Destroy Stock Return Predictability?", Journal of Finance, 2016, on the decline in published predictors' returns after publication.
GET /api/stats/universeon ValueScreener, read 16 September 2026: 116 indicators computed per company, the size of the indicator choice in the figure.
What this leaves out
- The three-row table is illustrative: it describes the shape of an honest and a fitted report, not any particular study.
- This lesson is about how to read a backtest. It is not a backtest, and this product does not publish one.
- Distinguishing a fitted result from a real one is not always possible from the outside, which is why the reporting requirements matter more than any test a reader can apply afterwards.
- Nothing here is advice about any security.