Lesson 7 of 7
Why backtests mislead
A backtest's return can be real and still mean little, once you count the rules that were tried and quietly dropped before the one that worked.
By Javier Sanz
Founder of ValueScreener. Built Ninety Nine, a securities services provider, and sold it to a tier-one global bank; ran operations and internationalization at Alpaca, the brokerage API.
Who writes thisEditorial processLinkedIn (opens in a new tab)X (opens in a new tab)
In one minute
- Testing many rules and publishing only the one that worked can make a useless rule look like a real discovery.
- A backtest run on survivors only, or using information not actually available at the time, looks better than a real, live test would have gone.
- Published return predictors earned less after publication, partly because the original result was partly luck and partly because investors traded on it.
A backtest runs a rule against history and reports what it would have earned. That number can be entirely real and still mean little, if the rule, the history, or the costs used to build it were chosen after seeing how well they worked.
How a backtest becomes a weaker, published result
- 1A rule is foundBy searching many candidate rules against history
- 2Tested on the same dataThe data used to find it in the first place
- 3PublishedAs a discovery, with its best-looking version reported
- 4Run by others, after costsOften weaker, sometimes gone entirely
Testing many rules finds patterns that are not there
Try enough rules against the same stretch of history and some of them will look like discoveries purely by chance, whether or not they have any real relationship to future returns. Testing many rules is not wrong by itself. Bailey, Borwein, Lopez de Prado and Zhu made the point that follows from it: reporting only the best rule out of many tried, without saying how many were tried, inflates how impressive that one rule looks.
Survivors and hindsight both flatter the past
A backtest run only on companies that still exist today leaves out every company that was delisted or went bankrupt during the period. A real investor at the time would have been exposed to those companies. A backtest that uses a figure the way it looks after a later restatement, rather than the way it was actually reported at the time, is testing information nobody had yet. A backtest that uses year-end figures before the 10-K was filed, up to 90 days later, has the same problem: it tests information nobody had yet. Both problems make a historical rule look steadier than a live version of it would have been.
Factors weaken once they are known
McLean and Pontiff studied a large set of published return predictors. Their returns tended to decline once the papers describing them were public, compared with the returns in the original study's own sample. Harvey, Liu and Zhu made a related point about the field as a whole. With hundreds of proposed factors already tested across finance research, a much higher statistical bar is needed before treating a new one as real, because some fraction of that large number will clear an ordinary bar by chance alone.
What a filed-based check is not
A retro-test of a published rule against one company's own ten years of filed figures is not a return backtest. It checks whether a condition would have held true or false, year by year, on real filed numbers. It does not report what a portfolio built on the rule would have earned. It is not immune either: choose a condition after seeing which one held, and you have fitted it to the past.
Check yourself
You test 20 rules with no real predictive power, each at a 5 percent significance threshold. On average, how many should appear to work by chance alone?
- Zero
- About one
- All twenty
Show the answer
Answer: B.
20 multiplied by 0.05 is 1. Testing many rules and reporting only the one that worked is exactly how a useless rule can look like a discovery.A backtest's universe includes only companies that still exist today. What does that leave out?
- Companies with a high market capitalization
- Companies that were delisted or went bankrupt during the tested period
- Companies that pay a dividend
Show the answer
Answer: B.
This is survivorship bias: the companies most likely to have hurt a real, live portfolio are exactly the ones missing from a survivors-only test.McLean and Pontiff found that published return predictors, after the papers describing them became public, tended to...
- Perform about as well as in the original study
- Perform noticeably worse than in the original study
- Only stop working outside the United States
Show the answer
Answer: B.
Their study found a decline in returns after publication, consistent with either the original result being partly a statistical artifact or with trading eroding a real edge once it became known.Sources
- Bailey, Borwein, Lopez de Prado and Zhu, "Pseudo-Mathematics and Financial Charlatanism," Notices of the AMS, 2014.
- Harvey, Liu and Zhu, "...and the Cross-Section of Expected Returns," Review of Financial Studies, 2016.
- McLean and Pontiff, "Does Academic Research Destroy Stock Return Predictability?," Journal of Finance, 2016.
What this leaves out
- Real trading costs, spreads and market impact are often left out of published backtests, and they fall hardest on strategies that trade often.
- The twenty-rule example is a simplified illustration of the underlying arithmetic, not a real published study's result.
- Nothing here is advice about any security, and this page is not itself a backtest of any strategy.