Lesson 5 of 9
Multiple testing
This product computes 116 indicators per company. Turned into simple rules that is 232 tests, and about 12 of them clear a 5 percent bar on noise alone.
By Javier Sanz
Founder of ValueScreener. Built Ninety Nine, a securities services provider, and sold it to a tier-one global bank; ran operations and internationalization at Alpaca, the brokerage API.
Who writes thisEditorial processLinkedIn (opens in a new tab)X (opens in a new tab)
In one minute
- A 5 percent threshold means one test in twenty clears it when nothing is there. That is a feature until you run many tests.
- The number of rules available in an ordinary screener is large enough that several will look significant on noise alone.
- The defence is to say how many things were tried, before the result is quoted, and to raise the bar accordingly.
A single test at a 5 percent threshold is a reasonable instrument. The same instrument used two hundred times, with only the best result reported, produces findings out of nothing. The arithmetic is not subtle and the practice is everywhere.
Testing 232 rules that have no real effect at all
- Rules tested232
- Expected to clear a 5 percent bar on noise alone11.6
The count is not hypothetical
GET /api/stats/universe reports 116 indicators computed per company on this
product. Turn each into the simplest possible rule, a threshold crossed in one
direction or the other, and that is 232 distinct rules before anyone gets
creative.
At a 5 percent threshold, 232 multiplied by 0.05 is 11.6. So if not one of the 116 indicators had any relationship to future returns, about twelve of the 232 rules would still look significant, and nothing in any individual test result would say which twelve.
The chance that at least one clears is 1 minus 0.95 raised to the 232nd power, which is 0.9999932. Finding something is not evidence. Finding something is guaranteed.
And 232 is the small number
What this does to a reported result
A published rule arrives with a p-value attached to the rule that was kept. The tests that were run and discarded do not appear anywhere, and without them the p-value cannot be interpreted.
Two researchers can report the identical rule with the identical p-value of 0.01. One tried a single pre-specified rule. The other tried four hundred. The first result is meaningful and the second is expected, and the paper looks the same either way.
The defences, and their cost
Say how many were tried. The cheapest and most effective. A result reported with "this was the only rule tested" and a result reported with "this was the best of four hundred" are different claims, and the reader can adjust.
Raise the bar for the number of tests. Dividing the threshold by the number of tests is the blunt version: for 232 tests a 5 percent family-wide bar becomes 0.000216 per test. It is conservative and it is honest about what searching costs.
Hold data back. Find the rule on one period, then test it once on a period never looked at. The second test has not been searched, so its p-value means what it says. This costs data and it costs the temptation to look, and it is the only defence that survives a determined searcher.
The one that does not work
Testing until something clears, then presenting that thing as if it were the first question asked. This does not require dishonesty. It is what a curious person does with a screener on an afternoon, and it is why the number of things tried has to be written down before the search begins rather than remembered afterwards.
Check yourself
You test 232 rules at a 5 percent threshold, and none of them has any real effect. About how many clear the bar?
- None
- About 12
- About 116
Show the answer
Answer: B.
232 multiplied by 0.05 is 11.6. The threshold is defined as the rate at which pure noise clears it, so a large enough search produces winners by construction.Two papers report the same rule with the same p-value of 0.01. One tried 1 rule, the other 400. What differs?
- Nothing, the p-value is the same
- The second result is roughly what 400 tests produce by chance, and the first is not
- The second has a larger sample
Show the answer
Answer: B.
A p-value describes one test. It cannot account for the tests that were run and discarded, so the number tried has to be reported alongside it for the number to be interpretable.Which defence survives a researcher who keeps looking?
- Reporting the p-value to more decimal places
- Holding out data and testing the final rule once on a period never examined
- Using a larger sample for the same search
Show the answer
Answer: B.
An untouched holdout has not been searched, so a test on it means what it says. A larger sample does not help, because the search is run over the larger sample too.Sources
GET /api/stats/universeon ValueScreener, read 16 September 2026:indicators_per_stockof 116, the figure the rule count is derived from.- Harvey, Liu and Zhu, "...and the Cross-Section of Expected Returns", Review of Financial Studies, 2016, on the higher statistical bar a field with hundreds of tested factors requires.
- Bailey, Borwein, Lopez de Prado and Zhu, "Pseudo-Mathematics and Financial Charlatanism", Notices of the AMS, 2014, on reporting the best result of many without saying how many were tried.
What this leaves out
- The 232 and 6,670 counts assume the tests are independent, and they are not: this product's indicators share inputs, so several of them move together. Dependence reduces the effective number of independent tests, which makes the expected false positive count an upper bound rather than a point estimate. It does not remove the problem, and the direction of the error is towards fewer than 11.6 rather than towards none.
- The 116 figure is what this product computes per company. It is used here as a concrete stand-in for the size of an ordinary search space, not as a claim about any other screener.
- Nothing here is advice about any security, and nothing here is a backtest of any rule.