Lesson 6 of 8
How many factors are real
Three researchers catalogued 316 published factors and argued that with that much searching, a t-statistic of 2 proves nothing. Their hurdle is 3.
By Javier Sanz
Founder of ValueScreener. Built Ninety Nine, a securities services provider, and sold it to a tier-one global bank; ran operations and internationalization at Alpaca, the brokerage API.
Who writes thisEditorial processLinkedIn (opens in a new tab)X (opens in a new tab)
In one minute
- Harvey, Liu and Zhu catalogued 316 factors from 313 published articles and working papers.
- When hundreds of candidates are tested, the usual significance bar admits a large number of results that are noise.
- Under their correction, of the 296 published significant factors, between 80 and 158 would be false discoveries depending on the method used.
Testing one idea and finding a significant result means something. Testing hundreds of ideas and reporting the significant ones means much less, and the finance literature has been doing the second for decades.
Published significant factors, and how many survive a correction for multiple testing
- Published significant factors296factors
- False discoveries under Bonferroni158factors
- False discoveries under Holm142factors
- False discoveries under BHY at 1 percent132factors
- False discoveries under BHY at 5 percent80factors
The problem, stated once
A test at the usual threshold accepts roughly a one in twenty chance of calling noise a finding. Run that test on one factor and the risk is small. Run it on hundreds, and a pile of significant-looking results arrives even if nothing is there at all.
The literature has run it on hundreds. Harvey, Liu and Zhu count 316 factors across 313 articles, 250 of them published, and note that their collection underrepresents the population: they only surveyed top journals, included a handful of working papers, and kept one representative version where a characteristic had many variants.
The hurdle they propose
Their conclusion is a number a reader can use. A new factor, they write, needs to clear a much higher hurdle: a t-statistic greater than 3.0, rather than the customary 2.0. They add the sentence that gives the paper its reputation: most claimed research findings in financial economics are likely false.
What it does and does not say about value
It does not say the value premium is false. Book-to-market is one of the oldest and most replicated results in the catalogue, tested across decades and across markets, and the drawdown lesson in this course measures it rather than doubting its existence.
What the paper does is set the price of admission for everything that came after. A factor published last year with a t-statistic of 2.3, found after searching a database of company characteristics, is a different kind of object from a relation documented in 1992 and re-examined by its own authors for thirty years.
The practical reading
Three questions to ask any published factor
-
How many candidates were searched before this one was reported. If the answer is not in the paper, the significance level in the paper is not the significance level of the finding.
-
Does it survive on data the author did not use. Out-of-sample decline is the subject of the previous lesson, and it is the first thing a mined result loses.
-
Is it one of hundreds explaining the same returns. Five principal components, the authors note, are hard to reconcile with the idea that there are truly hundreds of distinct factors.
The habit this builds is worth more than the specific hurdle. A reader who asks how many things were tried before a result was reported will discount most of what they read, and will discount it hardest exactly where the marketing is loudest. That is not cynicism about research. It is the correction the researchers themselves say their field needs, written by three of them in a journal that publishes the factors being counted.
Check yourself
What t-statistic hurdle do Harvey, Liu and Zhu argue a new factor should clear?
- 1.65
- 2.0
- 3.0
Show the answer
Answer: C.
They propose a hurdle of greater than 3.0 for a new factor, because the usual 2.0 does not account for how many candidates the field has tested.Of the 296 published significant factors, how many would be false discoveries under the Bonferroni correction?
- 80
- 132
- 158
Show the answer
Answer: C.
Their table gives 158 under Bonferroni, 142 under Holm, 132 under BHY at 1 percent and 80 under BHY at 5 percent.What does the multiple-testing problem imply about a single paper's significance level?
- It is always overstated by exactly half
- It describes one test, not the search that produced it, so it overstates the evidence when many candidates were tried
- It is irrelevant to finance
Show the answer
Answer: B.
A threshold designed for one test does not survive being applied to hundreds. What the reader needs to know is how many were tried, which papers rarely state.Sources
- Harvey, Liu and Zhu, "... and the Cross-Section of Expected Returns", Review of Financial Studies, 2016: 316 factors from 313 articles, the t-statistic hurdle of 3.0, and the counts of likely false discoveries under Bonferroni, Holm and BHY.
What this leaves out
- The false-discovery counts are the paper's own, under four named correction methods. They are estimates from a model, not a list of which factors are false.
- The catalogue covers top journals and a selection of working papers, so the true number of tested candidates is larger than 316.
- A correction for multiple testing says how much evidence a claim carries. It does not say which claims are true.
- Nothing here is advice about any security or strategy.