Lesson 4 of 9
What a p-value says
A p-value is the chance of seeing a result this extreme if nothing were there. It is not the chance nothing is there, and the difference is the whole subject.
By Javier Sanz
Founder of ValueScreener. Built Ninety Nine, a securities services provider, and sold it to a tier-one global bank; ran operations and internationalization at Alpaca, the brokerage API.
Who writes thisEditorial processLinkedIn (opens in a new tab)X (opens in a new tab)
In one minute
- A p-value is the probability of a result at least this extreme, computed on the assumption that there is no real effect.
- It is not the probability that there is no real effect, and swapping those two sentences is the most common error in applied statistics.
- A small p-value says nothing about how large the effect is, and a large one does not establish that there is none.
A p-value answers one narrow question. The trouble is that the question it answers and the question people want answered sound almost identical, and are not.
Two real comparisons, the same method, two outcomes
- Utilities against Consumer Defensive: difference in standard errors0.85
- Utilities against Financial Services: difference in standard errors3.53
The definition, exactly
Start by assuming there is no real difference. That assumption is the null hypothesis. Then ask: if it were true, how often would a measurement produce a gap at least as large as the one in front of me, purely through sampling variation.
That frequency is the p-value. It is a statement about data, computed inside an assumption, and every word of the definition is doing work.
The three things it is not
It is not the probability that the null hypothesis is true. The p-value was computed by assuming the null is true. A number derived inside an assumption cannot also be the probability of that assumption. Turning 0.0004 into "there is a 99.96 percent chance the sectors really differ" requires a prior, and Bayes, and is a different calculation.
It is not the probability the result was luck. This is the same error in friendlier words. The p-value says how often luck produces a result like this when nothing is there. It does not say how likely it is that nothing is there.
It is not a measure of size or importance. A p-value falls as the effect grows and also as the sample grows. With 50,000 observations a difference too small to act on will clear any threshold. The magnitude and the p-value are two separate outputs and both belong in the sentence.
Where 0.05 came from
Nowhere with authority. It is a convention that became a habit, and it means only that a result this extreme would arise one time in twenty when nothing is there. One in twenty is not rare when many things are being tested, which is the next lesson and the reason this one is not sufficient on its own.
How to read one honestly
State three numbers together: the effect, its standard error, and the p-value. An effect with no error beside it cannot be judged. A p-value with no effect beside it cannot be acted on. And the sentence that follows should be about data, not about truth: "a gap this large would be unusual if the sectors matched", not "the sectors are different".
Check yourself
A comparison returns p = 0.03. What does that mean?
- There is a 3 percent chance there is no real difference
- If there were no real difference, a result at least this extreme would occur about 3 percent of the time
- There is a 97 percent chance the effect is real
Show the answer
Answer: B.
The p-value is computed inside the assumption that there is no difference. It describes how often data like this arises under that assumption, and cannot also be the probability of the assumption.A study with 200,000 observations reports p less than 0.001 for a difference of 0.02 percent. What follows?
- The effect is large and important
- The effect is precisely measured and tiny
- The p-value must be wrong
Show the answer
Answer: B.
A p-value falls as the sample grows, whatever the size of the effect. Statistical significance and practical importance are different properties, and only the second one is about the world.Two sector means differ by 0.85 standard errors, giving a p-value of about 0.40. What is the honest conclusion?
- The two sectors have the same average
- The data does not distinguish them; both a real difference and none are consistent with it
- The comparison should be repeated until it reaches 0.05
Show the answer
Answer: B.
Failing to reject is not proving equality. The measurement was too imprecise to separate the two, which is a statement about the evidence and not about the sectors. Repeating until a threshold is cleared is the failure the next lesson is about.Sources
- Sector means, standard deviations and cohort sizes computed across active US equity listings with a positive P/E below 100, read from this product's own store on 16 September 2026: Utilities 67 listings, Consumer Defensive 98, Financial Services 471.
What this leaves out
- The p-values here come from a normal approximation to a two-sample comparison of means. They are printed to show the arithmetic, not as a published finding about any sector.
- More seriously, these cohorts are not random samples. They are entire sectors on one date, restricted to positive ratios below 100. A significance test asks how often a random sample would produce this gap, and there was no random sampling, so the number is a demonstration of the method rather than a valid inference.
- That limitation is why the lesson states what a p-value does not say twice as forcefully as what it does.
- Nothing here is advice about any security.