Why p-values over-estimate first order risk ?

 The short answer is:

👉 Because a p-value is computed conditional on the null hypothesis being true, it does not represent the probability of making a Type I error in the situation you are actually in. When it is interpreted as such, it systematically overstates (over-estimates) the “first-order risk”.

Below is the precise reasoning.

---

1. What “first-order risk” really is

The Type I error rate (first-order risk) is:

$$\alpha =P(\text{reject }{H}_0\mid H_0\text{ is true})$$

This is a long-run, pre-specified property of a decision rule (e.g. “reject if $p<0.05$”).

It is not a probability about the current experiment.

---

2. What a p-value actually is

A p-value is:

$$p=P(T\ge t_{\text{obs}}\mid H_0)$$

Key points:

• It is conditional on $H_0$ being true

• It is not $P(H_0\mid \text{data})$
  

• It is not $P(\text{Type I error})$
  

---

3. Where the over-estimation comes from

The common (incorrect) interpretation

“If $p=0.03$, there is a 3% risk that I am making a Type I error.”

This is false.

Why it over-estimates first-order risk

To make a Type I error in this experiment, two things must both be true:

1. $H_0$ is true

2. You rejected $H_0$
   

But the p-value already assumes (1) with probability 1.

The actual probability of a Type I error is:

$$P(H_0\mid \text{data})\times P(\text{reject}\mid H_0,\text{data})$$

Since:

$$P(H_0\mid \text{data})<1$$

the p-value necessarily exaggerates the chance of being wrong.

---

4. A simple Bayesian illustration

Suppose:

• Prior probability that $H_0$ is true: 0.5

• Observed p-value: 0.05

Under reasonable assumptions, the posterior probability that $H_0$ is true is often much larger than 0.05, typically 20–40%.

So:

• p-value = 0.05

• Actual probability of Type I error ≫ 5%

This is sometimes called the “p-value fallacy” or related to the false positive risk (Colquhoun).

---

5. Why this is unavoidable in frequentist testing

• The frequentist framework does not assign probabilities to hypotheses

• It only controls error rates before seeing the data

• Once the data are observed, the p-value has no direct decision-theoretic meaning

So the p-value is not wrong, but its interpretation is routinely wrong.

---

6. Correct interpretation

✔ Correct:

“If the null hypothesis were true, data at least this extreme would occur with probability p.”

❌ Incorrect:

“There is a p probability that I am making a Type I error.”

---

7. Key takeaway

p-values over-estimate first-order risk because they:

• condition on $H_0$ being true,

• ignore the probability that $H_0$ is false,

• are mistaken for posterior probabilities.

This is why:

• very small p-values are needed for strong evidence,

• replication matters,

• Bayesian or likelihood-based measures are often more informative.