Why p-values over-estimate first order risk ?
The short answer is: 👉 Because a p-value is computed conditional on the null hypothesis being true , it does not represent the probability of making a Type I error in the situation you are actually in. When it is interpreted as such, it systematically overstates (over-estimates) the “first-order risk”. Below is the precise reasoning. 1. What “first-order risk” really is The Type I error rate (first-order risk) is: α = P ( reject H 0 ∣ H 0 is true ) This is a long-run, pre-specified property of a decision rule (e.g. “reject if p < 0.05 ”). It is not a probability about the current experiment . 2. What a p-value actually is A p-value is: p = P ( T ≥ t obs ∣ H 0 ) Key points: It is conditional on H 0 being true It is not P ( H 0 ∣ data ) It is not P ( Type I error ) 3. Where the over-estimation comes from The common (incorrect) interpretation “If...