Hypothesis Testing
The statistical framework for deciding whether data support a claim, using null hypotheses, p-values, and error probabilities.
.jpg)
Hypothesis testing is the statistical procedure for deciding whether observed data are consistent with a claim. It was formalized by Jerzy Neyman and Egon Pearson in the 1920s–1930s, building on ideas of Ronald Fisher. The test starts with two competing statements: the null hypothesis H₀, typically the default or boring claim (a new drug has no effect; a coin is fair), and the alternative hypothesis H₁ (the drug works; the coin is biased). The data are then summarized by a test statistic whose distribution under H₀ is known.
The logic is indirect proof by improbability. The researcher assumes H₀ is true and asks: how likely would data at least as extreme as what we observed be, if H₀ were true? That probability is the p-value. If the p-value is very small, the data are unlikely under H₀, so either something rare happened or H₀ is false — and the conventional conclusion is to reject H₀ in favor of H₁. If the p-value is large, the data are compatible with H₀, and the test fails to reject it: absence of evidence, not evidence of absence.
The decision framework makes errors explicit. A Type I error (false positive) rejects H₀ when it is actually true; its probability is the significance level α, conventionally 0.05. A Type II error (false negative) fails to reject H₀ when H₁ is true; its probability is β, and power = 1 − β is the probability of detecting a real effect. Sample size determines power: tiny studies detect only huge effects, which is why underpowered research produces both false negatives and inflated effect estimates. The central limit theorem underpins many test statistics by describing the sampling distribution of means, and variance estimates such as the standard deviation enter the calculations directly.
Misuses of p-values are widespread and well documented. A p-value is not the probability that H₀ is true — it is computed assuming H₀ is true, and it says nothing about the prior plausibility of hypotheses; that question belongs to Bayesian statistics, where evidence updates prior probabilities via Bayes' theorem. p-hacking — running many analyses and reporting only the significant ones — inflates false positives, which is why preregistration, correction for multiple comparisons, and replication matter. Because a statistically significant result can still be trivially small in practical terms, reporting effect sizes and confidence intervals alongside p-values is now the recommended practice.
Tags
hypothesis tests p-values probability statistics