Hypothesis Testing: Critical Values & p-Values
Interactive workbench for z-tests, Student's t-tests, rejection regions, and error rates
The Anatomy of Hypothesis Testing
In classical Null Hypothesis Significance Testing (NHST), we assume the default null hypothesis $H_0: \mu = \mu_0$ is true until evidence contradicts it.
$$\text{Test Statistic:} \quad z = \frac{\bar{X} - \mu_0}{\sigma / \sqrt{n}} \quad \text{or} \quad t = \frac{\bar{X} - \mu_0}{s / \sqrt{n}} \quad (df = n - 1)$$
Critical Values vs p-Values
Critical Value ($z_{\text{crit}}$ or $t_{\text{crit}}$): The threshold set in advance by your chosen significance level $\alpha$. Values beyond this define the Rejection Region.
p-Value: The probability of observing a test statistic as extreme as (or more extreme than) the sample result, assuming $H_0$ is true.
$$\text{Decision Rule:} \quad p \le \alpha \iff |\text{test statistic}| \ge \text{critical value} \implies \text{Reject } H_0$$
Significance Workbench
Error Types & Statistical Decision Matrix
| Reality \ Your Decision | Fail to Reject $H_0$ (Negative) | Reject $H_0$ (Positive) |
|---|---|---|
| Null $H_0$ is True (No actual effect) | Correct Decision (Probability = $1 - \alpha$) | Type I Error (False Alarm) (Probability = $\alpha$) |
| Alternative $H_1$ is True (Real effect exists) | Type II Error (Missed Detection) (Probability = $\beta$) | Correct Decision (Statistical Power) (Probability = $1 - \beta$) |
For small $n$, the Student's $t$ curve has heavier tails than the normal curve. At $df = 4$, the critical value for $\alpha = 0.05$ (two-tailed) is $\pm 2.776$ instead of $\pm 1.960$. If you mistakenly used the normal $z$-test on a sample of size $n=5$, a test statistic of $2.15$ would falsely claim statistical significance ($2.15 > 1.96$), whereas the correct $t$-test shows it is not significant ($2.15 < 2.776$)!