Statistical Inference & Estimation
Confidence Intervals & Coverage Probability
Demystifying frequentist repeated sampling: watch how 95% confidence intervals trap the true mean
Frequentist Definition of a Confidence Interval
A $(1 - \alpha)$ confidence interval for a population mean $\mu$ is an interval $[\bar{X} - E, \bar{X} + E]$ computed from random sample data:
$$\text{z-Interval (known } \sigma\text{):} \quad \bar{X} \pm z_{1-\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}$$ $$\text{t-Interval (unknown } \sigma\text{):} \quad \bar{X} \pm t_{1-\alpha/2,\, n-1} \cdot \frac{s}{\sqrt{n}}$$
The Critical Frequentist Insight
The population parameter $\mu$ is fixed and unmoving. It is the interval that is random! "95% confidence" means that over hundreds of repeated independent experiments, approximately 95% of the calculated intervals will capture $\mu$, while 5% will miss.
Repeated Sampling Simulation
Repeated Sample Intervals (Green = Hit μ, Red = Missed μ)
100 Intervals
Selected Sample Inspection
Latest Sample
Batch Coverage Rate
95.0%
Batch Hit / Miss Tally
95 / 5
Cumulative Long-Run Coverage
95.0%
Critical Value (z* or t*)
1.960
Mean Margin of Error (E)
±3.58
True Mean (μ)
50.00
Common Fallacies & Practical Rules
Fallacy: "There is a 95% probability that $\mu$ lies between $46.42$ and $53.58$."
Reality: Once the interval numbers are calculated from data, $\mu$ is either inside it (probability = 1) or outside it (probability = 0). The 95% probability applies to the process of generating intervals over repeated samples before data is gathered.
Reality: Once the interval numbers are calculated from data, $\mu$ is either inside it (probability = 1) or outside it (probability = 0). The 95% probability applies to the process of generating intervals over repeated samples before data is gathered.
Why Sample Size ($n$) Shrinks Interval Width:
Margin of error shrinks inversely with the square root of $n$: $E \propto 1/\sqrt{n}$. To cut your interval width in half, you must collect 4× the sample size.
Margin of error shrinks inversely with the square root of $n$: $E \propto 1/\sqrt{n}$. To cut your interval width in half, you must collect 4× the sample size.
When to use $t$ instead of $z$:
In almost all real-world data science problems, the true population variance $\sigma$ is unknown and must be estimated via sample standard deviation $s$. Using the normal $z^*$ instead of Student's $t^*$ on small samples ($n < 30$) understates error and produces intervals that are too narrow, causing empirical coverage to fall below the nominal 95% target!
In almost all real-world data science problems, the true population variance $\sigma$ is unknown and must be estimated via sample standard deviation $s$. Using the normal $z^*$ instead of Student's $t^*$ on small samples ($n < 30$) understates error and produces intervals that are too narrow, causing empirical coverage to fall below the nominal 95% target!