A/B Testing & Two-Sample Inference
Real-world experimentation in production: conversion lift, Welch's t-tests, and the sequential peeking hazard
How Two-Sample Experimentation Works
In an A/B test, users are randomly assigned to either the Control (A) baseline or the Treatment (B) variant. Depending on the metric, statistical inference tests whether the observed difference is real or merely sampling noise:
$$\text{Proportions (z-Test):} \quad z = \frac{\hat{p}_B - \hat{p}_A}{\sqrt{\hat{p}_{\text{pool}}(1-\hat{p}_{\text{pool}})\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}} \quad \sim N(0, 1)$$ $$\text{Continuous Means (Welch's t-Test):} \quad t = \frac{\bar{X}_B - \bar{X}_A}{\sqrt{\frac{s_A^2}{n_A} + \frac{s_B^2}{n_B}}} \quad \sim t(df)$$
Difference Confidence Interval
$$\text{Lift CI:} \quad (\hat{\theta}_B - \hat{\theta}_A) \pm z^* \cdot SE_{\text{diff}}$$ If the confidence interval for the difference does not contain zero, the observed lift is statistically significant at significance level $\alpha$.
Experimentation Workbench
Practical Rules & Experimentation Pitfalls
Observing p-values continuously and declaring a winner as soon as $p < 0.05$ creates an optical illusion of significance. Random stochastic walk guarantees that even when there is zero real difference, the p-value will dip below 0.05 at some point during sample collection over 30% of the time!
Never stop an experiment early or run it without an a priori power calculation. Determine your required sample size $N$ upfront based on baseline rate, desired statistical power ($80\%$), and the smallest business lift worth detecting.