Parametric Estimation · Linear Models

OLS Regression & Outlier Robustness

Drag a single outlier point and watch how Ordinary Least Squares minimization tilts the entire line of best fit

Closed-Form Ordinary Least Squares (OLS)

In simple linear regression, we model $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$. The OLS objective minimizes the sum of squared vertical residuals:

$$\min_{\beta_0, \beta_1} \sum_{i=1}^{n} \left(y_i - (\beta_0 + \beta_1 x_i)\right)^2$$

Setting the partial derivatives with respect to $\beta_1$ and $\beta_0$ to zero yields the exact closed-form solution:

$$\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{\text{Cov}(X, Y)}{\text{Var}(X)}, \qquad \beta_0 = \bar{y} - \beta_1 \bar{x}$$

The Zero Breakdown Point: Because OLS minimizes squared deviations $(y - \hat{y})^2$, errors grow quadratically. A single extreme outlier with high leverage can exert infinite torque on the slope, rotating the entire model arbitrarily!

Interactive Regression Workbench

High Leverage & Influential

Outlier Position & Data

Outlier X Coordinate 82.0

Outlier Y Coordinate 35.0

Base Data Noise (σ) 5.5

Show Residual Lines (e = y - ŷ) Show Uncorrupted Baseline Fit

Statistics & Impact Live OLS

Fitted Equation (with Outlier) y = 0.95x + 42.1

Mean Squared Error (MSE) 648.20

R² (Variance Explained) 0.582

Clean Baseline (15 pts, No Outlier)

Baseline Equation y = 1.82x + 14.8

Baseline MSE 28.40

Baseline R² 0.965

Slope Distortion (Δβ₁) -0.870 (-47.8%)

MSE Explosion 22.8× higher

Outlier Location (82.0, 35.0)

Interactive: Drag the red diamond or click anywhere to move the outlier

High Leverage vs. Influential Observations
A data point has high leverage if its predictor value $x_i$ is far from the mean $\bar{x}$. If this point also deviates from the regression line (large residual), it becomes an influential point that pulls the line toward itself like a physical lever!

Why Squaring Magnifies Errors: L₂ vs L₁
Because the objective squares residuals $e_i^2$, an error of $50$ contributes $2,500$ to the loss, whereas an error of $5$ contributes only $25$. Minimizing total error forces the line to sacrifice accuracy on the 15 clean points just to reduce the outlier's quadratic penalty.

Robust Regression Alternatives

1. Huber Loss (Smooth L₁/L₂)

Huber loss blends the best of both worlds: it behaves quadratically ($\frac{1}{2} e^2$) for small residuals $|e| \le \delta$, but switches to linear ($\delta(|e| - \frac{1}{2}\delta)$) for large errors, preventing outliers from dominating.

2. Theil-Sen Estimator

Computes the median of all pairwise slopes $\frac{y_j - y_i}{x_j - x_i}$. It has a high breakdown point of 29.3%, meaning it can tolerate nearly a third of all data points being corrupted without skewing.

3. RANSAC (Random Sample Consensus)

Iteratively fits models to minimal random subsets of data (2 points in simple regression), counts the inliers within an error threshold, and returns the model supported by the largest consensus set.