OLS Fundamentals

This page covers the statistical theory behind the Linear Regression tab. See that page for usage instructions.

Model Formulation

The linear regression model is formulated as:

Y = X\beta + \varepsilon

where $Y$ is the $n \times 1$ response vector, $X$ is the $n \times p$ design matrix (predictors and intercept), $\beta$ is the $p \times 1$ coefficient vector, and $\varepsilon$ is the error term.

The OLS estimator minimizes the residual sum of squares $\|Y - X\beta\|^2$ and is obtained from the normal equations:

\hat\beta = (X'X)^{-1}X'Y

The properties of this estimator depend on the assumptions placed on $\varepsilon$ . Consistency is an asymptotic property (convergence as $n \to \infty$ ) and unbiasedness is a finite-sample property ( $E[\hat\beta] = \beta$ ); they are conceptually independent. For example, maximum likelihood estimators are consistent under regularity conditions but generally biased in finite samples.

Consistency: Under $\operatorname{plim}(X'\varepsilon/n) = 0$ (where $\operatorname{plim}$ denotes the probability limit) and $\operatorname{plim}(X'X/n)$ nonsingular, the OLS estimator is consistent. Homoscedasticity and uncorrelated errors are not required. However, when predictors are correlated with the error term, for example through omission of relevant variables or measurement error in the predictors, this condition fails and the OLS estimator loses consistency.

Unbiasedness: Under $E[\varepsilon \mid X] = 0$ , the OLS estimator is unbiased. Homoscedasticity is not required.

BLUE: Under $E[\varepsilon \mid X] = 0$ and $\operatorname{Var}(\varepsilon \mid X) = \sigma^2 I$ (homoscedastic and uncorrelated), the Gauss-Markov theorem guarantees the OLS estimator has minimum variance among linear unbiased estimators (Best Linear Unbiased Estimator). Here, a linear estimator is one that can be written as $CY$ , where $C$ is a matrix that depends only on $X$ .

Normality-based inference: Under $\varepsilon \sim N(0, \sigma^2 I)$ , the sampling distribution of $\hat\beta$ is exact in finite samples, and the $t$ -based confidence interval $\hat\beta \pm t_{\alpha/2,\, n-p} \times \operatorname{SE}(\hat\beta)$ has exact coverage. Without normality, if the errors have finite variance, the central limit theorem ensures asymptotic normality of $\hat\beta$ in large samples, and the coverage of the confidence interval approaches the nominal level. The required sample size depends on the true error distribution, so no universal threshold applies. If the residual Q-Q plot shows strong skewness or heavy tails, the asymptotic approximation becomes less reliable (see residual diagnostic plots).

OLS is a special case of GLM (Gaussian family with identity link).

Standardized Residuals and Diagnostic Statistics

Residual diagnostics in OLS use the internally studentized residual $r_i^*$ :

r_i^* = \frac{e_i}{\hat\sigma\sqrt{1 - h_i}}

where $e_i = y_i - \hat y_i$ is the residual, $\hat\sigma = \sqrt{\text{RSS}/(n - p)}$ is the error standard deviation estimated from all observations, and $h_i = \operatorname{diag}(H)_i$ is the diagonal element of the hat matrix $H = X(X'X)^{-1}X'$ (leverage). $p$ is the number of columns in the design matrix $X$ , including the intercept. Since $H$ is symmetric and idempotent (an orthogonal projection matrix), $0 \le h_i \le 1$ . For models with an intercept, $h_i \ge 1/n$ . Leverage measures how far an observation's predictor values are from the others. Since $\operatorname{tr}(H) = p$ , the average leverage is $p/n$ , and $2p/n$ is the conventional threshold for high leverage.

Cook's Distance combines residual magnitude and leverage into a single influence measure:

D_i = \frac{r_i^{*2}}{p} \cdot \frac{h_i}{1 - h_i}

Multicollinearity and VIF

When predictors are highly correlated, $(X'X)$ approaches singularity and coefficient estimates become unstable.

VIF (Variance Inflation Factor) = $1 / (1 - R_j^2)$ is computed from $R_j^2$ , the R-squared obtained by regressing $X_j$ on all other predictors. A high $R_j^2$ means most of the variation in $X_j$ is already explained by other variables, leaving little unique information. VIF tells you how many times the variance of $\hat\beta_j$ is inflated as a result. For example, VIF = 5 means the standard error of $\hat\beta_j$ is $\sqrt{5} \approx 2.2$ times wider than it would be with uncorrelated predictors. $\hat\beta_j$ itself remains unbiased, but the inflated variance widens the confidence interval.

OLS Fundamentals

Model Formulation

Standardized Residuals and Diagnostic Statistics

Multicollinearity and VIF

See also