OLS Fundamentals

This page covers the statistical theory behind the Linear Regression tab. See that page for usage instructions.

Model Formulation

The linear regression model is formulated as:

Y = X\beta + \varepsilon

where $Y$ is the $n \times 1$ response vector, $X$ is the $n \times p$ design matrix (predictors and intercept), $\beta$ is the $p \times 1$ coefficient vector, and $\varepsilon$ is the error term.

The OLS estimator minimizes the residual sum of squares $\|Y - X\beta\|^2$ and is obtained from the normal equations:

\hat\beta = (X'X)^{-1}X'Y

The properties of this estimator depend on the assumptions placed on $\varepsilon$ .

Assuming $\operatorname{plim}(X'\varepsilon/n) = 0$ : The OLS estimator is consistent. Homoscedasticity and uncorrelated errors are not required.

Further assuming $E[\varepsilon] = 0$ and $\operatorname{Var}(\varepsilon) = \sigma^2 I$ (homoscedastic and uncorrelated): By the Gauss-Markov theorem, the OLS estimator is the Best Linear Unbiased Estimator (BLUE).

Further assuming $\varepsilon \sim N(0, \sigma^2 I)$ (normality): Exact finite-sample distributions for $t$ -tests and $F$ -tests are obtained.

Without normality, the central limit theorem ensures the test statistics are asymptotically normal in large samples. The required sample size depends on the true error distribution, so no universal threshold applies.

OLS is a special case of GLM (Gaussian family with identity link).

Standardized Residuals and Diagnostic Statistics

Residual diagnostics in OLS use the internally studentized residual $r_i^*$ :

r_i^* = \frac{e_i}{\hat\sigma\sqrt{1 - h_i}}

where $e_i = y_i - \hat y_i$ is the residual, $\hat\sigma = \sqrt{\text{RSS}/(n - p)}$ is the error standard deviation estimated from all observations, and $h_i = \operatorname{diag}(H)_i$ is the diagonal element of the hat matrix $H = X(X'X)^{-1}X'$ (leverage). $p$ is the number of columns in the design matrix $X$ , including the intercept. Leverage measures how far an observation's predictor values are from the others. Since $\operatorname{tr}(H) = p$ , the average leverage is $p/n$ , and $2p/n$ is the conventional threshold for high leverage.

Cook's Distance combines residual magnitude and leverage into a single influence measure:

D_i = \frac{r_i^{*2}}{p} \cdot \frac{h_i}{1 - h_i}

Multicollinearity and VIF

When predictors are highly correlated, $(X'X)$ approaches singularity and coefficient estimates become unstable.

VIF (Variance Inflation Factor) = $1 / (1 - R_j^2)$ is computed from $R_j^2$ , the R-squared obtained by regressing $X_j$ on all other predictors. A high $R_j^2$ means most of the variation in $X_j$ is already explained by other variables, leaving little unique information. VIF tells you how many times the variance of $\hat\beta_j$ is inflated as a result. For example, VIF = 5 means the standard error of $\hat\beta_j$ is $\sqrt{5} \approx 2.2$ times wider than it would be with uncorrelated predictors. $\hat\beta_j$ itself remains unbiased, but its estimate becomes less reliable and the confidence interval widens.

OLS Fundamentals

Model Formulation

Standardized Residuals and Diagnostic Statistics

Multicollinearity and VIF

See also