OLS Fundamentals

This page covers the statistical theory behind the Linear Regression tab. See that page for usage instructions.

Model Formulation

The linear regression model is formulated as:

Y = X\beta + \varepsilon

where $Y$ is the $n \times 1$ response vector, $X$ is the $n \times p$ design matrix whose columns are the intercept and the predictors, $\beta$ is the $p \times 1$ coefficient vector, and $\varepsilon$ is the error term.

The OLS estimator minimizes the residual sum of squares $\|Y - X\beta\|^2$ and is obtained from the normal equations:

\hat\beta = (X'X)^{-1}X'Y

where $X'$ denotes the transpose of $X$ . When $X$ has full column rank, $(X'X)$ is nonsingular and this solution is unique. See Linear Regression for how MIDAS behaves when the predictors are perfectly linearly dependent and $X$ does not have full column rank.

OLS is a special case of GLM with the Gaussian family and identity link.

Error Assumptions and the Reliability of Results

How far you can trust $\hat\beta$ and its confidence intervals depends on the assumptions placed on the error term $\varepsilon$ . The coefficient estimates and the confidence intervals rest on different assumptions, so this section treats them separately.

Coefficient Estimates

The central assumption behind trusting the coefficient estimates is exogeneity: the expected value of the error, conditional on the predictors, is zero, that is, $E[\varepsilon \mid X] = 0$ . Under this assumption $\hat\beta$ is unbiased, and $E[\hat\beta] = \beta$ holds at any sample size.

Why exogeneity is the key becomes visible when you substitute $Y = X\beta + \varepsilon$ into the normal-equation solution:

\hat\beta = (X'X)^{-1}X'(X\beta + \varepsilon) = \beta + (X'X)^{-1}X'\varepsilon

The estimator splits into the true value $\beta$ and a second term driven by the errors. Under exogeneity, $E[(X'X)^{-1}X'\varepsilon \mid X] = (X'X)^{-1}X'\,E[\varepsilon \mid X] = 0$ , so the second term vanishes in expectation. Everything else in this subsection is about when this second term vanishes and when it persists.

Consistency, the property that $\hat\beta$ converges in probability to $\beta$ as $n \to \infty$ , holds under weaker conditions: $\operatorname{plim}(X'\varepsilon/n) = 0$ (where $\operatorname{plim}$ denotes the probability limit) and $\operatorname{plim}(X'X/n)$ nonsingular are sufficient. The first condition requires the predictors and the error to be asymptotically uncorrelated. Exogeneity implies that the error is uncorrelated with any function of $X$ , so under the usual sampling assumptions it is the stronger requirement.

Conversely, correlation between the predictors and the error is called endogeneity. Under endogeneity the second term of the decomposition does not vanish, so unbiasedness and consistency are lost together, and the bias persists as the probability limit of that term no matter how large the sample grows. Endogeneity typically arises through three routes.

Omitted variables: When a variable that affects the response and is correlated with the included predictors is missing from the model, its effect is absorbed into the error term, which then correlates with the predictors. The classic example is regressing the number of drowning accidents on ice cream sales: unless temperature is included, the ice-cream coefficient absorbs the effect of temperature. The direction and size of the bias depend on how the omitted variable affects the response and how it correlates with the included predictors.
Measurement error in the predictors: When a predictor is observed as the true quantity plus measurement noise, the noise flows into the error term and correlates with the observed predictor. In the classical setting where the noise is unrelated to the true quantity, a single predictor's coefficient is biased toward zero (attenuation bias). With multiple predictors, the direction of the bias is generally not simple.
Reverse causality and simultaneity: The response feeds back into the predictors. For pairs determined jointly, like price and quantity, the predictor values are partly determined by the error, creating correlation between the predictors and the error.

Endogeneity does not show up in residual plots. The normal equations choose $\hat\beta$ so that the residual vector $e = Y - X\hat\beta$ is orthogonal to every predictor ( $X'e = 0$ ), so even when the errors are correlated with the predictors in the population, the in-sample residuals mechanically erase any trace of that correlation. Whether endogeneity is present therefore cannot be judged from diagnostic statistics; it has to be judged from how the data were generated. Three things to check: whether any variable that affects the response is left unmeasured, whether the predictors measure what you intend, and whether the response feeds back into the predictors.

Standard Errors and Confidence Intervals

Standard errors are computed from $\operatorname{Var}(\hat\beta \mid X) = \sigma^2 (X'X)^{-1}$ , a formula that assumes $\operatorname{Var}(\varepsilon \mid X) = \sigma^2 I$ (homoscedastic and uncorrelated errors). When this assumption fails, $\hat\beta$ remains unbiased, but the standard error estimates are biased and the confidence interval widths no longer reflect the actual uncertainty. Homoscedasticity can be checked with the Scale-Location plot.

Under homoscedastic and uncorrelated errors, the Gauss-Markov theorem guarantees that $\hat\beta$ has minimum variance among linear unbiased estimators. An estimator with this property is called the BLUE (Best Linear Unbiased Estimator).

The coverage of the confidence intervals also depends on the error distribution. If the errors follow a normal distribution $\varepsilon \sim N(0, \sigma^2 I)$ , the $t$ -based confidence interval $\hat\beta \pm t_{\alpha/2,\, n-p} \times \operatorname{SE}(\hat\beta)$ has exact coverage in finite samples. Even without normality, if the errors are homoscedastic with finite variance, the sampling distribution of $\hat\beta$ approaches a normal distribution in large samples by the central limit theorem, and the coverage approaches the nominal level. The sample size needed depends on the true error distribution, so there is no universal rule. If the residual Q-Q plot shows strong skewness or heavy tails, this asymptotic approximation becomes less reliable.

Standardized Residuals and Diagnostic Statistics

Residual diagnostics in OLS use the internally studentized residual $r_i^*$ :

r_i^* = \frac{e_i}{\hat\sigma\sqrt{1 - h_i}}

where $e_i = y_i - \hat y_i$ is the residual, $\hat\sigma = \sqrt{\text{RSS}/(n - p)}$ is the error standard deviation estimated from all observations, and $h_i = \operatorname{diag}(H)_i$ is the diagonal element of the hat matrix $H = X(X'X)^{-1}X'$ (leverage). $p$ is the number of columns in the design matrix $X$ , including the intercept. Since $H$ is symmetric and idempotent (an orthogonal projection matrix), $0 \le h_i \le 1$ . For models with an intercept, $h_i \ge 1/n$ . Leverage measures how far an observation's predictor values are from the others. Since $\operatorname{tr}(H) = p$ , the average leverage is $p/n$ , and $2p/n$ is the conventional threshold for high leverage.

Cook's Distance combines residual magnitude and leverage into a single influence measure (Cook, 1977):

D_i = \frac{r_i^{*2}}{p} \cdot \frac{h_i}{1 - h_i}

See Linear Regression for thresholds and visualization.

Multicollinearity and VIF

When predictors are highly correlated, $(X'X)$ approaches singularity and coefficient estimates become unstable.

VIF (Variance Inflation Factor) = $1 / (1 - R_j^2)$ is computed from $R_j^2$ , the R-squared obtained by regressing $X_j$ on all other predictors. A high $R_j^2$ means most of the variation in $X_j$ is already explained by other variables, leaving little unique information. VIF tells you how many times the variance of $\hat\beta_j$ is inflated as a result. For example, VIF = 5 means the standard error of $\hat\beta_j$ is $\sqrt{5} \approx 2.2$ times as large as it would be with uncorrelated predictors. $\hat\beta_j$ itself remains unbiased, but the inflated variance widens the confidence interval. Many references treat VIF > 10 as a conventional sign of serious multicollinearity (Marquardt, 1970), but this is not a universal cutoff — it depends on the precision your analysis requires. See Linear Regression for remedies (removing or combining redundant predictors).

References

Cook, R. D. (1977). Detection of influential observation in linear regression. Technometrics, 19(1), 15-18. https://www.jstor.org/stable/1268249
Marquardt, D. W. (1970). Generalized inverses, ridge regression, biased linear estimation, and nonlinear estimation. Technometrics, 12(3), 591-612. https://www.jstor.org/stable/1267205