GLM Fundamentals

This page covers the statistical theory behind the GLM tab. See that page for usage instructions.

Model Formulation

GLM generalizes the normal linear model to the exponential family of distributions, as introduced by Nelder & Wedderburn (1972). A GLM is defined by three components:

Distribution family: The response variable $Y$ follows a distribution in the exponential family
Linear predictor: $\eta = X\beta$ (a linear combination of explanatory variables)
Link function: A monotonic function $g$ such that $\eta = g(\mu)$ , connecting the linear predictor to the mean $\mu = E[Y]$

OLS is a special case of GLM (Gaussian family with identity link). In this case, IRLS converges in a single iteration to the normal equations solution, and the Wald test is equivalent to the OLS $t$ -test.

Exponential Family

A family of distributions is called an exponential family if its density (or mass) function can be written as:

f(y \mid \theta, \phi) = \exp\!\left\{\frac{y\theta - b(\theta)}{a(\phi)} + c(y, \phi)\right\}

where $\theta$ is the natural (canonical) parameter, $\phi$ is the dispersion parameter, and $b(\theta)$ is the log-partition function. The mean and variance are derived from $b(\theta)$ :

$E[Y] = b'(\theta) = \mu$
$\operatorname{Var}(Y) = b''(\theta) \cdot a(\phi)$

Rewriting $b''(\theta)$ as a function of $\mu$ rather than $\theta$ gives the variance function $V(\mu)$ , so $\operatorname{Var}(Y) = V(\mu) \cdot a(\phi)$ . For example, Poisson has $b(\theta) = e^\theta$ , giving $b'(\theta) = e^\theta = \mu$ and $b''(\theta) = e^\theta = \mu$ , hence $V(\mu) = \mu$ .

Exponential family parameters for each distribution family:

Family	$\theta$ (natural parameter)	$a(\phi)$	$b(\theta)$	$c(y, \phi)$
Gaussian	$\mu$	$\phi$	$\theta^2/2$	$-\bigl(y^2/(2\phi) + \log(2\pi\phi)/2\bigr)$
Binomial	$\log\!\bigl(\mu/(1-\mu)\bigr)$	$1/n$	$\log(1+e^\theta)$	$\log\binom{n}{k}$
Poisson	$\log\mu$	$1$	$e^\theta$	$-\log(y!)$
Gamma	$-1/\mu$	$\phi$	$-\log(-\theta)$	$(1/\phi - 1)\log y + (1/\phi)\log(1/\phi) - \log\Gamma(1/\phi)$
Negative Binomial	$\log\!\bigl(\mu/(\mu+r)\bigr)$	$1$	$-r\log(1-e^\theta)$	$\log\Gamma(y+r) - \log\Gamma(r) - \log(y!)$

In the Binomial row, $y$ is the proportion of successes ( $y = k/n$ , $0 \le y \le 1$ ), $k$ is the number of successes, $n$ is the number of trials, and $\mu$ is the success probability. When $n=1$ , it reduces to the Bernoulli distribution
The $r$ in Negative Binomial corresponds to the shape parameter $\theta$ in the MIDAS UI. The Negative Binomial belongs to the exponential family only when $r$ is known. In MIDAS's automatic estimation mode, $r$ is estimated in an outer loop
The $\theta$ in this table is the exponential family natural parameter, which is distinct from the Negative Binomial shape parameter $\theta$ in the MIDAS UI

Link Functions

The link function is a monotonic function $\eta = g(\mu)$ connecting the linear predictor $\eta$ to the expected value $\mu$ of the response. A link function satisfying $g(\mu) = \theta$ (the natural parameter) is called the canonical link.

Link Function	Formula	Canonical Link For
Identity	$\eta = \mu$	Gaussian
Logit	$\eta = \log\!\bigl(\mu / (1 - \mu)\bigr)$	Binomial
Log	$\eta = \log(\mu)$	Poisson, Negative Binomial
Inverse	$\eta = 1/\mu$	Gamma
Probit	$\eta = \Phi^{-1}(\mu)$	—

The canonical link has important properties: since $\eta = \theta$ , $X'y$ becomes a sufficient statistic for $\beta$ , and the log-likelihood is concave in $\beta$ . When the design matrix $X$ has full rank, this guarantees uniqueness of the MLE and stable IRLS convergence.

Non-canonical links forfeit these properties but may be chosen for easier coefficient interpretation. For example, the canonical link for Gamma is Inverse ( $\eta = 1/\mu$ ), which puts coefficients on a $1/\mu$ scale that is hard to interpret. The Log link ( $\exp(\beta)$ as a multiplicative effect) is more commonly used in practice.

Parameter Estimation (IRLS)

GLM parameters $\beta$ are estimated by maximum likelihood. Since no closed-form solution exists, IRLS (Iteratively Reweighted Least Squares) is used.

At each iteration, working weights $W$ and an adjusted dependent variable $z$ are computed, then the weighted least squares problem:

\hat\beta^{(t+1)} = (X'W^{(t)}X)^{-1}X'W^{(t)}z^{(t)}

is solved to update $\beta$ . See McCullagh & Nelder (1989, Ch. 2) for the derivation of $W$ and $z$ . Iteration stops when the maximum absolute change in coefficients falls below the convergence threshold.

With the canonical link, the concavity of the log-likelihood ensures stable convergence. Non-canonical links may lead to slower convergence or convergence failure.

Variance Functions and Overdispersion

As described in the Exponential Family section, the variance function $V(\mu) = b''(\theta)$ is the second derivative of the log-partition function rewritten in terms of $\mu$ . Through the relationship $\operatorname{Var}(Y) = V(\mu) \cdot a(\phi)$ , it defines the mean-variance relationship for each family.

Family	$b''(\theta)$	$V(\mu)$	$a(\phi)$	$\operatorname{Var}(Y)$
Gaussian	$1$	$1$	$\phi$	$\phi$ (= $\sigma^2$ )
Binomial	$\dfrac{e^\theta}{(1+e^\theta)^2}$	$\mu(1 - \mu)$	$1/n$	$\mu(1-\mu)/n$
Poisson	$e^\theta$	$\mu$	$1$	$\mu$
Gamma	$1/\theta^2$	$\mu^2$	$\phi$	$\mu^2 \phi$
Negative Binomial	$\dfrac{re^\theta}{(1-e^\theta)^2}$	$\mu + \mu^2/r$	$1$	$\mu + \mu^2/r$

Poisson and Binomial assume a dispersion parameter $\phi = 1$ . When the actual data variance exceeds this assumption, the condition is called overdispersion. Overdispersion leads to underestimated standard errors and confidence intervals that are too narrow.

When overdispersion is detected with Poisson data, switching to Negative Binomial adds a $\mu^2/r$ term to the variance, explicitly modeling the extra dispersion.

For Binomial overdispersion, MIDAS does not currently support quasi-binomial or Beta-Binomial alternatives. When overdispersion is suspected, check the estimated dispersion parameter and consider that standard errors and confidence intervals may be underestimated.

References

Nelder, J. A., & Wedderburn, R. W. M. (1972). Generalized linear models. Journal of the Royal Statistical Society: Series A, 135(3), 370-384.
McCullagh, P., & Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Chapman and Hall/CRC.