Glossary of Statistical Terms

Definitions of statistical terms assumed as prerequisites in concepts pages. Terms are listed in alphabetical order.

Asymptotic normality

The property that the distribution of an estimator converges in distribution to a normal distribution as the sample size $n \to \infty$. Under appropriate normalization,

$$\sqrt{n}(\hat\theta_n - \theta) \xrightarrow{d} N(0, V)$$

$V$ is the asymptotic variance matrix, which depends on the type of estimator. MLEs possess asymptotic normality under regularity conditions (Casella & Berger, 2002, Ch. 10). For OLS without normality assumptions, the central limit theorem ensures that $\hat\beta$ is asymptotically normal in large samples (OLS Fundamentals).

Consistency

The property that an estimator $\hat\theta_n$ converges in probability to the true parameter $\theta$ as $n \to \infty$, written $\hat\theta_n \xrightarrow{p} \theta$.

Consistency is a basic requirement for estimators: it guarantees that estimates approach the true value as data accumulate. Consistency alone says nothing about estimation precision at finite sample sizes. The OLS estimator is consistent under $\operatorname{plim}(X'\varepsilon/n) = 0$ ($\operatorname{plim}$ denotes the probability limit). Homoscedasticity and uncorrelated errors (required by Gauss-Markov) are not needed for consistency (OLS Fundamentals).

Convergence in distribution

A mode of convergence for a sequence of random variables $X_n$ where the distribution approaches another distribution as $n \to \infty$. Formally, $X_n \xrightarrow{d} X$ if the distribution functions satisfy $F_n(x) \to F(x)$ at every continuity point of $F$.

Convergence in probability means "approaching a specific value"; convergence in distribution means "the shape of the distribution approaches a specific distribution." Convergence in probability to a constant implies convergence in distribution to the degenerate distribution at that constant, but not vice versa. Asymptotic normality is defined using this concept.

Convergence in probability

A mode of convergence for a sequence of random variables $X_n$ toward a value $c$. For every $\varepsilon > 0$,

$$P(|X_n - c| > \varepsilon) \to 0 \quad (n \to \infty)$$

Written $X_n \xrightarrow{p} c$. As $n$ grows, the probability that $X_n$ deviates from $c$ by more than $\varepsilon$ vanishes. Consistency of an estimator is defined using this concept. The notation $\operatorname{plim} X_n = c$ is equivalent to $X_n \xrightarrow{p} c$.

Delta method

A technique for approximating the variance of a nonlinear function $g(\hat\theta)$ of an estimator. By taking a first-order Taylor expansion of $g$ around $\hat\theta$:

$\operatorname{Var}(g(\hat\theta)) \approx g'(\theta)^2 \operatorname{Var}(\hat\theta)$

In the multivariate case, use the gradient vector $\nabla g$ and the variance-covariance matrix $\Sigma$: $\nabla g^\top \Sigma \, \nabla g$.

In dose-response analysis, the delta method is used to construct confidence intervals for nonlinear functions of regression coefficients, such as LD50. For LD50 = $\exp(-\hat\beta_0 / \hat\beta_1)$, the partial derivatives of $g(\hat\beta_0, \hat\beta_1) = -\hat\beta_0 / \hat\beta_1$ and the coefficient variance-covariance matrix yield an approximate variance, from which a confidence interval is computed on the log scale and then exponentiated.

The delta method relies on asymptotic normality and may be inaccurate in small samples. For ratios of parameters, Fieller's method is known to perform better in small samples (Casella & Berger, 2002).

Fieller's method

A method for constructing a confidence interval directly for the ratio $\rho = \beta_0 / \beta_1$ of two parameters. Unlike the delta method, which linearizes $g(\hat\theta)$, Fieller's method starts from the definition of the ratio and derives an exact confidence region.

Solving $\hat\beta_0 - \rho \hat\beta_1 = 0$ for $\rho$ yields a confidence interval as the solution to a quadratic inequality based on the variance of $(\hat\beta_0 - \rho \hat\beta_1)$. Because it does not rely on Taylor approximation, the interval appropriately widens when $\hat\beta_1$ is close to zero (i.e., when the ratio is unstable).

In dose-response analysis, Fieller's method is used for interval estimation of LD50 = $\exp(-\beta_0 / \beta_1)$ (Fieller, 1954).

Deviance

A measure of model fit based on the log-likelihood difference from the saturated model:

$$D = 2(\ell_{\text{saturated}} - \ell_{\text{model}})$$

The saturated model has as many parameters as observations and perfectly reproduces the data. Deviance generalizes the residual sum of squares from OLS to GLMs. Larger deviance indicates poorer model fit. In GLMMs, penalized deviance is used for parameter estimation (GLMM Fundamentals).

Estimator

A function of data used to infer an unknown parameter. Since data are random variables, an estimator is itself a random variable that takes different values across samples. The specific numerical value obtained by applying an estimator to observed data is called an estimate.

For example, the sample mean $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$ is an estimator of the population mean $\mu$. The quality of an estimator is evaluated through properties such as consistency, unbiasedness, and asymptotic normality.

Likelihood and log-likelihood

The likelihood is the same formula as the probability density (or mass) function, read as a function of the parameter $\theta$. For a single observation, $L(\theta) = f(y \mid \theta)$; for $n$ independent observations, $L(\theta) = \prod_{i=1}^n f(y_i \mid \theta)$. While probability varies over possible data for a given parameter, likelihood varies over possible parameters for observed data.

The log-likelihood $\ell(\theta) = \log L(\theta)$ converts products of independent observations into sums, making numerical computation more tractable. Since logarithm is monotone, maximizing likelihood and log-likelihood yield the same result.

GLM parameter estimation (GLM Fundamentals), Laplace approximation in GLMMs (GLMM Fundamentals), and Cox model partial likelihood (Survival Analysis Fundamentals) are all based on log-likelihood.

Maximum likelihood estimator (MLE)

The parameter value that maximizes the likelihood function: $\hat\theta_{\text{ML}} = \arg\max_\theta L(\theta; y)$.

When the model is correctly specified and regularity conditions (technical conditions on the smoothness of the likelihood function and the parameter space) hold, MLEs possess consistency, asymptotic normality, and asymptotic efficiency (the asymptotic variance is minimal among regular consistent estimators) (Casella & Berger, 2002, Ch. 10). In GLMs, the MLE has no closed-form solution and is computed numerically via IRLS (GLM Fundamentals).

Overdispersion

A condition where the observed variance in data exceeds the variance assumed by the model. Poisson and Binomial families assume the dispersion parameter $\phi = 1$, but real data often exhibit greater variability.

Overdispersion leads to underestimated standard errors and overly narrow confidence intervals. When overdispersion is detected in Poisson models, switching to Negative Binomial explicitly models the extra variance. For Binomial overdispersion, see GLM Fundamentals.

Sufficient statistic

A statistic that retains all information in the data about a parameter $\theta$. Formally, $T(X)$ is sufficient for $\theta$ if the conditional distribution of $X$ given $T(X)$ does not depend on $\theta$ (Fisher-Neyman factorization theorem).

Summarizing data through a sufficient statistic loses no information relevant to estimating $\theta$. In GLMs with canonical links, $X'y$ is a sufficient statistic for $\beta$, and the log-likelihood is concave in $\beta$. When the design matrix has full rank, this guarantees uniqueness of the MLE and stable IRLS convergence (GLM Fundamentals).

Unbiasedness

The property that the expected value of an estimator equals the true parameter: $E[\hat\theta] = \theta$.

Unbiasedness and consistency are independent properties: an estimator can be unbiased but inconsistent, or consistent but biased in finite samples. The OLS estimator is unbiased under $E[\varepsilon] = 0$. With the additional assumptions of homoscedasticity and uncorrelated errors ($\operatorname{Var}(\varepsilon) = \sigma^2 I$), the Gauss-Markov theorem guarantees it has minimum variance among linear unbiased estimators (BLUE). MLEs are generally biased in finite samples but possess consistency and asymptotic efficiency (OLS Fundamentals).

Variance function

In exponential family distributions, the function $V(\mu)$ that determines the mean-variance relationship: $\operatorname{Var}(Y) = V(\mu) \cdot a(\phi)$, where $a(\phi)$ is a scaling function of the dispersion parameter $\phi$. $V(\mu)$ is the second derivative of the log-partition function $b(\theta)$, expressed as a function of $\mu$.

For Poisson, $V(\mu) = \mu$; for Binomial, $V(\mu) = \mu(1-\mu)$; for Gamma, $V(\mu) = \mu^2$ (GLM Fundamentals, McCullagh & Nelder, 1989).

References

• Casella, G., & Berger, R. L. (2002). Statistical Inference (2nd ed.). Duxbury.
• Fieller, E. C. (1954). Some problems in interval estimation. Journal of the Royal Statistical Society: Series B, 16(2), 175-185.
• McCullagh, P., & Nelder, J. A. (1989). Generalized Linear Models (2nd ed.). Chapman and Hall/CRC.