Probability Distribution Fundamentals

A probability distribution describes how likely each value is to appear. This page explains, for 7 probability distributions, the meaning of their parameters, the range of values, the shape of the distribution, and the mean and variance.

The distributions covered are the 7 available in the Synthetic Data Generator tab, and the parameter names correspond to the input field names of that tab. For choosing a GLM family as the distribution of a response variable, see GLM Fundamentals.

Normal Distribution

The normal distribution is a bell-shaped distribution symmetric around Mean ($\mu$). SD (standard deviation, $\sigma > 0$) sets the spread, and about 95% of the values fall within $\mu \pm 2\sigma$. Values range over all real numbers.

The density function is:

$$f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

The mean is $\mu$ and the variance is $\sigma^2$.

Use it for quantities determined by the sum of many small factors, such as measurement errors and residuals.

Uniform Distribution

The uniform distribution gives every value in the interval from Min (lower bound $a$) to Max (upper bound $b$, $a < b$) the same density. Values are real numbers from $a$ to $b$, and no value appears outside the interval.

The density function is:

$$f(x) = \frac{1}{b - a} \quad (a \le x \le b)$$

The mean is $(a + b)/2$ and the variance is $(b - a)^2/12$.

Use it for quantities whose range is known but where no tendency within the range is assumed.

Bernoulli Distribution

The Bernoulli distribution produces 1 with probability P (success probability $p$, $0 \le p \le 1$) and 0 with probability $1 - p$. The only values are 0 and 1.

The probability mass function is:

$$P(X = 1) = p, \quad P(X = 0) = 1 - p$$

The mean is $p$ and the variance is $p(1-p)$.

Use it for binary outcomes such as pass and fail, or occurrence and non-occurrence.

Categorical Distribution

The categorical distribution selects one of the levels listed in Levels. Weights (weights $w_i \ge 0$) determine how likely each level is to be selected. The values are level names.

The probability mass function is as follows, where $c_i$ is the $i$-th level.

$$P(X = c_i) = \frac{w_i}{\sum_j w_j}$$

MIDAS treats the weights as ratios, so they do not have to sum to 1. Without weights, MIDAS selects every level with equal probability.

Because the values are not numbers, there is no mean or variance.

Use it for nominal-scale variables such as groups, categories, and treatment conditions. See Data Types and Measurement Scales for nominal scales.

Poisson Distribution

The Poisson distribution describes the number of events that occur under a constant average rate. Lambda (average number of occurrences, $\lambda > 0$) determines how often events occur. Values are integers of 0 or more.

The probability mass function is:

$$P(X = x) = \frac{\lambda^x e^{-\lambda}}{x!} \quad (x = 0, 1, 2, \ldots)$$

Both the mean and the variance are $\lambda$. Equality of the mean and the variance is a strong assumption of the Poisson distribution, and the state where the variance of the data exceeds this assumption is called overdispersion. See GLM Fundamentals: Variance Functions and Overdispersion for details.

Use it for count data such as the number of visits or the number of failures.

Gamma Distribution

The gamma distribution is a distribution with a long right tail. Shape ($k > 0$) determines the form of the distribution. With Shape below 1 the density concentrates near 0, with Shape equal to 1 it becomes the exponential distribution, and as Shape grows the form approaches symmetry. Scale ($\theta > 0$) stretches the distribution horizontally without changing its form. Values are positive real numbers.

The density function is as follows, where $\Gamma$ is the gamma function.

$$f(x) = \frac{x^{k-1} e^{-x/\theta}}{\Gamma(k)\,\theta^k} \quad (x > 0)$$

The mean is $k\theta$ and the variance is $k\theta^2$.

Use it for positive, right-skewed quantities such as waiting times and monetary amounts.

Weibull Distribution

The Weibull distribution has two parameters, Shape ($k > 0$) and Scale ($\theta > 0$). Scale stretches the distribution horizontally without changing its form. Values are positive real numbers.

Shape determines how the hazard changes over time. With Shape below 1, the hazard decreases over time. With Shape equal to 1, the hazard is constant and the distribution coincides with the exponential distribution. With Shape above 1, the hazard increases. See Survival Analysis Fundamentals for the definition of the hazard.

The density function is:

$$f(x) = \frac{k}{\theta}\left(\frac{x}{\theta}\right)^{k-1} \exp\left(-\left(\frac{x}{\theta}\right)^k\right) \quad (x > 0)$$

The mean is $\theta\,\Gamma(1 + 1/k)$ and the variance is $\theta^2\left[\Gamma(1 + 2/k) - \Gamma(1 + 1/k)^2\right]$.

Use it for lifetimes and times to failure.

Also available as a Markdown file.