---
title: Prediction Accuracy Metrics
description: Definitions and interpretation of the prediction accuracy metrics shown in MIDAS. Covers R², RMSE, MAE, Mean Deviance, Brier Score, and AUC, including which metric suits which type of response.
priority: 0.5
audience: GLM の予測タブなどが表示する精度指標の意味を確かめたい読者。回帰分析の基礎は仮定する。予測評価の指標の知識は仮定しない
criteria: MIDAS が表示する予測精度指標の定義・読み方・使い分けを指標単位で扱う。MIDAS が表示しない指標は扱わない。どのタブにどの指標が表示されるかの対応は各タブのページに置き、このページには導線だけを置く
---

# Prediction Accuracy Metrics {#prediction-accuracy-metrics}

Prediction accuracy metrics summarize how good a model's predictions are by comparing predicted values with observed values.

The appropriate metric depends on the type of response. Metrics that measure the size of errors suit continuous responses, while binary responses call for dedicated metrics that evaluate predicted values between 0 and 1.

The values of these metrics also depend on which data are used for evaluation. The model chooses its coefficients to fit the fitting data as closely as possible. Metrics evaluated on the same data therefore come out better than values evaluated on new data. Fitting the chance variation in the data as well is called overfitting, and the stronger the overfitting, the larger this gap. To learn how well the model predicts new data, evaluate on data not used for fitting.

## RMSE and MAE {#rmse-and-mae}

RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) summarize the size of prediction errors. Both values are expressed in the same unit as the response variable.

$$
\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat\mu_i)^2}
\qquad
\text{MAE} = \frac{1}{n}\sum_{i=1}^{n}\lvert y_i - \hat\mu_i \rvert
$$

The two metrics differ in their sensitivity to large errors. RMSE squares the errors before averaging, so large errors count more heavily. MAE averages the absolute errors as they are. On the same data, RMSE is always at least as large as MAE, and a few large errors increase RMSE more. A large gap between the two indicates that the size of the errors varies widely across observations.

## R² and Predictive R² {#r-squared}

R² is the fraction by which the squared error is reduced compared with a baseline model that predicts the mean of the observed values.

$$
R^2 = 1 - \frac{\sum_i (y_i - \hat\mu_i)^2}{\sum_i (y_i - \bar{y})^2}
$$

Values near 1 indicate small errors. A value of 0 indicates predictions no better than the mean, and negative values indicate predictions worse than the mean[^r2-undefined].

[^r2-undefined]: The denominator is the variability of the observed values, so R² cannot be computed when all observed values are identical.

The R² shown in the GLM Prediction tab is a predictive R², using the mean $\bar{y}$ of the observed values in the evaluation dataset itself as the baseline. It is a different quantity from the R² computed on the fitting data. Even when the latter is positive, the predictive R² on new data can be negative.

## Responses with Mean-Dependent Variance and Mean Deviance {#mean-deviance}

For families whose variance depends on the mean (Poisson with $V(\mu) = \mu$, Gamma with $V(\mu) = \mu^2$, and so on), metrics based on squared error are misleading as a measure of fit. For this reason, MIDAS does not show R² for these families. Metrics based on squared error average the errors of all observations with equal weight. In these families, observations with larger means naturally vary more. Observations with large means therefore account for most of the sum of squared errors, and R² mostly reflects the fit to those observations.

Mean Deviance is the [deviance](glossary#deviance) $D$ on the evaluation data divided by the number of observations $n$.

$$
\text{Mean Deviance} = \frac{D}{n}
$$

Because the deviance measures the discrepancy between observations and predictions through the family's likelihood, it summarizes the errors without being skewed toward large-mean observations even for families whose variance depends on the mean. Smaller values indicate predictions closer to the observations[^gaussian-deviance]. The scale of the value depends on the unit of the response and on the family, so there is no standalone threshold for a good value. Use it to compare models on the same family and the same evaluation data.

[^gaussian-deviance]: For Gaussian, the deviance equals the residual sum of squares, so Mean Deviance is the square of RMSE. Mean Deviance can be seen as the mean squared error generalized to other families.

## Brier Score {#brier-score}

A model for a binary response outputs a predicted value between 0 and 1 for each observation. The Brier Score evaluates this predicted value as the probability of a 1. It is computed as the mean squared difference between the observed values (0 or 1) and the predicted values.

$$
\text{Brier} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{p}_i)^2
$$

Values near 0 indicate predicted values close to the observations. An uninformative prediction that assigns 0.5 to every observation scores 0.25, so values around 0.25 indicate probabilities carrying little information.

## The ROC Curve and AUC {#roc-curve-and-auc}

Turning predicted values into 0/1 decisions creates two kinds of errors that trade off against each other depending on the threshold. A decision is made by choosing a threshold. Observations whose predicted value exceeds it are classified as 1, and the rest as 0. The two kinds of errors are classifying a 0 observation as 1, and missing a 1 observation by classifying it as 0. Lowering the threshold classifies more observations as 1, so fewer 1s are missed. In exchange, more 0s are classified as 1.

The ROC curve draws this whole trade-off as a single curve. It plots the fraction of 0 observations classified as 1 on the horizontal axis and the fraction of 1 observations classified as 1 on the vertical axis. The points traced as the threshold moves from the maximum to the minimum form the curve. The curve runs from the bottom-left point, where no observation is classified as 1, to the top-right point, where every observation is. The better the model discriminates, the more 1s it detects before misclassifying many 0s, so the curve approaches the top-left corner.

AUC (Area Under the Curve) is the area under the ROC curve and summarizes the model's ability to discriminate without depending on the choice of threshold. A value of 1.0 means some threshold classifies every observation at or below it as 0 and every observation above it as 1, discriminating perfectly. A value of 0.5 means the assignment is no better than random. Which threshold to choose is a decision outside the analysis, made by weighing the two kinds of errors against each other. AUC measures how good a trade-off the model can achieve before that decision is made.

AUC uses the predicted values only as scores to compare against the threshold and does not evaluate them as probabilities. For example, a model that assigns 0.9 to every observation with outcome 1 and 0.8 to every observation with outcome 0 discriminates perfectly with a threshold of 0.85, so its AUC is 1.0. The Brier Score of the same model, however, is 0.325 on data with half 1s and half 0s, worse than the 0.25 of an uninformative prediction that assigns 0.5 to every observation. To assess the accuracy of the probability values, use the [Brier Score](#brier-score).

For models fitted in the successes-and-trials form (Grouped Binomial), each row is treated as its number of trials of 0/1 observations, and the Brier Score and AUC are weighted by the number of trials. Per-observation 0/1 data corresponds to the case of one trial and is computed with the same formulas.

## See also {#see-also}

- [The GLM Tab: Prediction Accuracy Metrics](glm#prediction-accuracy-metrics) — which metrics appear for each family
- [GLM Fundamentals: Variance Functions and Overdispersion](concepts-glm#variance-functions-and-overdispersion) — the mathematics of families with mean-dependent variance
- [Glossary: Deviance](glossary#deviance) — definition of the deviance