Numerical Accuracy

This page explains how to verify the accuracy of MIDAS statistical computations yourself.

NIST Statistical Reference Datasets

NIST Statistical Reference Datasets (StRD) are benchmark datasets published by the National Institute of Standards and Technology for evaluating the numerical accuracy of statistical software. Each dataset comes with certified values computed to 15 significant digits.

NIST StRD has five categories: Univariate, Linear Regression, Nonlinear Regression, ANOVA, and MCMC. This page covers Univariate and Linear Regression, which correspond to features available in MIDAS. MIDAS does not have features corresponding to Nonlinear Regression, ANOVA, or MCMC.

You are not limited to NIST datasets. Any dataset where you know the expected results from R, Python, or another tool can be used for verification.

Verification

Verify with the UI

Download a CSV file from the table below
Open MIDAS and load the CSV
Open a Linear Regression tab and set the response variable y and explanatory variables x based on the column names in the CSV
Compare the displayed coefficients, R-squared, and other statistics against the NIST certified values listed below

How to read the results

The numbers in the tables are Log Relative Error (LRE). LRE is the negated common logarithm of the relative error between the MIDAS result and the NIST certified value, and corresponds to the number of matching significant digits.

\text{LRE} = -\log_{10} \frac{|\text{computed} - \text{certified}|}{|\text{certified}|}

All browser applications, including MIDAS, compute with IEEE 754 double-precision floating-point numbers. The significand is 52 bits, so the theoretical upper limit for LRE is approximately 15.9.

Univariate Summary Statistics

The mean and standard deviation displayed in the MIDAS Data Table panel were compared against NIST certified values. The standard deviation divides by $n - 1$ (sample standard deviation).

Dataset	n	LRE(Mean)	LRE(SD)
PiDigits	5000	15	14.9
Lottery	218	15.2	15.7
Lew	200	15	15.2
Mavro	50	15	13.1
Michelso	100	15	13.9
NumAcc1	3	15	15
NumAcc2	1001	15	14.2
NumAcc3	1001	15.9	9.5
NumAcc4	1001	15.7	8.3

Dataset details are published at NIST StRD Univariate.

Linear Regression Datasets

Each dataset was run through MIDAS, and the resulting coefficients, standard errors, R-squared, residual SD, and F statistic were compared against the NIST certified values. The numbers in the table are LRE values for each category. Where a category has multiple parameters, the minimum LRE is shown.

Dataset	n	LRE(Coef.)	LRE(SE)	LRE(R²)	LRE(Resid. SD)	LRE(F)
Norris	36	12.3	13.8	15.5	13.9	11.5
Pontius	40	11.9	13	16	13	9.5
NoInt1	11	14.7	15.4	15.7	15.3	13.9
NoInt2	3	15.3	15.8	16	15.5	14.2
Filip	82	7.3	7.5	10.4	8.2	7.9
Longley	16	13	12.3	14.3	12.3	12
Wampler1	21	9.5	exact	15	exact	exact
Wampler2	21	12.6	exact	15	exact	exact
Wampler3	21	9.5	13.6	16	14.4	11
Wampler4	21	7.8	13.5	15.9	14.8	15.7
Wampler5	21	5.8	13.5	13.7	14.8	13.7

Wampler1 and Wampler2 are noise-free data where the model fits the data exactly. All residuals are zero, so SE, residual SD, and F are also zero. "exact" in the table indicates that both the NIST certified value and the MIDAS result are zero.

Model specifications, certified values, and dataset descriptions for all 11 datasets are published at NIST StRD Linear Regression.

Known Limitations

Filip: A 10th degree polynomial with a design matrix condition number around $10^6$ . The larger the condition number, the more rounding errors affect the result. The coefficient accuracy is 7 significant digits. Orthogonal polynomials can reduce the condition number by transforming the polynomial basis, but MIDAS does not currently support this.

Wampler5: The highest-noise 5th degree polynomial dataset. The design matrix condition number is large, and the coefficient accuracy is 6 significant digits.

Data Source

Filliben, J. and Possolo, A. (1999). Statistical Reference Datasets. NIST Standard Reference Database 140. National Institute of Standards and Technology. https://doi.org/10.18434/T43G6C