TY - JOUR

T1 - The Net Reclassification Index (NRI)

T2 - A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets

AU - Pepe, Margaret S.

AU - Fan, Jing

AU - Feng, Ziding

AU - Gerds, Thomas

AU - Hilden, Jorgen

PY - 2015/10

Y1 - 2015/10

N2 - The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the receiver operating characteristic curve, the net benefit function, and the Brier score, cannot be large due to poorly fitting risk functions.

AB - The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the receiver operating characteristic curve, the net benefit function, and the Brier score, cannot be large due to poorly fitting risk functions.

U2 - 10.1007/s12561-014-9118-0

DO - 10.1007/s12561-014-9118-0

M3 - Journal article

C2 - 26504496

VL - 7

SP - 282

EP - 295

JO - Statistics in Biosciences

JF - Statistics in Biosciences

SN - 1867-1764

IS - 2

ER -