Calibration: the Achilles heel of predictive analytics

Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative

doi:10.1186/s12916-019-1466-7

Calibration: the Achilles heel of predictive analytics

Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative

Research output: Contribution to journal › Article › peer-review

242 Citations (Scopus)

3 Downloads (Pure)

Abstract

BACKGROUND: The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.

MAIN TEXT: Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.

CONCLUSION: Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

Original language	English
Article number	230
Journal	BMC medicine
Volume	17
Early online date	16 Dec 2019
DOIs	https://doi.org/10.1186/s12916-019-1466-7
Publication status	Published - 16 Dec 2019

Keywords

Calibration
Risk prediction models
Predictive analytics
Overfitting
Heterogeneity
Model performance

Access to Document

10.1186/s12916-019-1466-7Licence: CC BY

VanCalster_CalibrationAchillesHeel_VOR
© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Final published version, 620 KBLicence: CC BY

Cite this

@article{4fe5353b57e84b55b7c24fe7e5059c6d,

title = "Calibration: the Achilles heel of predictive analytics",

abstract = "BACKGROUND: The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.MAIN TEXT: Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.CONCLUSION: Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.",

keywords = "Calibration, Risk prediction models, Predictive analytics, Overfitting, Heterogeneity, Model performance",

author = "{Van Calster}, Ben and McLernon, {David J} and {van Smeden}, Maarten and Laure Wynants and Steyerberg, {Ewout W} and {Topic Group {\textquoteleft}Evaluating diagnostic tests and prediction models{\textquoteright} of the STRATOS initiative}",

note = "Acknowledgements This work was developed as part of the international STRengthening Analytical Thinking for Observational Studies (STRATOS) initiative. The objective of STRATOS is to provide accessible and accurate guidance in the design and analysis of observational studies (http://stratos-initiative.org/). Members of the STRATOS Topic Group {\textquoteleft}Evaluating diagnostic tests and prediction models{\textquoteright} are (alphabetically) Patrick Bossuyt, Gary S. Collins, Petra Macaskill, David J. McLernon, Karel G.M. Moons, Ewout W. Steyerberg, Ben Van Calster, Maarten van Smeden, and Andrew Vickers. Funding This work was funded by the Research Foundation – Flanders (FWO; grant G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). The funders had no role in study design, data collection, data analysis, interpretation of results, or writing of the manuscript. Contributions All authors conceived of the study. BVC drafted the manuscript. All authors reviewed and edited the manuscript and approved the final version.",

year = "2019",

month = dec,

day = "16",

doi = "10.1186/s12916-019-1466-7",

language = "English",

volume = "17",

journal = "BMC medicine ",

issn = "1741-7015",

publisher = "BioMed Central",

}

TY - JOUR

T1 - Calibration

T2 - the Achilles heel of predictive analytics

AU - Van Calster, Ben

AU - McLernon, David J

AU - van Smeden, Maarten

AU - Wynants, Laure

AU - Steyerberg, Ewout W

AU - Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative

N1 - Acknowledgements This work was developed as part of the international STRengthening Analytical Thinking for Observational Studies (STRATOS) initiative. The objective of STRATOS is to provide accessible and accurate guidance in the design and analysis of observational studies (http://stratos-initiative.org/). Members of the STRATOS Topic Group ‘Evaluating diagnostic tests and prediction models’ are (alphabetically) Patrick Bossuyt, Gary S. Collins, Petra Macaskill, David J. McLernon, Karel G.M. Moons, Ewout W. Steyerberg, Ben Van Calster, Maarten van Smeden, and Andrew Vickers. Funding This work was funded by the Research Foundation – Flanders (FWO; grant G0B4716N) and Internal Funds KU Leuven (grant C24/15/037). The funders had no role in study design, data collection, data analysis, interpretation of results, or writing of the manuscript. Contributions All authors conceived of the study. BVC drafted the manuscript. All authors reviewed and edited the manuscript and approved the final version.

PY - 2019/12/16

Y1 - 2019/12/16

N2 - BACKGROUND: The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.MAIN TEXT: Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.CONCLUSION: Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

AB - BACKGROUND: The assessment of calibration performance of risk prediction models based on regression or more flexible machine learning algorithms receives little attention.MAIN TEXT: Herein, we argue that this needs to change immediately because poorly calibrated algorithms can be misleading and potentially harmful for clinical decision-making. We summarize how to avoid poor calibration at algorithm development and how to assess calibration at algorithm validation, emphasizing balance between model complexity and the available sample size. At external validation, calibration curves require sufficiently large samples. Algorithm updating should be considered for appropriate support of clinical practice.CONCLUSION: Efforts are required to avoid poor calibration when developing prediction models, to evaluate calibration when validating models, and to update models when indicated. The ultimate aim is to optimize the utility of predictive analytics for shared decision-making and patient counseling.

KW - Calibration

KW - Risk prediction models

KW - Predictive analytics

KW - Overfitting

KW - Heterogeneity

KW - Model performance

UR - http://www.scopus.com/inward/record.url?scp=85077149933&partnerID=8YFLogxK

U2 - 10.1186/s12916-019-1466-7

DO - 10.1186/s12916-019-1466-7

M3 - Article

C2 - 31842878

VL - 17

JO - BMC medicine

JF - BMC medicine

SN - 1741-7015

M1 - 230

ER -

Calibration: the Achilles heel of predictive analytics

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this