Predictive Abilities of Machine Learning Techniques May Be Limited by Dataset Characteristics: Insights From the UNOS Database

P Elliott Miller 1, Sumeet Pawar 1, Benjamin Vaccaro 1, Megan McCullough 1, Pooja Rao 2, Rohit Ghosh 2, Prashant Warier 2, Nihar R Desai 3, Tariq Ahmad 4

Affiliations

PMID: 30738152
DOI: 10.1016/j.cardfail.2019.01.018

Abstract

Background: Traditional statistical approaches to prediction of outcomes have drawbacks when applied to large clinical databases. It is hypothesized that machine learning methodologies might overcome these limitations by considering higher-dimensional and nonlinear relationships among patient variables.

Methods and results: The Unified Network for Organ Sharing (UNOS) database was queried from 1987 to 2014 for adult patients undergoing cardiac transplantation. The dataset was divided into 3 time periods corresponding to major allocation adjustments and based on geographic regions. For our outcome of 1-year survival, we used the standard statistical methods logistic regression, ridge regression, and regressions with LASSO (least absolute shrinkage and selection operator) and compared them with the machine learning methodologies neural networks, naïve-Bayes, tree-augmented naïve-Bayes, support vector machines, random forest, and stochastic gradient boosting. Receiver operating characteristic curves and C-statistics were calculated for each model. C-Statistics were used for comparison of discriminatory capacity across models in the validation sample. After identifying 56,477 patients, the major univariate predictors of 1-year survival after heart transplantation were consistent with earlier reports and included age, renal function, body mass index, liver function tests, and hemodynamics. Advanced analytic models demonstrated similarly modest discrimination capabilities compared with traditional models (C-statistic ≤0.66, all). The neural network model demonstrated the highest C-statistic (0.66) but this was only slightly superior to the simple logistic regression, ridge regression, and regression with LASSO models (C-statistic = 0.65, all). Discrimination did not vary significantly across the 3 historically important time periods.

Conclusions: The use of advanced analytic algorithms did not improve prediction of 1-year survival from heart transplant compared with more traditional prediction models. The prognostic abilities of machine learning techniques may be limited by quality of the clinical dataset.

Keywords: Advanced analytics; heart transplantation; prediction algorithms.

Comment in

The Promise of Machine Learning: When Will It Be Delivered?
O Akbilgic et al. J Card Fail 25 (6), 484-485. Jun 2019. PMID 30978508.
Most of the performance-focused critics on machine learning are because the bar is set unfairly too high for machine learning. In most cases, machine learning methods pro …