Does Random Data Splitting Select the Wrong Model? Ranking Stability of Ten Machine Learning Algorithms Under Temporal Validation in MIMIC-IV Physionet Dataset

Abstract. ML models for intensive care outcome prediction are almost universally benchmarked by randomly partitioning a single dataset, and the highest-scoring algorithm is carried forward. Deployed models, however, always predict forward in time. Using MIMIC-IV v3.1 (n = 32,315 first ICU stays; 2008–2022), we trained ten algorithms to predict in-hospital mortality from the first 24 hours of data and evaluated each under two designs holding cohort, features, preprocessing, tuning budget and seeds identical: a conventional random 70/15/15 split, and a forward temporal split (train 2008–2016, validate 2017–2019, test 2020–2022). Overall ranking agreement was moderate (Kendall’s τ-b = 0.778), but the algorithm a researcher would have selected changed: XGBoost under random splitting, LightGBM under temporal validation. Random forest moved two places. High aggregate agreement therefore did not protect the single decision that matters in practice. The margin separating the top two algorithms (0.0098 AUPRC) was smaller than the largest across-seed standard deviation (0.0393). Because prevalence rose from 11.2% to 13.6%, raw AUPRC appeared to improve by 4.5%; after normalising by prevalence, discrimination fell by 13.9%. Model selection in clinical ML is usually justified by benchmarks run on a random split of one dataset. This assumes training and test data come from the same distribution, which fails whenever a model trained on past data is applied to future patients. Prior work has measured how far ICU model performance decays under temporal shift and tested ways to reduce it [3, 4], but has looked at one model class at a time. We ask instead whether the ranking of algorithms is stable: does the algorithm a study would select under a random split remain the best choice under forward-in-time validation? Methods: From MIMIC-IV v3.1 [1] we took the first ICU stay of each adult patient lasting at least 30 hours, excluding stays ending inside the prediction window, and kept only stays whose era could be assigned without ambiguity (n = 32,315; 3,611 deaths, 11.2%). The 260 predictors come only from the first 24 hours: demographics, vital signs and Glasgow Coma Scale, laboratory results, comorbidity flags, and an indicator for whether each laboratory test was ordered. Two input formats were built from one extraction: summary statistics for the tree and linear models, and an hourly 24 × 22 tensor with a missingness mask for the sequence models, so that differences reflect the model, not the preprocessing. Ten algorithms were compared: XGBoost, LightGBM and CatBoost; a multilayer perceptron and an FT-Transformer; and LSTM, GRU-D and a Temporal Fusion Transformer. Modified SOFA and SAPS-II scores were computed as a non-learned reference line. Each was given the same tuning budget (30 Optuna trials per design, tuned on validation AUPRC) and five seeds. Imputation and scaling were fitted on training data only, so in Design B they come from 2008–2016 and are applied unchanged to 2020–2022. AUPRC was selected as the primary metric. Result: Aggregate agreement was moderate, but the selected model changed. Kendall’s τ-b between the two rankings was 0.778, so most pairs of algorithms kept their relative order. Ranking by AUROC gave the same value. XGBoost ranked first under random splitting and second under temporal validation, while LightGBM took first place, and random forest moved from sixth to eighth. Because only the top-ranked model is normally carried forward, a study using the conventional design would have selected a different algorithm. The margin between the top two under Design A was 0.0098 AUPRC, smaller than the largest across-seed standard deviation observed (0.0393), i.e. within run-to-run noise. Bootstrap resampling gave a median τ of 0.51 (95% interval 0.156–0.778), so with ten algorithms and 807 test events the ranking is imprecisely estimated. Raw AUPRC rose by 4.5% under temporal validation, which would suggest robustness. AUPRC increases with outcome prevalence, and the test era prevalence is 13.6% against 11.2%. Normalising by prevalence reverses the sign, giving a mean degradation of 13.9%; AUROC, which is prevalence-insensitive, was flat (−0.3%), consistent with rank ordering of risk being conserved while absolute precision deteriorates. Loss was largest for boosted trees (−16.2%) and smallest for penalised logistic regression (−10.0%). Sequence models ranked 8th–10th under both designs, and the highest-capacity tabular model needed roughly 40 times the tuning compute of LightGBM while ranking below it.