An Individual-Level Stability Framework for Evaluating Clinical Risk Prediction Models

During validation, clinical prediction models (CPMs) are commonly evaluated using aggregate metrics, yet in practice they inform decisions for individual patients. Resampling methods such as bootstrapping are typically used to estimate generalization performance and can also characterize sampling-induced instability in individual predictions. However, they do not generally isolate data-induced instability from instability caused by stochastic model fitting—a distinction that is increasingly relevant as flexible, stochastic CPMs become more widely used. Consequently, standard model selection may overlook two implementation-relevant properties: how the spread of individual risk estimates changes with model flexibility and how often repeated fits of the same pipeline yield different threshold-based decisions. We propose empirical prediction interval width (ePIW) and empirical decision flip rate (eDFR), model-agnostic extensions of resampling-based measures of prediction spread and decision disagreement that can be applied under distinct pipeline perturbations, including training-set resampling and changes to seed-dependent components of model fitting. Using controlled simulations, we show that individual-level stability depends jointly on model flexibility, data dimensionality, signal strength, and whether perturbations arise from the training sample or the fitting procedure. Prediction instability peaks near the interpolation boundary and only partially recovered in the overparameterized regime, with weaker outcome signal amplifying both (Figure 1). For flexible stochastic pipelines (e.g., neural networks), optimization-induced instability was comparable in magnitude to instability caused by resampling the training data. Localizing ePIW and eDFR across the risk spectrum revealed regions in which pipelines with similar aggregate performance produced different individual risk estimates and threshold-based decisions. When the empirical distribution of an individual’s predicted risks crossed a clinically meaningful threshold, repeated pipeline fits could yield different clinical recommendations. Stronger regularization and larger training samples reduced instability, while simpler, deterministic pipelines were largely insensitive to fixed-data fitting perturbations. Individual-level pipeline stability can be a complementary model-selection criterion for determining whether the predictive benefits of added model flexibility justify its stability cost. We further emphasize that stability-based model selection should be task-dependent, as a pipeline can yield highly consistent threshold-based decisions while producing continuous risk estimates that remain sensitive to learning-pipeline perturbations.