When Evidence Changes, Does the Literature Change? A Full-Text Audit of External-Validation Evidence for the Epic Sepsis Model
Abstract
External validation can materially change what readers should believe about a biomedical machine-learning system, but that update matters only if later scientific communication carries the evidence forward. We study this problem through a longitudinal case audit of the Epic Sepsis Model (ESM) after a 2021 independent external validation. An initial metadata-based audit identified 10 post-validation publications, all of which cited the validation. We therefore rebuilt the downstream denominator using Europe PMC and PMC full text. The search returned 210 unique post-validation candidates; 167 had retrievable open full text and 155 passed an exact model-name screen. After removing 46 reference-list-only hits, five table/figure-only hits, and one ESM-v2-only record, the primary corpus contained 103 v1-era or version-unclear prose mentions. Only 8/103 (7.8%) were visible to the exact title/abstract search used for the narrow audit; 95/103 (92.2%) required full text. Among 96 primary papers with resolvable reference lists, 75 directly cited Wong et al. 2021 (78.1%, Wilson 95% CI 68.9–85.2), rather than 100% in the initial selected subset. The central finding is methodological: estimates of evidence propagation in biomedical ML can change substantially with denominator construction. We release a deterministic, cache-backed workflow and argue that communication audits should be full-text, temporally ordered, and explicit about model version and the difference between citation and substantive engagement.