When Failure Ranking Misses Intervention Value: A Calibration-Time Audit for Selective Prediction

Selective-prediction systems commonly choose which cases to defer by evaluating uncertainty scores according to how well they rank model failures. At a fixed operating point, a failure-only criterion is appropriate when reviewing a failure has constant value and every review has the same cost. However, when the value of intervention varies across cases with the same failure status, failure ranking can select a poor deferral policy. Global ranking metrics introduce a second mismatch because they summarize performance across thresholds rather than evaluate a particular operating point. We study this problem through excess allocation value, which measures how much better a score-based deferral policy allocates a given review budget than random deferral. We show that this value decomposes into failure enrichment and residual benefit alignment, the latter capturing intervention-relevant information that failure-ranking metrics do not observe. We then introduce threshold-corrected calibration value (TCV), a multiplicity-penalized calibration criterion for selecting among a finite, prespecified set of uncertainty scores and thresholds. On MineThatData with spend-weighted intervention benefit, TCV and Failure AUROC select different scores in 16 of 18 independently retrained predictors. On held-out data, policies selected by Failure AUROC and threshold-specific failure enrichment have negative excess allocation value, whereas the TCV-selected policy achieves 12.5\% of the benefit oracle's value. This ordering persists at fixed review rates of 5\%, 10\%, and 20\%. Experiments across clinical referral, public-policy intervention, and activity recognition show that disagreement is conditional rather than universal. These results suggest that when intervention benefit is heterogeneous, selective-prediction policies should be evaluated by the value of the cases they defer, not only by whether those cases are prediction failures.