The Repairability Gap in Compound AI Routing: Why Uncertainty Is Not Enough for Model Escalation

Cascaded and routed compound-AI systems escalate to a stronger fallback model when a baseline prediction looks uncertain, implicitly treating uncertainty as a proxy for whether escalation will help. We test this assumption across four benchmarks (FEVER, SimpleQA, TruthfulQA, MMLU) using a fork-vote pipeline, and across three independent model families: GPT-4o-mini$\to$GPT-4o (full scale, $N=684$-$3{,}000$ per dataset), a small Llama-3.1-8B$\to$Llama-3.3-70B check, and Qwen3.6-Flash$\to$Qwen3.7-Plus at the same full scale as GPT. We define repairability as the probability that fallback converts an incorrect baseline prediction into a correct one, and show that although oracle routing headroom is large (up to $+0.215$ accuracy) and repair events are common, standard uncertainty and disagreement features predict repairability only weakly for GPT (AUROC $0.531$-$0.569$) but meaningfully better for Qwen on two of four datasets (FEVER $0.644$, MMLU $0.679$; confirmed not a label-identity artifact via a stratified audit). Cross-model transfer is correspondingly weak and asymmetric. Together these results show repairability is not simply weak or strong in general: it is deployment-specific, varying by task and by model pair, so uncertainty alone is not a reliable universal proxy for escalation utility and routers require per-task, per-model-pair calibration rather than a fixed global threshold. We conclude that uncertainty-aware routing should be evaluated as an intervention-selection problem: systems should estimate conditional repair and harm under the deployed model pair, rather than treating uncertainty or fallback-model accuracy as universal proxies for escalation utility.