When Confidence Predicts Failure: Uncertainty Decomposition in Reinforcement Learning under Train-Deploy Distribution Mismatch

Reinforcement learning agents for rare-event detection are routinely trained on artificially balanced experience, because training on the true distribution collapses to the trivial all-negative policy. The agent then inherits a decision boundary calibrated to the training prior rather than the deployment prior. We show that under severe mismatch the effect is not merely overconfidence but a reversal: confidence becomes an anti-predictor of correctness, ranking predictions worse than chance. We train a Rainbow DQN agent with a C51 distributional head on episodes built at a 90:10 negative:positive ratio, then evaluate at 0.86% positive (CTU-13) and 19.7% positive (CICIDS-2017). For 10,000 stratified test samples per dataset we compute six uncertainty signals and regress binary correctness on them. Three are internal, taken from the agent's own output distribution: the probability assigned to the selected action, the entropy of that distribution, and the standard deviation across the C51 atoms. Three are external, computed from the test point's geometry relative to frozen training data: aleatoric, epistemic, and distributional shift. Under severe mismatch, confidence carries a negative coefficient and its AUPRC for ranking correct predictions falls below the random baseline, while entropy is the strongest positive predictor. Internal signals carry 91% of explained variance against 9% for external ones, so the failure lies in the agent's self-representation rather than in noisy or out-of-support inputs. The effect is graded: at 19.7% positives the inversion weakens and explained variance drops from 0.86 to 0.26. Post-hoc temperature scaling does not repair it, with T=0.841 moving ECE from 0.445 to 0.448, because bin accuracy is non-monotonic in confidence and no monotonic scalar transform can reorder it. We frame this as a calibration-detection dilemma: balanced experience is what lets the agent detect anything at all, and is also what breaks the meaning of its confidence at deployment. Entropy rather than confidence should gate autonomous action under mismatch, and calibration must be measured on the deployment prior before any confidence-based control policy is trusted.