Beyond Reactivity: Proactive Adaptive Conformal Inference for Online LLM Factuality

Large language models can produce factually incorrect outputs even when their responses are fluent, making reliable uncertainty control important for high-stakes use. Conformal prediction provides finite-sample coverage guarantees under exchangeability, but online distribution shifts can invalidate static calibration. We propose PACE (Proactive Adaptive Conformal InferencE), which augments reactive error feedback with a proactive signal derived from the incoming query. Guided by theoretical analysis, PACE combines recent calibration mismatch with Relative Mahalanobis Distance-based shift detection to adapt the update magnitude while retaining error feedback for the update direction. We evaluate PACE on synthetic streams and two LLM deployment scenarios, MMLU and WikiData. Compared with dynamically tuned adaptive conformal inference, PACE reduces tracking error by approximately 60% on MMLU and shortens recovery from 24.6 to 3.0 steps. On WikiData, it maintains factuality close to the 80% target without requiring more aggressive filtering, demonstrating that input-side shift information can reduce adaptation lag while preserving output utility.