PLLS-CP: Unsupervised Hallucination Detection with Layer Selection and Cross-Domain Conformal Guarantees

Ensuring reliability of large language models (LLMs) is critical for their safe deployment in autonomous and resource-constrained applications, where hallucinations can lead to incorrect or harmful outputs. Detecting hallucinations from internal model states without external knowledge is promising, but existing methods lack distribution-free guarantees and fail under domain shift. We introduce PLLS-CP, a unified framework for hallucination detection that addresses both challenges. First, PLLS uses pseudo-label probing across transformer layers to identify the most informative ones, and trains an unsupervised detector on the selected layers. Then, a Mondrian conformal layer wraps the detector to provide provable coverage guarantees. Under domain shift, only the conformal threshold is updated on a small target calibration set — leaving the PLLS-trained detector fixed and avoiding retraining. We provide theoretical guarantees: marginal coverage under cross-domain recalibration, exponential concentration of the coverage gap, and non-degradation of the validation-selected ensemble. Experiments on 18 domains from TrueFalse, HaluEval, and HELM show that PLLS-CP substantially improves accuracy retention under cross-domain transfer while maintaining empirical coverage close to the nominal target level. On informative domains the method loses less than 2.5% of selective accuracy under transfer, suggesting a practical path to deployable, distribution-shift-robust hallucination detectors.