Toward Trustworthy Hypotheses: Interpretable Clinical Prediction with LLMs

Large language models (LLMs) can translate patient profiles into natural language hypotheses, offering a promising path toward interpretable clinical prediction. However, directly prompted hypotheses are often clinically unreliable and lack specificity, and no existing pipeline verifies whether their associations hold on patients that were not used to form them. We introduce CLIP-H (Clinical Latent Interpretation \& Prediction via Hypotheses), a framework that grounds hypothesis generation in latent patient representations and filters out unreliable outputs. CLIP-H generates hypotheses in three phases. It first learns latent concepts from patient-profile embeddings with a Top-$K$ sparse autoencoder and selects those associated with the outcome. An LLM then names each selected concept in natural language, and names that fail to reproduce the concept's firing pattern are discarded. Then, a curation stage (i) refines hypotheses through an iterative critic–generator self-refine control loop grounded in representative patient profiles, (ii) removes implausible or outcome-leaking hypotheses through LLM-based critique with majority voting, (iii) merges behaviorally equivalent concepts by their patient-level firing patterns, and (iv) retains only hypotheses whose associations generalize to held-out patients. The resulting hypothesis library serves as an interpretable feature space for downstream predictions. In a blinded clinical review in which the reviewer could not tell which method produced each hypothesis, 94.6\% of CLIP-H hypotheses were judged medically sound, against 81.3\% for direct generation. Across five MIMIC-IV prediction tasks, CLIP-H improves AUROC over direct hypothesis generation on every task and both LLM backbones, and approaches strong feature-based baselines while making every prediction decomposable into named, auditable clinical concepts.