Behaviorally-consistent explanations for clinical LLMs via counterfactual finetuning

LLMs are increasingly being deployed in high-stakes settings such as healthcare, where a model is tasked with making a decision (say, determining diagnosis for an emergency room case) and offering a supporting explanation. For it to be trustworthy, an explanation must be consistent with the model's behavior. In other words, the factors stated in an explanation must influence the model's output. However, verifying an explanation for behavioral consistency is difficult since LLM explanations are largely post hoc rationalizations and may state plausible but not necessarily actual factors that influence the model's output. In addition, the unstructured nature of explanations makes verifying them a challenge. In this work, we propose to train LLMs to generate structured and behaviorally-consistent explanations that are verifiable. Specifically, we train models to generate counterfactual explanations that are structured as atomic claims. Counterfactual explanations permit verifiability: by training the LLM to state minimal changes that will change its prediction, we obtain testable claims that can be verified by intervening on the input and checking for prediction change. Moreover, the structured nature allows easier intervention and verifiability.