Preserving Clinical Facts Does Not Preserve Empathy: A Directional-Inflation Diagnostic for LLM-Generated Counterfactual Audits
Abstract
Clinical empathy systems must determine not only how to respond empathetically, but also when empathetic behaviors are appropriate. This study examines whether empathy-applicability judgments remain stable when clinical facts are preserved but communication style varies. Using the Empathy Applicability Framework, we generate 500 style-counterfactuals from 30 seed queries through a constrained agentic pipeline that preserves clinical facts while varying contextual cues and expression. GPT-4o evaluations show substantial and directional label shifts, with 40.0% of variants changing on at least one empathy-applicability dimension. Inspection reveals that generated variants can introduce affective content absent from the original queries, confounding style with emotionally relevant information. We introduce a directional-inflation diagnostic, characterized by asymmetric base-rate movement and ceiling effects, to identify this failure mode. These findings highlight a broader limitation of LLM-generated, LLM-judged counterfactual evaluation and motivate stronger controls for trustworthy clinical AI auditing.