Counterfactual Legal Reasoning Graphs: Separating Correct Answers from Correct Reasoning

As large language models answer legal questions with increasing accuracy, justice systems are relying on AI to detect crime patterns, draft police reports, classify cases, retrieve precedent and assess risk. Axon Draft One, for example, may correctly identify every person, statement and event in body-camera audio while attributing a statement or action to the wrong actor. Likewise, judicial AI may retrieve the correct precedent while overlooking a legally material difference between cases. How can public actors compare opaque AI systems beyond accuracy? As opacity shapes public debate over legal AI, we introduce Counterfactual Legal Reasoning Graphs (CLRG), an evaluation framework that tests whether an LLM changes its answer when legally material relations change while remaining invariant to legally irrelevant edits. We combine three elements: expert-specified counterfactual interventions on legal relations rather than arbitrary token edits; joint sensitivity and invariance metrics; and a constant-output baseline. To operationalize the interventions, CLRG maps each scenario as a typed directed graph $G = (V, E)$ whose nodes encode facts, norms, and outcomes, and whose typed edges encode which norms take priority. This separates two failure sources: extracting the legally relevant structure, and applying the correct rule. A legal expert defines outcome-changing interventions $\Delta^{+}$, such as changing a precedent-relevant fact, and outcome-preserving interventions $\Delta^{0}$, such as party names or dates. For system $f$ and label $y^{\star}$, we measure $$\mathrm{Sens}(f) = \frac{1}{|\Delta^{+}|} \sum_{\tilde{s} \in \Delta^{+}} \mathbf{1}[f(\tilde{s}) = y^{\star}(\tilde{s})], \qquad \mathrm{Inv}(f) = \frac{1}{|\Delta^{0}|} \sum_{\tilde{s} \in \Delta^{0}} \mathbf{1}[f(\tilde{s}) = f(s)].$$ We report both because either can be trivially maximized alone. We additionally define the Consistent Reasoning Score (CRS), which credits a scenario only when the original and all of its counterfactuals are correct. The illusion gap $\Gamma = \mathrm{Acc} - \mathrm{CRS}$ then measures how much conventional accuracy overstates consistency. We compare against a constant-output baseline that always predicts the original outcome; matching it indicates no demonstrated sensitivity. Pilot. Scenarios are expert-authored around judicial precedent applicability with binary outcomes; the expert records material facts, the relevant precedent, and whether it applies. Counterfactual text is generated from these specifications by an LLM and then expert-verified. CLRG retrieves the governing text and prompts the model to extract the typed reasoning graph. A second call, conditioned on that graph, predicts the outcome; when extraction fails, the system falls back to direct prediction. A control with the same retrieval and call budget skips graph extraction and predicts directly from text. We ran three scenarios. All systems answer the original question correctly on the same two scenarios. We restrict analysis to this diagnostic subset, holding original-answer accuracy constant. Even the constant-output baseline achieves 2/2, so conventional accuracy cannot distinguish the systems. The prompted LLM fails all outcome-changing probes and incorrectly changes predictions after two outcome-preserving edits, while RAG answers all eight probes correctly, a ceiling at this probe count rather than evidence of reliable reasoning. Reporting Acc, CRS, $\Gamma$, Sens, Inv for each system: constant-output baseline (analytic), 2/2, 0/2, 1.00, 0/3, 5/5; prompted LLM, 2/2, 0/2, 1.00, 0/3, 3/5; LLM + RAG, 2/2, 2/2, 0.00, 3/3, 5/5; LLM + RAG two-stage control, 2/2, 1/2, 0.50, 2/3, 5/5; LLM + RAG + CLRG, 2/2, 1/2, 0.50, 2/3, 5/5. This is a pilot with Qwen3-8B and one sample per item; counts, not rates. The diagnostic subset contains 2 scenarios, 3 outcome-changing probes, and 5 outcome-preserving probes. No statistical significance is claimed. The pilot does not establish a benefit from explicit graph extraction, which matches the two-stage control and underperforms RAG. Yet this reinforces CLRG's evaluative contribution. In a public debate dominated by opacity, CLRG changes the question from how an AI system works to whether it behaves reliably when legally consequential relations change, giving public actors a reproducible basis for comparing competing systems beyond headline accuracy.