Auditing Identity Bias in Large Language Model Legal Outcome Predictions

Large language models (LLMs) are increasingly integrated into legal research, case summarization, and decision-support systems, raising concerns about whether their outputs are influenced by legally irrelevant identity attributes. Existing benchmarks largely emphasize predictive accuracy or concordance with historical outcomes, offering limited insight into identity-based bias when case facts are held constant. This paper introduces a controlled auditing framework for measuring identity sensitivity in legal-domain LLMs through matched counterfactual vignettes. Paired cases preserve all legally material facts while systematically varying identity cues—including party names, employer characteristics, and socioeconomic indicators. Multiple LLMs are evaluated under identical prompting conditions, with predicted outcomes, confidence scores, and reasoning compared via paired statistical analysis. Preliminary results indicate that identity-attribute variation produces measurable differences in predictions, confidence calibration, and explanatory reasoning despite fact-invariant scenarios, suggesting that legal LLMs may encode subtle identity-associated biases undetected by conventional accuracy metrics. The proposed framework offers a lightweight, reusable methodology for auditing identity bias in legal AI, contributing toward the development of fairer and more transparent decision-support tools for high-stakes legal contexts.