When Do Language Models Cave? A Label-Noise Ceiling in Legal Sycophancy Detection

Large language models are sycophantic, endorsing a user's asserted conclusion rather than reasoning consistently, causing a safety risk in legal analysis, where when it occurs and how reliably it can be detected remains unclear. SycoLex is a public benchmark of real court cases in which a model is queried twice about a case outcome, told each side won in turn, and labelled sycophantic only if it agrees with both contradictory assertions, with paired human and LLM-judge labels. Analysis of these dual labels shows they agree on only $\approx$70% of the labelled subset (Cohen's $\kappa \approx 0.34$, fair agreement), suggesting the task carries substantial label noise. Building on this benchmark, a response-centric detector is proposed: it reads the two model responses rather than the prompt wording, tags each with the side its prompt asserted, truncates head-and-tail, and randomizes response order, so surface prompt style cannot be exploited. Training and validation use six prompt variants (explain-why, confused-user, devil's-advocate, hypothetical-bet, leading-loaded, tentative), while the test set uses ten unseen variants, implying a genuine distribution shift. Across four encoders (RoBERTa, XLM-RoBERTa, InLegalBERT, MuRIL), a two-stage decomposition, and an ensemble, a straightforward RoBERTa detector reaches $\approx$0.685 F1 (multi-seed), and neither the specialized encoders, decomposition ($\approx$0.63), nor the ensemble ($\approx$0.69–0.71) surpass it, consistent with a label-noise ceiling; leave-one-variant-out shows lift on held-out variants, indicating transfer rather than memorization. Hand annotation of the unseen test prompts reveals sycophancy is structural: concentrated in frame-imposing variants that make the model advocate a side or continue a sentence already naming a winner ($\approx$0.6–0.75) and near-zero in verification and edge-case variants; against these annotations the detector attains $\approx$0.77–0.80 F1. These findings recast legal sycophancy detection as a data-centric problem bounded by label reliability rather than model capacity, and the frame-imposing versus verification distinction gives a concrete axis for auditing sycophancy in deployed systems.