Talking a Rejection Into an Accept: How Confidence Framing Fools (Some) LLM Reviewers
Prabhjot Singh ⋅ Somnath Luitel ⋅ Manmeet Singh
Abstract
Large language models already review real submissions at real venues: AAAI-26 generated an AI review for all 22,977 papers that reached full review, and EMNLP 2026 is piloting a comparable system. Prior work treats the “the AI reviewer” as a single actor and reports pooled bias estimates, leaving open a more consequential question: is style sensitivity a general property of LLM reviewers, or concentrated in specific models? We test 9 open-weight reviewer models spanning three families and multiple parameter scales on 100 real ICLR 2026 and ICML 2026 abstracts (70 rejected, 30 accepted), rewriting each into five controlled tones while holding every factual claim fixed. Hedging is punished in all 9 of 9 models, the only effect that replicates without exception ($\beta = -0.367$, $p < 0.001$). Overclaiming is punished on average but splits models in opposite directions, from a large penalty in one model (Cohen's $d = -1.64$) to a significant reward in others ($d = +0.69$). Pooled metrics hide concentrated harm: L2-grounded phrasing shows no average score effect yet still flips decisions in the same susceptible models. Under FDR-corrected McNemar tests, confidence reframing flips a genuine rejection into an accept in up to 35.7\% of eligible cases, but only in 2 of 9 models with adequate sample size; the remaining 6 show no decision-level sensitivity, several because their unmodified-text judgments already fail to track real human review (per-model correlation with human scores ranges from $-0.078$ to $+0.299$; Krippendorff's $\alpha = 0.105$). Which reviewer model is deployed explains more variance in a paper's score (between-model SD $\approx 0.91$) than any style manipulation we test. Heterogeneity is not a caveat to the confidence-framing story: it is the story, and the actionable lever is model-selection auditing, not blanket AI-reviewer distrust.
Chat is not available.
Successful Page Load