Where to Trust Large Language Models in Peer Review?

Large language models (LLMs) are increasingly used in academia. Whether they can be trusted in peer review remains unclear. The process comprises distinct roles, but existing evaluations focus primarily on how well LLMs write reviews. We evaluate nine models on 700 papers from ICLR, NeurIPS, and ICML across the five roles of the peer-review pipeline: writing reviews, scoring evidence, rating individual strengths and weaknesses, handling human-authored rebuttals, and making accept-or-reject decisions. We isolate these roles through controls that hold input evidence fixed and replace humans with models one stage at a time. We find that model reliability is role-dependent. As review writers, LLMs identify strengths and weaknesses that overlap with those in human reviews, but coverage varies across models. Models miss some points identified by humans while raising valid points that humans did not identify. Differences in which strengths and weaknesses models identify account for more of the disagreement between models than differences in how they rate the same items. As scorers, models assign scores close to mean human rating, whether they judge the evidence as a whole or rate it item by item. Scoring item by item, with a fixed aggregation formula, also reduces most of the observed disagreement between models. We also measure how confident models are when they score. Models are no less confident on borderline papers than on clear ones. In handling rebuttals and making decisions, LLMs tend to be more permissive. After models read human-authored rebuttals, their scores increase without moving closer to each paper's mean human reviewer rating. In an LLM-simulated review, rebuttal, and decision process, models accept 81\% of papers in a sample in which half were accepted at the venue. Peer review can benefit from LLMs as a second opinion for review writing and scoring, particularly when used with structured rubrics and a fixed aggregation formula. However, our findings also highlight important limitations and the continued need for human judgment throughout the review process.