Skip to yearly menu bar Skip to main content


NeurIPS 2026 AI-Assisted Reviewing Experiment

Objective

The purpose of this experiment is to understand how NeurIPS reviewers could interactively engage with large language models (LLMs) for peer review. We will use a custom NeurIPS-specific LLM interface integrated into OpenReview for this experiment.

The Experiment

In this experiment, participant reviewers will be randomly assigned to one of three conditions per paper assigned to them. Condition 1 is an unassisted peer review. Condition 2 and 3 offer the opportunity to interact with an LLM through the OpenReview interface to assist in peer review. Condition 2 will be open-ended with minimal guidance, while condition 3 will include more structured guidance for interaction with the LLM. Area chairs, blind to the condition of the review, will be asked to provide review assessments. We will analyse the impact of the different conditions on the review, and the types of interactions the reviewers engaged in.

Participants and Inclusion Criteria

Participants will be NeurIPS reviewers who consent to participate in the experiment. The papers assigned to participants in the experiment will be assigned based on those whose authors consent to include their paper in the experiment.

Data Collection

All reviewer interactions with the LLM, including inputs and outputs, will be logged, along with the final review and survey responses. No identities - author, reviewer, AC, PC, or any others - will be logged. All identities will be replaced with randomized strings.

Contact Information

  • Principal Investigator: Joydeep Biswas, joydeepb@cs.utexas.edu (for questions about the study)
  • Institutional Review Board (IRB): The University of Texas at Austin Institutional Review Board, Phone: 512-232-1543, Email: irb@austin.utexas.edu (for questions about your rights as a participant)