Skip to yearly menu bar Skip to main content


Area Chair (AC) Guidelines

Thank you for agreeing to serve on the program committee of NeurIPS 2021! As an area chair (AC), your job is to ensure that all the submissions you are assigned have high quality reviews and good discussions. You should become familiar with the technical contents of all your submissions and are responsible for making the initial acceptance decisions, with guidance from your senior area chair (SAC).

If you encounter a situation that you are unable to resolve on your own:

  • If it is an issue involving OpenReview, please contact neurips2021@openreview.net.

  • If it is a general question, please contact the program chairs at neurips2021pcs@gmail.com.

  • If you have specific questions related to the handling of a particular paper, please email your SAC. You can find their name and email address at the top of your AC console in OpenReview.

Main Tasks

  1. Preparation:

    • Please ensure that your preferred email address is accurate in your OpenReview profile. We will send most emails from OpenReview (noreply@openreview.net). Such emails are sometimes accidentally marked as spam (and sometimes as updates in Gmail). Please check your spam folder regularly. If you find such an email in there, please whitelist the OpenReview email address so that you will receive future emails from OpenReview.

    • Please log into OpenReview and make sure that your profile is up to date.

    • Read and agree to abide by the NeurIPS code of conduct.

    • Read what constitutes conflict of interest for NeurIPS 2021.

    • In addition to the guidelines below, please familiarize yourself with the reviewer responsibilities and guidelines. You will be interacting significantly with reviewers, so please make sure you understand what is expected of them.

  2. Bid on papers: Monday, May 31, 2021 – Monday, June 7, 2021.

    • Log into OpenReview and bid on submissions that you’d like to oversee.

  3. Modify reviewer assignment: Monday, June 14, 2021 – Monday, June 21, 2021.

    • Make sure that every submission in your batch is matched with suitable reviewers whom you can trust on this submission, ideally with a diverse set of opinions.

    • It really pays off to invest some time before the reviewing process starts to ensure that your batch has expert reviewers!

    • You can continue inviting reviewers as needed after the assignment is released to reviewers on June 21.  Note that because the review process is already underway, reviewers will need to agree to review each additional paper before they can be assigned.  You can follow these instructions for sending invitations to review a specific paper.

  4. Ensure all papers have quality reviews: Friday, July 16, 2021 – Tuesday August 3.

    • Initial reviews are due Friday, July 16. You might have to send multiple reminder emails, but please be courteous. If a reviewer is unable to deliver a review, please find a replacement reviewer.

    • Read all reviews carefully. If a review is substandard, you should ask the reviewer to improve their review. Please remember to be polite and provide concrete guidance.

    • You may also need to get an additional review if the current reviews are particularly unhelpful, either in the existing pool or by inviting external reviewers.

    • Make sure that any questionable papers are flagged for ethics review. These papers will be assigned ethics reviewers, who will effectively join the paper's assigned program committee. Their reviews will be released to the authors on August 20, and authors will have an opportunity to respond to these reviews.

  5. Discuss with reviewers and authors: Tuesday, August 10, 2021 – Thursday, September 2 2021.

    • As soon as the author response is entered in the system, initiate and lead a discussion via OpenReview for each submission, and make sure the reviewers engage in the discussion phase.

    • Monitor and moderate the discussion to ensure that it is respectful of everyone’s opinion. Read the submissions in your stack to steer the discussion towards the most critical aspects that need discussion.

    • Make sure your reviewers read and respond to all author responses. To minimize the chance of misunderstandings during the reviewing process, you can also initiate a rolling discussion with the authors after initial reviews and author responses are submitted. You can restrict visibility of your comment to any set of readers (authors, reviewers, senior area chair, or program chairs) as appropriate.

  6. Make initial accept/reject decisions with SACs: Thursday, September 2, 2021 – Thursday, September 9, 2021.

    • Meet with your SAC to discuss your decisions, especially the borderline cases as well as ones where you are making a recommendation that goes against the reviewers. The SAC will help you calibrate across other ACs.

    • Write a meta-review that explains your decision (accept, reject, borderline) to the authors. Your comments should augment the reviews, and explain how the reviews, author response, and discussion were used to arrive at your decision. Do not dismiss or ignore a review unless you have a good reason for doing so. If the reviewers cannot come to a consensus, you should read the paper carefully and write a detailed meta-review. You are expected to discuss such difficult cases with your SAC.

    • Rate the reviewers. You should use the information in the reviewer instructions, as well as how helpful the review and subsequent discussion by the reviewer were in making your decision about the submission. The lowest-scoring reviewers may not be invited to review for future conferences.

  7. Notification: Tuesday, September 28, 2021.

Best Practices

  • Please respect deadlines and respond to emails as promptly as possible.

  • It is okay to be unavailable for part of the review process (e.g., on vacation for a few days), but if you will be unavailable for more than that—especially during important windows (e.g., discussion, decision-making)—you must let your SAC know as soon as you can.

  • If you notice a conflict of interest with a submission that is assigned to you, please contact your SAC immediately so that the paper can be reassigned.

  • Be professional and listen to the reviewers, but do not give in to undue influence. We expect you to be familiar with all the papers that are assigned to you and to be able to argue about their technical content and contributions. Your responsibility is to make good decisions, not just facilitate reviewer discussions.

  • Be kind. It is important to acknowledge that personal situations, in particular during this year of a global pandemic, may lead to late or unfinished work among reviewers. In the event that a reviewer is unable to complete their work on time, we encourage you to be considerate of the personal circumstances; you might have to pick up the slack in some cases. In all communications, exhibit empathy and understanding.

  • DO NOT talk to other ACs about submissions that are assigned to you; other ACs may have conflicts with these submissions. If you feel that it’s important to discuss one submission in the context of another, please email neurips2021pcs@gmail.com. In general, your primary point of contact for any discussions should be your SAC. Your SAC does not have any conflicts with any of the submissions that are assigned to you.

  • DO NOT talk to other SACs or ACs about your own submissions (i.e., submissions you are an author on) or submissions with which you have a conflict of interest.

  • If you notice unethical or suspect behavior involving either authors or reviewers, please notify your SAC and program chairs.

Writing the meta-review

These guidelines are due to Chris Williams and John Lafferty and lightly adapted.

  • Don't focus too much on the scores. Instead, look carefully at the comments. Judge the quality of the review rather than taking note of the reviewer's confidence score; the latter may be more a measure of personality.

  • Indicate that you have read the author response, even if you just say "the rebuttal did not overcome the reviewer's objections."

  • If you use information that is not in the reviews (e.g., from corresponding with one of the reviewers after the rebuttal period), tell the authors (a) that you have done so and (b) what that information is.

  • If you find yourself wanting to overrule a unanimous opinion of the referees, the standards for your summary should be at the level of a full review. In these cases, it would probably be best to solicit an auxiliary review.

  • Please attempt to take a decisive stand on borderline papers. Other than papers where there is a genuine disagreement, much of our work will involve borderline papers where no one confidently expresses excitement, nor are any major problems identified. These are the tough decisions where we need your judgement!

  • Try to counter biases you perceive in the reviews. Unfashionable subjects should be treated fairly but often aren't, to the advantage of the papers on more mainstream approaches. To help the NeurIPS community move faster out of local minima, it is important to encourage risk and recognize that new approaches can't initially yield state-of-the-art competitive results. Nor are they always sold according to the recipes we are used to.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code and results from submissions in your own work until they become publicly available (e.g., via a technical report or a published paper for ideas/results, via open source for code, etc.). Do not talk about or distribute submissions (whether it is the code, or the ideas and results described in them) to anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed.