Skip to yearly menu bar Skip to main content


2024 Area Chair (AC) Guidelines

Thank you for agreeing to serve on the program committee of NeurIPS 2024! As an area chair (AC), your job is to ensure that all the submissions you are assigned have high quality reviews and good discussions. You should become familiar with the technical contents of all your submissions and are responsible for making the initial acceptance decisions, with guidance from your senior area chair (SAC).

If you encounter a situation that you are unable to resolve on your own:

  • If it is an issue involving OpenReview, please contact info@openreview.net.
  • If it is a general question, please contact the program chairs at pc2024@neurips.cc.
  • If you have specific questions related to the handling of a particular paper, please leave a comment on the OpenReview forum for that paper with the readers set to SACs and PCs.

Important Dates

 

Dates may be subject to change.

Reviewers bid on papers:  Thur, May 23  – Thur, May 30
Check paper assignments: Mon, June 5
First round reviewing: Mon, June 10 – Fri, July 12
Invite emergency reviewers: Fri, July 12 – Wed, July 17
Second round reviewing: Wed, July 17 - Fri, July 26
Reviews released to authors: Tues, July 30
Author Rebuttal: Tues, July 30 - Tues, Aug 6
Reviewer-Author Discussions: Tue, Aug 6 – Tue, Aug 13
Reviewer-AC Discussions: Tue, Aug 13  - Fri, Aug 23
Metareviews due:  Fri, Aug 30
SAC-AC Discussions: Fri, Aug 30 - Mon, Sep 9
SAC-PC Discussions: Tues, Sep 10 – Thu, Sep 19
Author Notifications: Wed, Sept 25

Main Tasks

  1. Preparation:
    • Please ensure that your preferred email address is accurate in your OpenReview profile. We will send most emails from OpenReview (noreply@openreview.net). Such emails are sometimes accidentally marked as spam (and sometimes as updates in Gmail). Please check your spam folder regularly. If you find such an email in there, please whitelist the OpenReview email address so that you will receive future emails from OpenReview.
    • Please log into OpenReview and make sure that your profile is up to date.
    • Read and agree to abide by the NeurIPS code of conduct.
    • Read what constitutes conflict of interest for NeurIPS 2024.
    • Read the guidelines of the paper checklist for 2024 authors.
    • In addition to the guidelines below, please familiarize yourself with the reviewer responsibilities and guidelines. You will be interacting significantly with reviewers, so please make sure you understand what is expected of them. You may also refer to the SAC guidelines.
  2. Paper assignments: Wed, June 5
    • This year, we do not have AC bidding. Instead we will use the matching algorithm and assign the papers automatically. While it is possible that this may result in less than optimal assignments, we rely on your experience and insight in handling papers from a variety of topics rather than on your specific topic expertise. Please bear in mind that we have an extra large number of submissions this year, so your flexibility is greatly appreciated. On or after June 10, please log into OpenReview, check your assignments.
  3. Modify reviewer assignment and add reviewers if necessary: Wed, June 5 – Sun, June 10
    • Make sure that every submission in your batch is matched with suitable reviewers whom you can trust on this submission, ideally with a diverse set of opinions.
    • It really pays off to invest some time before the reviewing process starts to ensure that your batch has expert reviewers!
    • You can continue inviting reviewers as needed after the assignment is released to reviewers on June 11.  Note that because the review process would already be underway, reviewers will need to agree to review each additional paper before they can be assigned.  You can follow these instructions for sending invitations to review a specific paper.
  4. Ensure all papers have quality reviews: Mon, June 10 – Fri, July 12
    • Reviews are due Friday, July 12. You might have to send multiple reminder emails, but please be courteous. If a reviewer is unable to deliver a review, please find a replacement reviewer.
    • Read all reviews carefully. If a review is substandard, you should ask the reviewer to improve their review. Please remember to be polite and provide concrete guidance.
    • You may also need to get an additional review if the current reviews are particularly unhelpful, either in the existing pool or by inviting external reviewers.
    • Make sure that any questionable papers are flagged for ethics review. These papers will be assigned ethics reviewers, who will effectively join the paper's assigned program committee. Their reviews will be released to the authors on August 20, and authors will have an opportunity to respond to these reviews.
  5. Discuss with reviewers and authors: Tuesday, August 6 – Friday, August 23 
    • As soon as the author response is entered in the system, initiate and lead a discussion via OpenReview for each submission, and make sure the reviewers engage in the discussion phase.
    • Monitor and moderate the discussion to ensure that it is respectful of everyone’s opinion. Read the submissions in your stack to steer the discussion towards the most critical aspects that need discussion.
    • Make sure your reviewers read and respond to all author responses. To minimize the chance of misunderstandings during the reviewing process, you can also initiate a rolling discussion with the authors after initial reviews and author responses are submitted. You can restrict visibility of your comment to any set of readers (authors, reviewers, senior area chair, or program chairs) as appropriate.
    • This year the discussion period is separated into the following three phases.
    • Author-Reviewer Discussions (Aug 6 - Aug 13): During this first phase, reviewers are asked to read all other reviews and the author responses and engage in an open exchange with the authors. Please ping the reviewers, as this is a one-week period.
    • Reviewer-AC Discussions (Aug 13 - Aug 23): During this second phase, please actively discuss the paper, the reviews, and the author responses with the reviewers.
  6. Writing the metareviews: by Friday, Aug 30
    • This is considered the last phase of the discussions, and you will be writing the metareviews. You can elicit further comments and clarifications from the reviewers. Please reach out to your SAC if you need help. Please help us by being on time with your metareviews, as we have only a couple of weeks before the notifications are sent to the authors on Sept 25. 
    • Write a meta-review that explains your decision (accept, reject, borderline) to the authors. Your comments should augment the reviews, and explain how the reviews, author response, and discussion were used to arrive at your decision. Do not dismiss or ignore a review unless you have a good reason for doing so. If the reviewers cannot come to a consensus, you should read the paper carefully and write a detailed meta-review. You are expected to discuss such difficult cases with your SAC.
  7. Make initial accept/reject decisions with SACs: Friday, Aug 30 – Monday, September 9
    • Discuss with your SAC your accept/reject recommendations, especially the borderline cases as well as ones where you are making a recommendation that goes against the reviewers. The SAC will help you calibrate across other ACs.
    • This year we have an optional rating of the reviews. This will happen after the notifications, and we will contact you separately to opt-in. Please participate in this extra task to help improve the NeurIPS peer reviewing process.
  8. Notification: Wednesday, September 25.

Best Practices

  • Please respect deadlines and respond to emails as promptly as possible.
  • It is okay to be unavailable for part of the review process (e.g., on vacation for a few days), but if you will be unavailable for more than that—especially during important windows (e.g., discussion, decision-making)—you must let your SAC know as soon as you can.
  • If you notice a conflict of interest with a submission that is assigned to you, please contact your SAC immediately so that the paper can be reassigned.
  • Be professional and listen to the reviewers, but do not give in to undue influence. We expect you to be familiar with all the papers that are assigned to you and to be able to argue about their technical content and contributions. Your responsibility is to make good decisions, not just facilitate reviewer discussions.
  • Be kind. It is important to acknowledge that personal situations, in particular during this year of a global pandemic, may lead to late or unfinished work among reviewers. In the event that a reviewer is unable to complete their work on time, we encourage you to be considerate of the personal circumstances; you might have to pick up the slack in some cases. In all communications, exhibit empathy and understanding.
  • DO NOT talk to other ACs about submissions that are assigned to you; other ACs may have conflicts with these submissions. If you feel that it’s important to discuss one submission in the context of another, please email program-chairs@neurips.cc. In general, your primary point of contact for any discussions should be your SAC. Your SAC does not have any conflicts with any of the submissions that are assigned to you.
  • DO NOT talk to other SACs or ACs about your own submissions (i.e., submissions you are an author on) or submissions with which you have a conflict of interest.
  • If you notice unethical or suspect behavior involving either authors or reviewers, please notify your SAC and program chairs.

Writing the meta-review

These guidelines are due to Chris Williams and John Lafferty and lightly adapted.

  • Don't focus too much on the scores. Instead, look carefully at the comments. Judge the quality of the review rather than taking note of the reviewer's confidence score; the latter may be more a measure of personality.
  • Indicate that you have read the author response, even if you just say "the rebuttal did not overcome the reviewer's objections."
  • If you use information that is not in the reviews (e.g., from corresponding with one of the reviewers after the rebuttal period), tell the authors (a) that you have done so and (b) what that information is.
  • If you find yourself wanting to overrule a unanimous opinion of the referees, the standards for your summary should be at the level of a full review. In these cases, it would probably be best to solicit an auxiliary review.
  • Please attempt to take a decisive stand on borderline papers. Other than papers where there is a genuine disagreement, much of our work will involve borderline papers where no one confidently expresses excitement, nor are any major problems identified. These are the tough decisions where we need your judgment!
  • Try to counter biases you perceive in the reviews. Unfashionable subjects should be treated fairly but often aren't, to the advantage of the papers on more mainstream approaches. To help the NeurIPS community move faster out of local minima, it is important to encourage risk and recognize that new approaches can't initially yield state-of-the-art competitive results. Nor are they always sold according to the recipes we are used to.

Confidentiality

You must keep everything relating to the review process confidential. Do not use ideas, code and results from submissions in your own work until they become publicly available (e.g., via a technical report or a published paper for ideas/results, via open source for code, etc.). Do not talk about or distribute submissions (whether it is the code, or the ideas and results described in them) to anyone without prior approval from the program chairs. Code submitted for reviewing cannot be distributed.