Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions

Selective AI conferences have recently begun enforcing ethics review processes, with the goal of steering research towards safer and more responsible practices before publication. But do ethics flags actually steer research as intended? In this paper, we show that authors often revise the framing of their project following ethics flags rather than redirecting their underlying research agendas. We first study the longitudinal effects of ethics flags, tracing flagged ICLR (International Conference on Learning Representations) submissions that were rejected or withdrawn into subsequent public resubmissions and tracking manuscript changes made after the review process ends when the original reviewers no longer oversee the project. We qualitatively classify these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and interview authors about their rebuttal processes and resubmission decisions. We find that authors often make concessions during rebuttal while reviewers can still update their scores, but drop those concessions after rejection, when the original reviewers no longer have leverage. Interview participants describe publication changes as separate from research direction changes, calling review an “editorial process” that shapes “what stories get seen” and, in another case, saying peer reviews are “mostly to filter out papers.” Since authors more readily change what they publish than what they study or build, we recommend several policy changes, especially disclosure of prior ethics flags upon resubmission so accountability for unresolved concerns carries over.