How Effective is Your Rebuttal? Identifying Causal Models from the OpenReview System

The peer review process is central to scientific publishing, with the rebuttal phase offering authors a critical opportunity to address reviewers’ concerns. Yet the causal mechanisms underlying rebuttal effectiveness, particularly how author responses influence final review decisions, remain unclear. To uncover the mechanisms driving rebuttal effectiveness, we model it as a causal representation learning (CRL) problem. Using data from the OpenReview system for ICLR submissions, we examine how rebuttal characteristics of both reviewers and authors causally affect post-rebuttal rating changes. We introduce a weakly supervised disentangled CRL framework that leverages review subscores (e.g., openness, clarity, directness) as concept-level supervision. Theoretically, we establish identifiability conditions for latent variables across multiple distributions, showing that human-interpretable concepts can be recovered under mild assumptions. Empirically, our results uncover distinct causal patterns governing successful rebuttals, revealing how specific strategies differentially influence review criteria. These findings provide actionable guidance for authors in crafting effective rebuttals, while offering broader implications for transparency, fairness, and efficiency in the peer review process.