ARA: Agentic Reproducibility Assessment For Scalable Support Of Scientific Peer-Review

Scientific peer review increasingly struggles to assess reproducibility at the scale and complexity of modern research output. Evaluating reproducibility requires reconstructing experimental dependencies, methodological choices, data flows, and result-generating procedures, which often exceeds what human reviewers can provide given the growing number of submissions and time constraints. Agentic Reproducibility Assessment (ARA) formalizes reproducibility assessment as a structured reasoning task over scientific documents. Given a paper, ARA extracts a directed workflow graph linking sources, methods, experiments, and outputs, then evaluates its reconstructability using structural and content-based scores for reproducibility assessments. Experiments on 213 ReScience C articles – the largest cross-domain benchmark of human-validated computational reproducibility studies considered to date – demonstrate ARA’s generalizability and consistent workflow reconstruction and assessment across LLMs, model temperatures, and scientific domains. ARA consistently achieves 61% accuracy on three reproducibility assessment benchmarks, highlighting its potential to complement human review at scale and enabling next-generation peer review.