We Need to Improve Benchmarks in AI4Math
Abstract
Benchmarks used to evaluate AI systems for mathematics, in both natural and formal languages, exhibit critical shortcomings that limit community progress toward powerful, open, community-driven mathematical copilots. These limitations range from an excessive focus on theorem proving and autoformalization at the expense of other important mathematical activities, such as conjecture generation, to barriers to evaluating models on privately held research-level benchmarks. Compounding these issues is a dynamic reminiscent of Goodhart's law: as performance on narrow benchmarks becomes a primary optimization target, benchmark scores may become less reliable indicators of broader mathematical capability. We highlight the main limitations and argue that addressing them requires a course correction in mathematical benchmark design. We conclude with a series of recommendations for improving benchmarking standards and moving toward more holistic assessments of the capabilities needed by mathematical copilots.