The ability to measure AI is being quickly outpaced by the ability to develop it. Agents have moved from simply answering questions, to doing real work, to now solving truly complex problems. In this, benchmarks matter more than ever and the need for trustworthy, reliable, and lasting ones is continuously growing. Three predominant failure modes exist as major threats, including benchmaxxing (models tuned to the test), benchslop (poor task design, including hidden requirements, contradictory instructions, and overly strict tests), and reward hacking (capable models gaming the grader).

This talk highlights that a great agent benchmark works to develop a research agenda (the art) and is built as a trustworthy measurement instrument (the science). The art of the work contributes to the discussion at the frontier, acts as a roadmap others can extend, and presents a design that is simple to run and one to which others can contribute. The science is oriented around task quality, distributional control, real headroom, and scoring that goes beyond accuracy.

This discussion employs Senior SWE-Bench as the case study. Built with researchers at Princeton and UW-Madison, it works to evaluate agents in comparison to senior engineers, uses purposefully under-specified instructions, contains longer-horizon tasks, and has more rigorous code-quality evaluations. The talk will highlight Senior SWE-Bench’s quality-control processes, from multi-run reliability checks to expert-calibrated judges, and what it takes to make long-horizon QC trustworthy.

The talk will close with three axes for agentic evaluation (e.g. environment complexity, autonomy horizon, output complexity) and lessons from the Open Benchmarks Grants program. Attendees leave with a practical checklist for designing agent benchmarks that are difficult, fair, and useful.