Skip to yearly menu bar Skip to main content


Turing

Expo Talk Panel

CEO Bench: Evaluating frontier agents on real company work

Grand BR B1
[ ]
Sun 6 Dec 11 a.m. AEDT — noon AEDT

Abstract:

Most benchmarks ask whether an agent can complete a well-defined task. Real company work requires agents to reconcile conflicting data, follow governance constraints, and produce decisions that hold up under scrutiny.

CEO Bench is the first benchmark built on a complete, expert-authored simulated company. Its setting, Cirrus Sleep, Inc., is a fictional direct-to-consumer sleep brand with a full fiscal year and 1,100+ files spanning accounting, finance, contracts, HR and payroll, board materials, operations, marketing, customer support, and a year of internal Slack. On top of this corpus, practitioners authored a hierarchy of tasks: executive-level engagements representing ~100 hours of professional work, and department-level tasks of 10–20 hours derived from the same events — so the benchmark measures the same underlying business reality at every altitude of the org chart, from analyst deliverable to board-ready decision.

Agents must determine which conflicting numbers are authoritative, identify what the board approved, reconcile evidence across departments, decide what to disclose, and assemble an executive-ready package. Work is graded against expert-written rubrics with must-pass integrity gates rather than a single string-matched answer. Across six frontier models on the 40 publicly scored tasks, the top score is roughly 41/100 — and every model failed at least one integrity trap a first-year analyst would catch. The talk shows why evaluating agents on real company work requires measuring more than task completion: it must assess whether the long-horizon financial and operational work companies depend on actually holds up.

Chat is not available.