Predictive AI Evaluation Competition

As AI systems are deployed across an ever-wider range of consequential settings and the benchmarks used to evaluate them have proliferated accordingly, exhaustive evaluation has become infeasible---yet the field still treats benchmark scores as descriptive endpoints rather than as data for predicting behavior in unseen settings. The Predictive AI Evaluation Competition reframes the evaluation of large language models and agentic systems as a predictive-modeling problem: given evaluation results on a sample of benchmark questions for a set of AI systems, participants predict how those same systems would perform on other questions they have not been evaluated on. Predictions must generalize to questions and benchmarks the participant has never observed during training, mirroring the deployment pattern in which a frontier model is asked about proprietary datasets, newly authored questions, and entirely new benchmarks. To reflect realistic deployments, an optional adaptive-labeling channel lets submissions request a small budget of ground-truth labels per round, embedding active learning into the protocol. We curate response data on 100 popular and diverse AI benchmarks, covering more than 1 million questions across hundreds of AI systems; a small public subset is released for training, while the remaining benchmarks are held out for evaluation. Submissions are scored on negative log-loss against the confidential held-out labels. The competition is hosted at https://aimslab.stanford.edu/competition.