Skip to yearly menu bar Skip to main content


Poster

VIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice Cloning

Xudong Tang ⋅ Matthew Groh
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A807
Speaker identification benchmarks evaluate how accurately a model can predict a ground-truth speaker label for a given utterance, treating voice identity as a property of the recording's producer. However, these benchmarks do not test whether a model aligns with human perception of voice identity. Whether a human would hear two voices and interpret them as the same speaker or not has consequences for intellectual property rights, cybersecurity, and everyday life. We make two contributions. First, we introduce the Voice Identity Perception benchmark (VIPBench), built from the identity judgments of English-speaking crowdworkers on stratified celebrity voice pairs. VIPBench consists of 124,876 same/different identity judgments from 1,290 participants on 9,800 voice pairs across 100 speakers, spanning real recordings, AI voice clones generated by a state-of-the-art Text-to-Speech (TTS) system, and continuously morphed voices. Our second contribution is to define four evaluation tasks: predicting the listener same-speaker agreement rate; binary same/different classification against the majority listener vote, evaluated for both ranking and calibration; alignment between the model's and listeners' speaker-similarity structures; and whether a predictor fit on real speech still works on voice clones and morphs. We report baselines for ten publicly available speech representations. We find that the perception target re-orders model rankings relative to metadata: supervised embeddings (trained on metadata speaker labels) still outperform self-supervised models (which learn voice structure without identity supervision), and no model reaches the noise ceiling. VIPBench enables systematic evaluation of speaker representations against human voice-identity perception across real, cloned, and morphed speech, motivating speaker models trained on perceptual judgments directly.
Show more
View full details
Poster

L2-Bench: An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education

James Edgell ⋅ Wm. Matthew Kennedy ⋅ Ben Knight ⋅ Danielle Carvalho ⋅ Martin Ku ⋅ Isaac J Pattis
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A923
Despite rapid AI adoption in education, rigorous evaluation of AI-powered educational (AIED) systems remains critically underdeveloped, particularly in second-language (L2) education, one of the most common yet least evaluated AI applications. We introduce L2-Bench, an open-source benchmark of 1,000+ task-response pairs to aid the pedagogy-led evaluation of LLM capabilities relating to language learning and assessment. Crucially, L2-Bench measures model performativity on the application of learning experience design principles rather than mere knowledge of those principles or broad learning outcomes. Our contributions include: (1) a validated taxonomy of 12 competencies and 31 subcompetencies validated by 200+ expert practitioners (task authenticity = 4.42/5.00, criteria adequacy = 4.18/5.00); (2) a rubric-based evaluation methodology that we believe can, if adapted, generalize to similar (open-ended, qualitative) disciplines; (3) an evaluation dataset that produces reliable signal about model strengths, weaknesses, and contextual robustness across diverse L2 education scenarios. We find that, among large models, Claude Opus 4.7 performs best overall (85.5\%), though is marginally outperformed on several constituent tasks. We also find that performance drops notably on harder tasks $\in$ (69.9\%–73.4\%). L2-Bench provides education stakeholders better methods to make more informed decisions about real-world AIED adoption, use, and governance, while advancing the maturing science of AI evaluations for education.
Show more
View full details
Poster

Evaluating Whether LLMs Can Reliably Connect the DOTs?

Eftekhar Hossain ⋅ John Salvador ⋅ Santu Karmaker
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A806
Access to real-world information is often noisy and fragmented. Constructing a coherent narrative from such fragments requires models to reconstruct missing spans within a broader storyline, commonly referred to as text infilling, while preserving consistency with both the local context and the global storyline. Despite using text infilling as a pre-training objective in many Large Language Models (LLMs), their actual performance on real-world narrative infilling remains underexplored. In this paper, we address this gap by introducing a multi-domain benchmark of ∼9.2K instances for narrative infilling, constructed by masking one to three sentences across four narrative types: encyclopedic text, commonsense stories, news articles, and visual narratives. Using this benchmark, we evaluate 20 instruction-tuned open-source LLMs ranging from 1.5B to 70B parameters across varying levels of instruction specificity and reasoning guidance. Outputs are assessed using standard automatic metrics and a qualitative framework covering five narrative dimensions. Results show that model scale does not reliably predict infilling quality: Gemma-2-2B achieves the highest qualitative score (4.02/5), outperforming models over ten times larger, including DeepSeek-Qwen-32B (3.77/5, ↓6.6%) and LLaMA-3.3-70B (3.71/5, ↓8.3%). We further find that explicit reasoning offers limited benefits as chain-of-thought reasoning yields only a marginal improvement (+0.6%). Additionally, short narratives and domain characteristics emerge as stronger predictors of task difficulty than infill position alone for narrative infilling in current LLMs.
Show more
View full details
Poster

Multi-site PPG: An In-the-Wild Physiological Dataset from Emerging Multi-Site Wearables

Jiayi Shao ⋅ Jiaying Ye ⋅ ShengYao Liu ⋅ Zachary Englhardt ⋅ Girish Narayanswamy ⋅ Vikram Iyer ⋅ Qiuyue (Shirley) Xue
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A707
Wearables are widely used for mobile health monitoring, and photoplethysmography (PPG) is a key sensing modality for heart rate and related physiological measurements. However, public in-the-wild PPG datasets remain largely wrist-centric or limited to short, controlled studies, constraining research on emerging wearable form factors. We present Multi-site PPG, an in-the-wild physiological dataset collected from four custom-developed unobtrusive wearables: a smart earring, ring, watch, and necklace. Each device records green and infrared reflective PPG, 3-axis acceleration, and temperature with timestamps for cross-device alignment, while a Polar H10 chest strap provides reference electrocardiogram (ECG). Participants wore the devices for one to multiple days during daytime activities while continuing their normal routines. The dataset contains over 350 hours of raw data and 230–290 hours of preprocessed, modeling-ready 8-second windows data per wearable. We benchmark heuristic, supervised, and self-supervised heart-rate estimation methods, showing substantial body-site differences: the best methods achieve mean absolute errors (MAEs) of 2.30 bpm on the earring, 5.13 bpm on the ring, 8.37 bpm on the watch, and 8.68 bpm on the necklace. We further analyze motion effects and evaluate multi-site and PPG–accelerometer fusion, demonstrating the dataset’s value for robust physiological sensing across emerging wearable form factors.
Show more
View full details
Poster

C3VD-DEFCOL: A Deformable Colonoscopy Dataset with Time-Resolved 3D Ground Truth and Realistic Appearance

Ethan Luk ⋅ Mayank Golhar ⋅ Anthony Song ⋅ Raúl Iranzo ⋅ Victor M. Batlle ⋅ Lalithkumar Seenivasan ⋅ JMM Montiel ⋅ Nicholas Durr
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A622
3D reconstruction could improve colonoscopy by estimating mucosal coverage and alerting clinicians to missed regions during screening. However, algorithm development is limited as no current datasets provide both a realistic in vivo appearance and dense, time-resolved 3D ground truth, especially under non-rigid deformation. We present C3VD-DEFCOL, a framework and dataset for evaluating deformable colonoscopy reconstruction with paired geometry and realistic texture. Starting from C3VD/C3VDv2 colon meshes and camera trajectories, we generate controlled deformations of the colon surface, including peristaltic waves and centerline motion, and render fisheye colonoscopy videos with frame-level depth, surface normals, optical flow, occlusion masks, camera poses, and time-stamped 3D meshes. We then use the rendered geometry, primarily depth, to condition an LTX-2.3-based sim-to-real translation model that produces RGB clips with in vivo-like mucosal color, texture, vasculature, and specular appearance while preserving the underlying 3D scene structure. The resulting dataset contains 110 videos from 12 unique colon mesh geometries, with varying camera trajectories, appearances, and multiple deformation regimes, including three peristaltic severity levels. We evaluate the generated videos using appearance realism, geometric consistency, and temporal consistency metrics, and use the paired ground truth to benchmark the downstream task of pose estimation in deformable 3D reconstruction. Our experiments show how pose estimation error increases with increasing deformation severity, providing a controlled stress test that is not possible with existing in vivo datasets. Overall, C3VD-DEFCOL is designed as a reproducible, quantitative evaluation platform for testing deformable 3D reconstruction algorithms, with the goal of reducing the domain gap between synthetic datasets and in vivo colonoscopy.
Show more
View full details
Poster

Cross-Cell-Line Perturbation Prediction Needs Controls

Xingyu Fan ⋅ Jinghao Wang ⋅ kim hsieh ⋅ Chunbin Gu ⋅ Pheng-Ann Heng
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A706
Predicting genetic perturbation responses across cell lines is challenging not only for models, but also for the metrics used to evaluate them. We introduce a strict cross-cell-line CRISPRi stress test on Replogle Perturb-seq, training on K562 and RPE1 (or additionally HepG2) and evaluating on held-out Jurkat. We pair this setting with two necessary controls: target-control-copy, which predicts no perturbation effect, and shifted-control baselines, which translate target-line controls by a source-derived mean shift. Across classical baselines, recent deep models, CellFlow, and context-conditioned flow-matching variants with five frozen single-cell foundation-model encoders plus a PCA/null context, we find that aggregate distributional metrics can be strongly driven by target-line cell-state variability rather than perturbation-specific response. Target-control-copy is competitive on aggregate distributional metrics, while shifted-control approaches CellFlow on Energy and substantially outperforms it on perturbation-specific Pearson. Used as fixed per-cell context for a shared downstream flow head, none of five frozen scFM encoders matches a PCA/null encoder in this strict cross-line setting. Finally, anchored mean-preserving residual-flow probes reveal a metric-specific trade-off: full-space residual flows improve $W_2$ but degrade per-gene moment matching, whereas low-rank constrained flows preserve moments but lose most $W_2$ gain. No tested method simultaneously dominates perturbation-specific response and biologically relevant distributional fidelity. These results argue for perturbation-aware cell-level evaluation in cross-cell-line virtual-cell benchmarks.
Show more
View full details
Poster

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

Fangrui Huang ⋅ Souhad Chbeir ⋅ Arpandeep Khatua ⋅ Sheng Wang ⋅ Sijun Tan ⋅ Kenan Ye ⋅ Lillian A Bailey ⋅ Merryn Daniel ⋅ Ryan Louie ⋅ Sanmi Koyejo ⋅ Ehsan Adeli
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A1020
Large language models (LLMs) are increasingly used for mental-health support; yet prevailing evaluation methods--fluency metrics, preference tests, and generic dialogue benchmarks--fail to capture the clinically critical dimensions of psychotherapy. We introduce THERAPYGYM, a framework that evaluates and improves therapy chatbots along two clinical pillars: fidelity and safety. Fidelity is measured using the Cognitive Therapy Rating Scale (CTRS), implemented as an automated pipeline that scores adherence to CBT techniques over multi-turn sessions. Safety is assessed using a multi-label annotation scheme, covering therapy-specific risks (e.g., failing to address harm or abuse). To mitigate bias and unreliability in LLM-based judges, we further release THERAPYJUDGEBENCH, a validation set of 116 dialogues with 1,270 expert ratings for auditing and calibration against licensed clinicians. THERAPYGYM also serves as a training harness: CTRS and safety-based rewards drive RL with configurable patient simulations spanning diverse symptom profiles. Models trained in THERAPYGYM improve on expert ratings, with average CTRS rising from 0.10 to 0.60 (and 0.16 to 0.59 under LLM judges). More broadly, THERAPYGYM provides foundational targets for responsible psychotherapy-oriented LLMs: clinical fidelity, therapy-specific safety, expert-calibrated evaluation, and multi-turn therapeutic process.
Show more
View full details
Poster

TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

Jiarui Yan ⋅ Weiwei Sun ⋅ Sijie Li ⋅ Wenhan Li ⋅ Yiming Yang
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A124
While LLMs excel at isolated coding tasks, their ability to perform autonomous, long-horizon machine learning (ML) development remains bottlenecked by poor strategic planning. Because current benchmarks rely on outcome-based metrics, they obscure process-level failures, making it impossible to diagnose where agent strategies diverge from human expertise. To solve this, we introduce TraceML, the first large-scale dataset of paired human and agent trajectories for long-horizon ML tasks. TraceML contains 4{,}672 Kaggle trajectories across 134 MLE-bench competitions, with 430 expert humans paired head-to-head with 207 LLM-agent runs on 7 of those competitions. Using a unified extraction schema to analyze step-by-step behavior, we reveal systematic gaps: agents execute local edits well but exhibit unstructured exploration, poor experiment prioritization, and weak long-term memory attribution compared to experts. Translating these diagnostic insights into action, we design a novel, planning-oriented agent harness based on human priors that significantly improves both process efficiency and final competitive performance.
Show more
View full details
Poster

Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency

Zhikai Wang ⋅ Jiashuo Sun ⋅ Wenqi Zhang ⋅ Zhiqiang Hu ⋅ Xin Li ⋅ Deli Zhao
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A1021
Recent advancements in Large Vision-Language Models (LVLMs) have significantly enhanced their ability to integrate visual and linguistic information, achieving near-human proficiency in tasks like object recognition, captioning, and visual question answering. However, current benchmarks typically focus on knowledge-centric evaluations that assess domain-specific expertise, often neglecting the core ability to reason about fundamental mathematical elements and visual concepts. We identify a gap in evaluating elementary-level math problems, which rely on explicit visual dependencies-requiring models to discern, integrate, and reason across multiple images while incorporating commonsense knowledge, all o which are crucial for advancing toward broader AGI capabilities. To address this gap, we introduce VCBench, a comprehensive benchmark for multimodal mathematical reasoning with explicit visual dependencies. VCBench includes 1,720 problems across six cognitive domains, featuring 6,697 images (averaging 3.9 per question) to ensure multi-image reasoning. We evaluate 26 state-of-the-art LVLMs on VCBench, revealing substantial performance disparities, with even the top models unable to exceed 50% accuracy. Our findings highlight the ongoing challenges in visual-mathematical integration and suggest avenues for future LVLM advancements.
Show more
View full details
Poster

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

Zhun Wang ⋅ Nico Schiller ⋅ Hongwei Li ⋅ Srijiith S Narayana ⋅ Milad Nasr ⋅ Nicholas Carlini ⋅ Xiangyu Qi ⋅ Eric Wallace ⋅ Elie Bursztein ⋅ Luca Invernizzi ⋅ Kurt Thomas ⋅ Yan Shoshitaishvili ⋅ Wenbo Guo ⋅ Jingxuan He ⋅ Thorsten Holz ⋅ Dawn Song
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A1019
AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is exploitation: turning a vulnerability, which is not yet an attack, into a concrete security impact, such as unauthorized file access or code execution. Exploitation is a particularly challenging task because it requires low-level program reasoning (e.g., about memory layout), runtime adaptation, and sustained progress over long horizons. Meanwhile, it is inherently dual-use, supporting defensive workflows while lowering the barrier for offense. Despite its importance and diagnostic value, exploitation remains under-evaluated. To address this gap, we introduce ExploitGym, a large-scale, diverse, realistic benchmark on the exploitation capabilities of AI agents. Given a program input that triggers a vulnerability, ExploitGym tasks agents with progressively extending it into a working exploit. The benchmark comprises 898 instances sourced from real-world vulnerabilities across three domains, including userspace programs, Google's V8 JavaScript engine, and the Linux kernel. We vary the security protections applied to each instance, isolating their impact on agent performance. All configurations are packaged in reproducible containerized environments. Our evaluation shows that while exploitation remains challenging, frontier models can successfully exploit a non-trivial fraction of vulnerabilities. For example, Claude Mythos Preview, Anthropic's latest model, produces working exploits for 208 instances. Notably, even with widely used defenses enabled, models retain non-trivial success rates. These results establish ExploitGym as an effective testbed for exploitation and highlight the growing cybersecurity risks posed by increasingly capable AI agents.
Show more
View full details
Poster

MITO: A Millimeter-Wave Dataset and Simulator for Non-Line-of-Sight Perception

Tara Boroushaki ⋅ Laura Dodds ⋅ Cusuh Ham ⋅ Fadel Adib
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A719
The ability to observe the world is fundamental to reasoning and making informed decisions on how to interact with the environment. However, optical perception can often be disrupted due to common occurrences like occlusions, which can pose challenges to existing vision systems. We present MITO, the first millimeter-wave (mmWave) dataset of diverse, everyday objects, collected using a UR5 robotic arm with two mmWave radars operating at different frequencies and an RGB-D camera. Unlike visible light, mmWave signals can penetrate common occlusions (e.g., cardboard boxes, fabric, plastic). MITO captures over 24 million mmWave frames and uses them to generate 550 high-resolution mmWave images in line-of-sight and non-light-of-sight (NLOS), as well as RGB-D images, segmentation masks, and raw mmWave signals, taken from 76 different objects. We develop an open-source simulation tool that can be used to generate synthetic mmWave images for any 3D triangle mesh. Finally, we establish benchmarks for NLOS segmentation and classification to demonstrate the utility of our dataset and simulator for enabling broader NLOS perception.
Show more
View full details
Poster

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

Yipeng Gao ⋅ Lei Shu ⋅ Genzhi Ye ⋅ Xi Xiong ⋅ Ameesh Makadia ⋅ Meiqi Guo ⋅ Laurent Itti ⋅ Jindong Chen
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A621
Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neural 3D generators inherently lack. Authoring such procedural content, however, demands deep expertise in 3D software APIs, parametric design, and code-level geometric reasoning. In this paper, we propose 3DCodeBench, a systematic benchmark for evaluating vision-language model (VLM) agents for procedural 3D generation in 3D modeling software. Specifically, 3DCodeBench evaluates how effectively 11 advanced VLMs can serve as procedural 3D modelers by transferring text and image references into procedural code for software. Recognizing that automated metrics may not fully capture the perceptual quality of 3D shapes, we build 3DCodeArena, a ranking platform based on human preference between generated pairs. From extensive evaluations and results, we observe that: (1) Failures mostly arise from API usage, while successful renders still suffer from disconnected or floating 3D geometric components. (2) Test-time scaling, such as higher thinking budgets and multi-turn refinement, improves performance overall. Our findings highlight a critical need for high-quality procedural coding data to advance commercial VLMs. Furthermore, effective procedural 3D modeling requires a robust execution environment that provides high-fidelity feedback for iterative refinement. To accelerate research in this domain, we release 3DCodeBench—including the curated data, benchmark, evaluation protocol, and qualitative results—as a foundational tool for developing VLM-based 3D modelers.
Show more
View full details
Poster

Mission Impossible: Diagnosing and Fixing Non-Operative Instruction Following in Image Editing

Guoyizhe Wei ⋅ Feng Wang ⋅ Alan Yuille ⋅ Rama Chellappa
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A618
Instruction-guided image editing models are mostly trained and evaluated on executable requests, yet many real user instructions are non-operative for the given image. Reliable editing therefore requires reasoning not only about how to edit, but also whether to edit and what to ignore. We introduce \bench{}, a large-scale benchmark for non-operative image editing, with 1.1M instructions over 31K images across seven categories spanning symbolic, grounded, physical, and logical non-executability, plus a 35K-sample partial-edit test set evaluating selective execution when one clause is feasible and another is not. A 700-sample internal audit confirms a 94.9\% true no-op rate with $\kappa=0.87$. Evaluating 13 systems reveals a clear action--abstention dissociation: frontier editors achieve strong standard editing quality but still over-edit more than 40\% of no-op samples. Three complementary perceptual, feature-level, and vision-language metrics further surface distinct failure modes across architectures. As a reference baseline, JUDGE-THEN-EDIT BAGEL—a feasibility-aware BAGEL variant trained on a 150K mixture of full-edit, partial-edit, and no-op data—reduces over-edit rate from 41.7\% to 12.6\%. Our results show that abstention is a distinct capability that current benchmarks fail to measure.
Show more
View full details
Poster

Evaluating Deployable Inference-Time Error Prediction in Vision MoEs

Aristeidis Tsaris ⋅ JangHyeon Lee ⋅ Abhishek Potnis ⋅ Philipe Dias ⋅ Waqwoya Abebe ⋅ Dan Lu ⋅ Feiyi Wang ⋅ Dalton Lunga
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A724
Sparse mixture-of-experts (MoE) models raise a natural inference-time question: can experts not selected by the router improve predictions or estimate their errors without retraining? We study this question in vision MoEs across five RGB and multispectral classification datasets and multiple expert counts. We evaluate additional expert inference for three outcomes: top-1 accuracy, raw ranking of default router errors, and calibrated prediction of whether the router-selected classification is wrong. An upper-bound analysis shows that non-selected experts often contain useful class signal when the routed path fails. However, this signal does not reliably improve deployable top-1 accuracy, and raw multi-forward uncertainty scores yield only limited gains for ranking router errors. The strongest benefit appears in calibrated router-error prediction: post-hoc models that combine router signals with features from non-selected experts reduce Router-error Expected Calibration Error (ECE) by roughly 10-13% on average, with several settings exceeding 20% relative improvement over a calibrated router-only baseline. Overall, additional inference-time expert usage in vanilla vision MoEs is most useful for estimating the reliability of the routed decision, rather than directly improving classification accuracy.
Show more
View full details
Poster

HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes

Bowen Cui ⋅ Junyu Yin ⋅ Tejas Ramesh ⋅ Leo Lim ⋅ Oscar Hernandez ⋅ Keren Zhou
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A408
Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning, motivating extensive benchmarks for evaluating functional correctness. However, performance optimization in High-Performance Computing (HPC) poses fundamentally different challenges that are not captured by existing evaluations: HPC performance depends on low-level code, intricate memory access patterns, and computational kernels embedded within sophisticated execution contexts. We introduce HPC-Bench, a performance-grounded benchmark for evaluating LLMs as optimizers of computational kernels in realistic HPC workloads, including $114$ workloads across $16$ HPC computational motifs. HPC-Bench provides a fully automated, end-to-end evaluation pipeline that combines program-level correctness verification with two correctness-gated metrics, $\mathrm{Fast}@k$ and $\mathrm{Speedup}@k$, to characterize reliability and speedup under a fixed sampling budget. Our evaluation of three representative LLMs across serial CPU, OpenMP, and CUDA settings shows that stricter speedup requirements substantially reduce the probability of finding a correct and fast optimization. HPC-Bench is available at https://github.com/neuripspaper2026/NeurIPS_26.
Show more
View full details
Poster

DD-Ranking: Rethinking the Evaluation of Dataset Distillation

Zekai Li ⋅ Xinhao Zhong ⋅ Samir Khaki ⋅ Zhiyuan Liang ⋅ Yuhao Zhou ⋅ Mingjia Shi ⋅ Ziqiao Wang ⋅ Xuanlei Zhao ⋅ Wangbo Zhao ⋅ Ziheng Qin ⋅ Mengxuan Wu ⋅ Pengfei Zhou ⋅ Haonan Wang ⋅ David Junhao Zhang ⋅ Jia-Wei Liu ⋅ Shaobo Wang ⋅ Dai Liu ⋅ Linfeng Zhang ⋅ Guang Li ⋅ Kun Wang ⋅ Zheng Zhu ⋅ Zhiheng Ma ⋅ Joey Tianyi Zhou ⋅ Jiancheng Lv ⋅ Yaochu Jin ⋅ Peihao Wang ⋅ Kaipeng Zhang ⋅ Yiran Huang ⋅ Zhiwei Deng ⋅ Xindi Wu ⋅ George Cazenavette ⋅ Yuzhang Shang ⋅ Justin CUI ⋅ Jindong Gu ⋅ Qian Zheng ⋅ Hao Ye ⋅ Shuo Wang ⋅ Xiaobo Wang ⋅ Yan Yan ⋅ Angela Yao ⋅ Mike Zheng Shou ⋅ Tianlong Chen ⋅ Hakan Bilen ⋅ Baharan Mirzasoleiman ⋅ Manolis Kellis ⋅ Konstantinos N Plataniotis ⋅ Zhangyang "Atlas" Wang ⋅ Bo Zhao ⋅ Yang You ⋅ Kai Wang
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A1018
Dataset distillation has provided a reliable solution for data compression, where models trained on the resulting smaller synthetic datasets achieve performance comparable to those trained on the original datasets. To further improve the performance of synthetic datasets, various training pipelines and optimization objectives have been proposed, greatly advancing the field of dataset distillation. Recent decoupled dataset distillation methods introduce soft labels and stronger data augmentation during the post-evaluation phase and scale dataset distillation up to larger datasets. However, this raises a question: Is accuracy still a reliable metric to fairly evaluate dataset distillation methods? Our empirical findings suggest that the performance improvements of these methods often stem from additional techniques rather than the inherent quality of the images themselves, with even randomly sampled images achieving superior results. Such misaligned evaluation settings severely hinder the development of DD. Therefore, we propose DD-Ranking, a unified evaluation framework, along with new general evaluation metrics to uncover the true performance improvements achieved by different methods. By refocusing on the actual information enhancement of distilled dataset, DD-Ranking provides a more comprehensive and fair evaluation standard for future research advancements.
Show more
View full details
Poster

PosteriorBench: From Point Estimates to Posterior Matching in Evaluating Generative Inverse Solvers

Jiachen Yao ⋅ Zi-Siang Hsu ⋅ Xi Deng ⋅ Aditi Gupta ⋅ Xin Ju ⋅ Animashree Anandkumar
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A204
Generative models are increasingly used to solve scientific inverse problems, but existing evaluations still focus primarily on whether a method can produce a single plausible reconstruction. This is insufficient for ill-posed problems, where multiple solutions may be consistent with the same sparse or noisy observations. In these settings, a method can achieve strong pointwise accuracy while still failing to capture the true posterior through mode collapse, over-confident uncertainty, or averaging incompatible solutions. We introduce PosteriorBench, a benchmark for evaluating the distributional accuracy of generative inverse solvers. PosteriorBench evaluates four physics-based inverse problems: Darcy flow inversion, Poisson source recovery, carbon capture and storage, and light transport material inference. For each task, we construct high-fidelity reference posteriors using slow but established procedures such as rejection sampling and Markov chain Monte Carlo, enabling direct assessment of whether solvers recover the full set of solutions rather than the single best sample. The benchmark spans sparse sensing, low-resolution observations, nonlinear forward models, varying noise levels, and multimodal priors, with a unified pipeline for distribution matching, uncertainty quantification, and hyperparameter calibration. Our experiments reveal substantial distribution-matching gaps across current solvers, while showing that neural operators improve resolution robustness and that guidance weights and generation noise are key to posterior-variance calibration.
Show more
View full details
Poster

LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression

Zishan Shao ⋅ Lixun Zhang ⋅ Kangning Cui ⋅ Wenhao Wu ⋅ Jinhee Kim ⋅ Yixiao Wang ⋅ Ting Jiang ⋅ Hancheng Ye ⋅ Qinsi Wang ⋅ Fan Yang ⋅ Danyang Zhuo ⋅ Hai Li ⋅ Yiran Chen
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A100
SVD-based low-rank compression has become a fast-growing direction for reducing the memory and computational cost of large language models (LLMs). However, fair comparisons across existing studies remain challenging due to historically inconsistent baseline results that continue to propagate and substantial differences in evaluation protocols. As a result, it remains unclear whether reported gains reflect method-level improvements or differences in evaluation protocol. This lack of comparability highlights the need for a unified, reproducible evaluation platform. To address this problem, we present $\textbf{LowRankArena}$, a standardized evaluation platform for SVD-based LLM compression. LowRankArena unifies cross-methods evaluation settings (e.g., task versions, compression ratio definition, inference setup), and provides a reproducible pipeline with over 3 TiB reproduced checkpoints. Using LowRankArena, our aligned audit of five representative SVD methods reveals that prior findings are highly conditional under standardized protocols: clear leaders and performance tiers shift across backbones and keep ratios, multiple-choice accuracy can hide large perplexity degradation, and nominal low-rank savings yield workload-dependent and often limited end-to-end speedups. Our code is available at: https://github.com/Zishan-Shao/lowrankarena.git.
Show more
View full details
Poster

EgoSurgHands: An Egocentric 3D Hand Pose Dataset & Benchmark for Open-Surgery Training

Ankit Pal ⋅ Guillaume Kugener ⋅ Sofia Volpi ⋅ Joshua Schwartz ⋅ Alex Idarraga ⋅ Pranav Rajpurkar ⋅ Gabriel A Brat
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A1017
We introduce EgoSurgHands, an egocentric 3D hand-pose dataset and benchmark for open-surgery training tasks captured with Meta Project Aria glasses on standard surgical training pads. To our knowledge, EgoSurgHands is the first public benchmark for surgical training that is simultaneously egocentric, sensor-grounded, and human-validated. EgoSurgHands contains 41,909 training hand-instance rows across 55 procedure recordings and 9,476 inter-annotator-agreement (IAA)-validated rows across 13 held-out procedure recordings from 4 wearers, 13 distinct tasks, 3 glove colors, and 3 stages of training. Each frame includes Aria intrinsics, SLAM extrinsics, MPS (Machine Perception Services) landmarks, per-joint confidence, timestamps, and eye gaze. Validation 3D keypoints are initialized from Aria MPS and independently corrected by four expert annotators under a multi-rater IAA workflow, yielding evaluation ground truth independent of any tested model's training signal. Using EgoSurgHands, we benchmark six state-of-the-art hand reconstructors under zero-shot transfer: HaMeR, WiLoR, WildHands, JointTransformer, SimpleHand, and MeshGraphormer. Ego-pretrained models consistently outperform non-egocentric baselines: WildHands improves over the non-ego HaMeR reference by 4.51 mm PA-MPJPE, while JointTransformer achieves the best zero-shot result at 14.95 mm PA-MPJPE. These results show that generalist egocentric pretraining transfers to surgical egocentric tasks, but also reveal a substantial remaining surgical-domain gap. As a lightweight diagnostic adaptation experiment, a 58K-parameter two-headed (pose + translation) residual corrector further reduces error on frozen backbones, reaching 10.40 mm PA-MPJPE and 75.8 mm absolute MPJPE on WildHands ($-$49.2\% and $-$85.8\% vs zero-shot). We release the dataset, benchmark code, checkpoints, and multi-rater annotation framework at [URL withheld for double-blind review].
Show more
View full details
Poster

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Mingxuan (Aldous) Li ⋅ Huanzhi Mao ⋅ Ruitao Zheng ⋅ Jarod Stanbury James ⋅ Yunze Li ⋅ Natanya Anderson ⋅ Joseph Gonzalez ⋅ Alex Dimakis
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A101
AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in Github repositories and Google Docs. It contains 140 queries spanning heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks are not reducible to static corpus question answering: agents must determine which conversations matter, traverse threads, inspect repository state, and synthesize evidence across communication and code artifacts. Across 20 frontier models and agent harnesses, the best systems score under 60\%, highlighting substantial headroom for improvement. SlackBench thus provides an extensible framework for evaluating agents in collaborative work environments.
Show more
View full details
Poster

A Large-Scale Multi-Source Dataset Linking Hacker Community Discourse to the CVE Vulnerability Lifecycle

Benjamin M Ampel ⋅ Sagar Samtani
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A120
We introduce HackerSignal, a benchmark for temporal out-of-distribution cyber threat intelligence (CTI) and cross-source CVE linkage. HackerSignal aggregates 7.45 million exact-deduplicated documents from 64 public forum/source identifiers spanning eight source layers and a 36-year window (1990-2026). In contrast to other publicly accessible cybersecurity datasets, HackerSignal is among the first public benchmark datasets that maps the full potential exploit to vulnerability trajectory from hacker community discourse, exploit databases with working and proof of concept exploits, vulnerability advisories, and software fix commits. HackerSignal creates these linkages through a shared CVE identifier space while preserving source-specific release modes to support a range of unique Artificial Intelligence (AI)-enabled cybersecurity analytics tasks. In this paper, we summarize HackerSignal and illustrate three selected benchmark tasks it uniquely supports: (1) CVE linkage retrieval (cross-source temporal out-of-distribution entity grounding); (2) exploit type classification (8-class vulnerability type prediction with temporal OOD evaluation); and (3) temporal generalization (prospective CVE-disjoint evaluation where $C_{\text{train}} \cap C_{\text{test}} = \emptyset$). All tasks use temporal splits to evaluate prospective generalization. We release source-shortcut and leakage diagnostics; manual-audit packets; a datasheet; and a release-governance addendum to support the dissemination of the dataset. HackerSignal's code, data, and Croissant metadata are available at hf.co/datasets/DatasetSubmission/HackerSignal.
Show more
View full details
Poster

MSConsensus: A Hundred-Million-Scale, Batch-Effect–Suppressed Dataset and Benchmark for Proteomics Machine Learning

Christopher Grams ⋅ Chang Liu ⋅ Michael Papka ⋅ Yu Gao
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A123
MSConsensus: A Hundred-Million-Scale, Batch-Effect–Suppressed Dataset and Benchmark for Proteomics Machine Learning Machine learning for proteomics is bottlenecked by three coupled and under-explored challenges: (i) noisy mass-spectrometry (MS) data dominated by instrument- and protocol-specific batch effects, (ii) slow data loading from legacy XML- and text-based file formats, and (iii) a fragmented evaluation landscape that prevents meaningful cross-study comparison. Each silently constrains what proteomics models learn, how efficiently they train, and how reliably they are compared. We address all three with MSConsensus, a multi-task dataset and benchmark built natively for proteomics ML. MSConsensus contains 110,209,043 consensus spectra distilled from 1.01 PB of public PRIDE data spanning 1,500 repositories and all major Orbitrap and timsTOF platforms, with per-peak ion annotations and stratified ML splits across instruments, species, and post-translational modifications. Unlike single-representative spectral libraries (MassIVE-KB v2, NIST, ProteomeTools), MSConsensus uses a new Weighted-Score Binning (WSBIN) algorithm that emits multiple consensus spectra per (peptide, charge) cluster, preserving intra-cluster variance required by contrastive and representation-learning objectives while suppressing context-specific artifacts. Across five held-out PRIDE repositories and four search engines (Comet, MetaMorpheus, MSFragger, Sage), WSBIN improves signal-to-noise ratio by an average of +3.86 dB (range +0.57 to +10.47 dB) without sacrificing protein, peptide, or PSM identification counts. To eliminate the I/O bottleneck during model training, we pair MSConsensus with MSCompress and a new msz binary container. msz compresses to 45% of mzML size losslessly and supports 25,378 random-access spectra/sec — 3.2× HDF5 — through a drop-in PyTorch / JAX dataloader. A unified public leaderboard standardizes evaluation across three core tasks—spectrum embedding, intensity prediction, and de novo peptide identification—and we report results for ten widely used systems (Casanovo, InstaNovo, Prosit, MS²PIP, GLEAMS, Spec2Vec, Comet, MetaMorpheus, MSFragger, Sage). Replacing raw training data with MSConsensus stabilizes Prosit's spectral angle on a never-before-seen 2025 instrument (PXD053296) from 0.6964 to 0.8137 and improves Spec2Vec top-1 retrieval from 39.2% to 46.7% (+19.0% relative). By jointly addressing scale, batch-effect suppression, I/O efficiency, and standardized evaluation, MSConsensus provides foundational infrastructure for training, evaluating, and comparing future proteomics models. The dataset is released on Harvard Dataverse (doi:10.7910/DVN/NUCT4N) under CC BY 4.0; MSCompress, MSDatasets, preprocessing pipelines, and baseline scripts are released on GitHub under Apache 2.0, with Docker containers for reproducibility.
Show more
View full details
Poster

$\chi$-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

Haolin Chen ⋅ Deon Metelski ⋅ Leon Qi ⋅ Tao Xia ⋅ Joonyul Lee ⋅ Steve Brown ⋅ Kevin Riley ⋅ Frank Wang ⋅ T. Y Liu ⋅ Hank Capps ⋅ Zeyu Tang ⋅ Xiangchen Song ⋅ Lingjing Kong ⋅ Fan Feng ⋅ Tianyi Zeng ⋅ Zhiwei Liu ⋅ Zixian Ma ⋅ Hang Jiang ⋅ Fangli Geng ⋅ Yuan Yuan ⋅ Chenyu You ⋅ Qingsong Wen ⋅ Hua Wei ⋅ Yanjie Fu ⋅ Yue Zhao ⋅ Carl Yang ⋅ Biwei Huang ⋅ Kun Zhang ⋅ Caiming Xiong ⋅ Sanmi Koyejo ⋅ Eric Xing ⋅ Philip S Yu ⋅ Weiran Yao
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A708
End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: *policy density*, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role *composition*: a single task requires the agent to play multiple roles with handoffs; and *multilateral interaction*: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $\chi$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document *managed-care operations handbook* skill. Across 30 agent harness/models configurations, the best agent resolves only **28.0%** of tasks, no agent clears **20%** on strict pass^3, and executing all tasks in a single session slumps the performance to **3.8%**. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.
Show more
View full details
Poster

Mini Amusement Parks (MAPs): A Testbed for Modelling Business Decisions

Stéphane Aroca-Ouellette ⋅ Ian Berlot-Attwell ⋅ Panagiotis Lymperopoulos ⋅ Abhiramon Rajasekharan ⋅ Tongqi Zhu ⋅ Herin Kang ⋅ Kaheer Suleman ⋅ Sam Pasupalak
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A102
Despite rapid progress in artificial intelligence, current systems struggle with the interconnected challenges that define real-world decision making. Practical domains such as business management require open-ended optimization, actively learning environment dynamics from sparse experience, planning over long horizons in stochastic settings, and reasoning over spatial information. Yet no existing human-AI benchmarks assess how well agents integrate these challenges in a grounded decision-making context. To this end, we introduce Mini Amusement Parks (MAPs), an amusement-park simulator designed to evaluate an agent’s ability to model its environment, anticipate long-term consequences under uncertainty, and strategically operate a complex business. We provide expert human performance and a comprehensive evaluation of state-of-the-art agents, finding experts outperform these systems by 11.4× on easy mode and 15.3× on medium mode. Our analysis reveals persistent weaknesses in long-horizon planning, sample-efficient learning, spatial reasoning, and modelling uncertainty. By unifying these challenges within a single environment, MAPs offers a new foundation for benchmarking agents capable of adaptable decision making.
Show more
View full details
Poster

s2n-bignum-bench: A practical benchmark for evaluating low-level code reasoning of LLMs

Balaji Rao ⋅ Soonho Kong ⋅ Juneyoung Lee ⋅ Carlo Lipizzi
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A604
Recent progress in neural theorem proving has been driven largely by mathematics-oriented and, more recently, by verification-condition and repository-scale benchmarks. These settings are valuable, but they do not directly test whether a model can synthesize machine-checkable proofs about concrete low-level implementations under an industrial proof-engineering stack. We address this gap with *s2n-bignum-bench*, a benchmark derived from AWS *s2n-bignum*, a formally verified library of hand-tuned big-integer assembly routines for ARM and x86. The current corpus packages $\mathbf{2{,}301}$ HOL Light proof obligations as standalone `setup.ml`/`query.txt` tasks spanning big-integer arithmetic, elliptic-curve routines, ML-KEM, SHA-3/Keccak, and shared ISA infrastructure. Each task reproduces the relevant proving environment, exposes the theorem statement to be proved, and expects a tactic expression that is accepted by HOL Light within a fixed timeout. The benchmark spans five categories so that one can separate generic HOL Light fluency from ISA-specific reasoning. We release the extraction pipeline, retrieval utilities, an offline evaluator with syntax/type pre-checking and integrity checks based on axiom-difference and forbidden-tactic detection, paired verbatim and obfuscated query variants, and a checkpointed assessment workflow to support iterative feedback-driven $\operatorname{pass}@K$ studies and future step-level proof construction workflows. Initial zero-shot baselines using GPT-5.3-based and other frontier models solve at most $\mathbf{6.35}$% of the problem set (when provided only with the goal term), highlighting the difficulty of HOL Light tactic synthesis for low-level verification obligations under trusted ISA semantics. The code to set up and use the benchmark is available at [s2n-bignum-bench](https://github.com/kings-crown/s2n-bignum-bench).
Show more
View full details
Poster

VIGOR: Benchmarking Visual Rationale Correctness in Multimodal Large Language Models

Yanlin Chen ⋅ Tang Li ⋅ Xi Peng
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A909
Multimodal large language models (MLLMs) are typically evaluated by whether they produce correct answers, but correctness alone does not reveal whether their outputs are supported by the right visual evidence. We introduce VIGOR, a benchmark for evaluating **VI**sually **G**r**O**unded **R**ationales in MLLMs. VIGOR represents visual rationales as explicit concepts paired with pixel-level groundings, covering multiple abstraction levels including low-level visual properties, intermediate structures, objects, parts and high-level motion, relations. To construct VIGOR at scale, we develop an automatic annotation framework that reuses existing dense supervision when available and generates missing concept groundings through proposal-based annotation followed by conservative multimodal verification. Using VIGOR, we evaluate whether MLLMs "look at what they say" by comparing model-derived visual evidence for each expressed concept against annotated ground-truth rationales. Our results show that current MLLMs exhibit limited visual rationale correctness and reveal distinct grounding failures across concept levels. VIGOR provides a benchmark and scalable data construction pipeline for studying whether MLLM outputs are supported by appropriate visual evidence.
Show more
View full details
Poster

Benchmarking Multi-Modal Graph-based Social Media Popularity Prediction

Utkarsh Sahu ⋅ Zhisheng Qi ⋅ Li Zhu ⋅ Yizhao Yang ⋅ Jun Li ⋅ Ryan Rossi ⋅ Yu Wang
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A906
Social media popularity prediction aims to forecast the future reach or influence of online content from early-stage observations. Accurate prediction enables key downstream applications, such as advertising optimization and strategic content planning by users, creators, and platforms. Despite substantial progress, existing popularity prediction works often fail to jointly consider multimodal content and temporal social interaction signals. Moreover, the literature remains highly fragmented across datasets, modalities, observation windows, prediction targets, and evaluation protocols. This fragmentation prevents fair comparison and obscures a systematic understanding of how textual, visual, temporal, and interaction-based signals jointly shape popularity dynamics. To address these challenges, we introduce MMG-Pop, a Multi-modal Graph-based Popularity Prediction benchmark, which unifies datasets, modalities, temporal interaction signals, and representative baselines under a standardized evaluation protocol. Furthermore, we propose MMG-PopNet, a unified multi-modal graph-based network that jointly models the aforementioned multi-modal signals and graph-structured social interactions. Extensive experiments on MMG-Pop, comprising four datasets across Bluesky and Reddit platforms, demonstrate the superior performance of MMG-PopNet and yield new insights into cross-platform training generalization, multi-task prediction benefits, multi-modality contributions, and LLM prediction limitation. These findings establish a unified foundation for future research on social dynamics modeling and intervention under heterogeneous modalities and socially-aware agentic ecosystem paradigms.
Show more
View full details
Poster

Beyond Truthfulness: Evaluating Honesty in Large Language Models

Richard Ren ⋅ Arunim Agarwal ⋅ Mantas Mazeika ⋅ Cristina Menghini ⋅ Brad Kenstler ⋅ Robert Vacareanu ⋅ Mick Yang ⋅ Isabelle Barrass ⋅ Alice Gatti ⋅ Xuwang Yin ⋅ Eduardo Trevino ⋅ Matias Geralnik ⋅ Dean Lee ⋅ Summer Yue ⋅ Dan Hendrycks
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A907
As large language models (LLMs) become more capable and agentic, the requirement for trust in their outputs grows significantly, yet at the same time concerns have been mounting that models may learn to lie in pursuit of their goals. To address these concerns, a body of work has emerged around the notion of "honesty" in LLMs, along with interventions aimed at mitigating deceptive behaviors. However, some benchmarks claiming to measure honesty in fact simply measure accuracy—the correctness of a model's beliefs—in disguise. Moreover, no benchmarks currently exist for directly measuring whether language models overtly lie. In this work, we introduce a realistic, large-scale human-curated dataset for directly measuring lies of commission, allowing us to disentangle accuracy from honesty. Across a diverse set of LLMs, we find that while larger models obtain higher accuracy on our benchmark, they do not become more honest. Surprisingly, most frontier LLMs obtain high scores on truthfulness benchmarks yet exhibit a substantial propensity to lie under pressure, resulting in low scores on our benchmark. We find that simple methods, such as representation engineering interventions, can improve honesty. These results underscore the growing need for robust evaluations and effective interventions to ensure LLMs remain trustworthy.
Show more
View full details
Poster

Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents

Wei-Chieh Huang ⋅ Weizhi Zhang ⋅ Yuchen Wu ⋅ Yankai Chen ⋅ Eric Hanchen Jiang ⋅ Wooseong Yang ⋅ Yiwei Yang ⋅ Henry P Zou ⋅ Hanrong Zhang ⋅ Ying Nian Wu ⋅ Haolun Wu ⋅ Kai-Wei Chang ⋅ Philip S Yu ⋅ Steve Liu ⋅ Aylin Caliskan
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A929
Memory is becoming core infrastructure for long-horizon LLM agents, yet existing evaluations offer limited guidance on which memory substrate should be used under different operating regimes. We present a controlled harness evaluation of memory substrates for memory-augmented agents, covering dense and sparse indices, text records, structural stores, hierarchical stores, refinement-based memories, parametric updates, and activation-compatible context mechanisms. Across three backbone models and four benchmark suites spanning user-centric question answering and agent-centric decision-making, we instrument 26 performance and efficiency metrics under a unified harness. Our results show that no single substrate consistently dominates: broad retrieval benefits long-context factual QA, while excessive retrieval can harm sequential decision-making by shifting attention away from action-critical context. Scalability further introduces an additional routing axis, as substrates that perform well at moderate history lengths can become costly or brittle at longer horizons. These findings motivate substrate routing as a necessary component of adaptive agent memory systems and provide empirical guidance for designing efficient, reliable, and regime-aware long-term memory for LLM agents.
Show more
View full details
Poster

Beyond Score: A Dataset for Joint Action and Score Predictions in 2v2 Sports

Yuting Wang ⋅ Peng Wen ⋅ Fengshun Wang ⋅ Jiayong Zuo ⋅ Xinyu Peng ⋅ Qiurui Wang
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A827
Current sports analysis methods often rely on high-level statistics and therefore provide limited insight into the action-level interactions underlying tactical outcomes. To enable fine-grained modeling of tactical processes, we present JAS-2v2, the first public dataset for action-based tactical analysis across 2v2 sports. By unifying beach volleyball, badminton doubles, tennis doubles, and table tennis doubles, JAS-2v2 enables comparative study of diverse tactical structures spanning both open and closed tactical systems under a shared sequence modeling framework. This dataset reframes score prediction as a sequence understanding problem, emphasizing how fine-grained action flows contribute to final tactical outcomes rather than only what the final outcome is. We further propose a unified action-to-score modeling framework based on graph reasoning and temporal modeling. Experiments show that our method outperforms strong baselines on both action prediction and score outcome prediction, and further demonstrate that fine-grained action semantics and sequential action dependencies are critical for tactical reasoning. In particular, using action prediction as an intermediate objective provides more effective representations for score outcome prediction.
Show more
View full details
Poster

Not All Spines Are Created Equal: How CT Segmenters Fail on Lumbosacral Transitional Vertebrae

Gregory Schwing ⋅ Patrick Schwing ⋅ Miraziz Ismoilov ⋅ Nizar Alnabahneh ⋅ Loren Schwiebert
Dec 10, 2:00 AM - 5:00 AM Hall C1 #A721
Wrong-level spine surgery remains a persistent never-event, with the Lumbosacral Transitional Vertebra (LSTV, 5–35% prevalence) as its largest anatomic driver. We audit TotalSegmentator — the most widely deployed CT segmentation system and default back-end of 3D Slicer — and show that on lumbarization cases its output is anatomically indistinguishable from a wrong-level surgical plan: lacking an L6 class, it shifts lumbar labels caudally by one level, with junction-DSC dropping 29 points on Any-LSTV. No existing CT benchmark surfaces this — not TotalSegmentator's own evaluation, not Li et al. (2025) (the per-class DSC frontier on the matched COLONOG split), and not VERIDAH (vertebra-labeling SOTA on a private CT cohort, operating on pre-localized crops with no pelvis). We contribute: (i) an LSTV-stratified evaluation protocol (per-class DSC, junction-DSC over a 40 mm L5/S1 window, per-class voxel confusion at the junction) translating the level-shift mechanism into quantifiable wrong-level surgical risk; (ii) CTSpinoPelvic1K — 1,153 CT volumes across 802 patients with unified spinopelvic masks, 33 LSTV-positive cases across a 6-way phenotype taxonomy, and radiologist Castellvi typing (I–IV) on all 33; (iii) the first publicly-released merge-based LSTV-handling dual-spinopelvic CT segmenter, exceeding TotalSegmentator on sacrum (TS=0.817 zero-shot) and uniquely delivering both spine and pelvis in a single forward pass with released weights; (iv) an empirical isolation showing VERIDAH's training-side L5/L6 merge eliminates the collision driving TotalSegmentator's level shift but exposes a residual L4/last_lumbar collision on count-style sacralization, motivating VERIDAH's sequence predictor as the necessary disambiguation component. CTSpinoPelvic1K, the protocol, and 5-fold checkpoints are released publicly.
Show more
View full details
Poster

Recreating Video Arenas via Automated Preference Scoring

Yue Zhao ⋅ Aniket Gupta ⋅ Juze Zhang ⋅ Tiange Xiang ⋅ Ryan Rong ⋅ Xuan Su ⋅ Fei-Fei Li ⋅ Ehsan Adeli
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1120
Evaluating video generation models is notoriously challenging; existing metrics either fail to align with human judgment or rely on costly, unscalable human ratings. In this work, we introduce \textbf{Video Preference Score} (VPS), a fully automated video preference score designed to directly model human judgment and accurately recreate large-scale, Elo-based video arenas. Trained on a newly collected dataset that augments generated videos with retrieved real-world footage, VPS evaluates paired videos against complex text prompts to predict human preference. Experiments demonstrate that VPS correlates highly with human consensus and generalizes zero-shot across unseen models and benchmarks. Furthermore, we showcase the utility beyond evaluation: as a plug-in reward model, VPS seamlessly integrates with Flow-GRPO to improve WAN-2.1's generative quality by 51 Elo points. Overall, VPS provides a scalable and generalizable solution for both evaluating and aligning modern video generation models.
Show more
View full details
Poster

SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

Liang You ⋅ Hengyu Shi ⋅ Dongwen Ou
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A200
Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy regions and under-covering hard ones. SimplexUQ is, to our knowledge, the first benchmark and reproducible protocol for measuring this allocation problem on simplex-valued predictions; it compares existing conformal wrappers rather than proposing a new one. Its task suite, SimplexTasks-12, combines six controlled synthetic regimes with six frozen-predictor real tasks spanning class probabilities, topic mixtures, spectral abundances, cell-type fractions, age distributions, and emotion mixtures. Each comparison fixes the predictor, score, and response-free stratification map, varies only the wrapper, and reports marginal coverage, worst-stratum coverage, max disparity, and within-task radius and compute. Global calibration can look valid while failing badly: on CIFAR-10 it attains 0.900 marginal coverage but only 0.542 in the worst entropy stratum, and Mondrian calibration raises that stratum to 0.886 while reducing max disparity from 0.358 to 0.022. No wrapper dominates, however. Under smooth synthetic heterogeneity, several repairs are competitive; fixed-map analyses show that rankings depend on the evaluation groups and protocol; and in a 12-task comparison, Mondrian has lower disparity on its single target partition for all 12 tasks, whereas BatchMVP has lower disparity over overlapping groups on five. These are empirical comparisons, not new coverage guarantees. A controlled predictor-bias sweep shows that removing predictor bias only partly reduces global-threshold disparity. We release task cards, result provenance, permitted derived arrays, and rebuild instructions, and treat wrapper selection as a diagnostic comparison rather than a universal ranking.
Show more
View full details
Poster

FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases

Xingjian Zhang ⋅ Sophia Moylan ⋅ Ziyang Xiong ⋅ Qiaozhu Mei ⋅ Yichen Luo ⋅ Jiaqi Ma
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A801
Scientific knowledge bases accelerate discovery by curating findings from primary literature into structured, queryable formats for both human researchers and emerging AI systems. Maintaining these resources requires expert curators to search papers, reconcile evidence across documents, and produce ontology-grounded annotations. Existing benchmarks usually evaluate isolated subtasks, such as named entity recognition or relation extraction, and therefore do not capture this end-to-end workflow. We present FlyAOC to evaluate AI agents on end-to-end agentic ontology curation from scientific literature. Given a gene symbol, a concise FlyBase gene description, access to a 16,898-paper corpus, and ontology resources, agents must search for evidence and recover as many curator-relevant structured annotations as possible. Outputs span standardized function terms, expression patterns, and historical synonyms linking decades of nomenclature. The benchmark includes 7,397 expert-curated annotations across 100 genes drawn from FlyBase, the Drosophila knowledge base. Across four baseline agent harnesses--memorization, fixed pipeline, single-agent, and multi-agent--FlyAOC is sensitive to harness design, model family, and tool-use reliability. These results reveal system-level failure modes that model-only evaluations do not capture. FlyAOC provides a reproducible testbed for retrieval-augmented scientific curation. - Code: https://github.com/xingjian-zhang/flyaoc - Data: https://huggingface.co/datasets/anonymous-042/flyaoc
Show more
View full details
Poster

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Gabriel Orlanski ⋅ Devjeet R Roy ⋅ Alexander Yun ⋅ Changho Shin ⋅ Alex Gu ⋅ Albert Ge ⋅ Dyah Adila ⋅ Nicholas Roberts ⋅ Frederic Sala ⋅ Aws Albarghouthi
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A919
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark comprising 36 problems and 196 checkpoints in which agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench identifies a failure mode that single-shot benchmarks cannot detect: agents that pass checkpoints while producing code that erodes and bloats with each extension.
Show more
View full details
Poster

StomataBench: Measuring Taxonomic Generalization in Stomatal Detection

Sainath Reddy Gummi ⋅ Hankui K Zhang ⋅ Chulwoo Pack ⋅ Shyam Solanki ⋅ Young Chang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A802
Accelerating stomata measurement is essential for developing drought-resilient plants. However, most automated stomata detectors are tested under narrow imaging and species conditions. Their behavior under broader ecological distribution shift remains unclear. We introduce StomataBench, a benchmark for stomatal object detection across species, acquisition protocols, and biological density regimes. The benchmark uses a curated multispecies training set of 3,154 images with 134,802 annotated stomata from 21 woody species. It is evaluated on four test datasets totaling 19,657 images, covering diverse woody taxa and heterogeneous public crop sources. Unlike standard detection benchmarks that report only AP, StomataBench evaluates measurement reliability. We measure COCO AP, F1, score-threshold sensitivity, image-level count error, and stomatal density agreement. We benchmark sixteen detectors spanning two-stage, one-stage, transformer, open-vocabulary, and promptable foundation models. The results show that model rankings change with the biological objective. GDINO-SB provides the strongest average AP generalization, while SAM3-OD gives reliable operating point for count and density estimation. Conventional detectors remain competitive on near-domain woody plants, but foundation models are robust under harder ecological shifts. Failure analysis shows that far-OOD detection is often limited by missing stomata. These results identify cross-lineage proposal generation as a key unsolved problem for reliable stomatal phenotyping under ecological distribution shift.
Show more
View full details
Poster

GeoFidelity-Bench: Evaluating Block-Conditioned Geographic Fidelity in Text-to-Image Street-View Generation

Kaizhen Tan
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A618
Text-to-image models can generate plausible street scenes from prompts such as “a street in New York City,” but it remains unclear whether these images resemble the requested place or show a generic city appearance. Commonly used evaluation metrics mainly measure image quality or text alignment and do not directly assess geographic fidelity. We introduce GeoFidelity-Bench, a benchmark for evaluating whether generated street-view images match a target location at the level of named street blocks. The benchmark contains 112 street blocks from 25 cities across six continents, each paired with a panel of real Mapillary images and geographic metadata. From this metadata we build prompts with the city only, with street and neighborhood names, and with these names plus raw GPS coordinates. Across six open-weight text-to-image models, street and neighborhood names improve similarity to the target location, while raw GPS coordinates add only a small and uneven gain. Control prompts that swap in the names of other blocks while keeping the city fixed reduce retrieval accuracy, with larger decreases when the neighborhood name or both names are replaced. Generated images also distinguish their target blocks from local alternatives less reliably than held-out real images. CLIP alignment between generated images and their prompts is nearly uninformative about whether the images achieve block-level geographic fidelity. GeoFidelity-Bench provides a controlled testbed for studying location-conditioned generation and for measuring progress beyond generic city-level plausibility.
Show more
View full details
Poster

Handwriting decoding as a challenging motor task for EEG Foundation Models

Srinivas Ravishankar ⋅ Ishayu Ghosh ⋅ Nora Zajzon ⋅ Teng (Simon) Fei ⋅ Virginia de Sa
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A823
Recent attempts at creating Foundation Models (FMs) for Electroencephalography (EEG) have achieved state-of-the-art performance on multiple tasks including Motor Imagery (MI). These MI tasks have typically involved coarse classification between imagined limb movements. However, the development of foundation models necessitates diverse datasets, both for pretraining and evaluating the progress of these models. In this work, we propose handwriting decoding as a challenging motor task for FMs. We show that several existing datasets are affected by confounds, and introduce a dataset that more rigorously evaluates models. On this dataset, we find that current FMs, despite showing SOTA performance in multiple MI datasets are outperformed by smaller task-specific models. We also highlight challenges specific to EEG-based handwriting decoding to inform future work. In our 4-letter classification task, we show that (a) Knowledge of movement-onset is crucial to reported decoding performance in prior works, with average performance across subjects dropping from $41.3\%$ to $32.4\%$. (b) Increasing test-time signal quality provides significant performance improvements ($45\%$ to $78\%$ in our best subject) compared to scaling training data with single-trial EEG. (c) While scaling training data steadily improves decoding performance, existing FMs do not outperform specialist models in handwriting decoding. We make our code and dataset available at https://anonymous.4open.science/r/EEG-Handwriting-BCI-DFCD
Show more
View full details
Poster

Composable Causality: A Toolkit for Systematic Time-Series Causal Discovery and Treatment-Effect Benchmarking

Muhammad Hasan Ferdous ⋅ Md Osman Gani
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A924
Real-world multivariate time series datasets rarely come with verified ground-truth causal structure, since the underlying mechanisms are typically unknown and randomized intervention is infeasible in most domains. Empirical research on time-series causal discovery and treatment-effect estimation therefore depends on synthetic data, where the data-generating process is specified by construction. Existing data generation tools fall into two categories. Some are fixed datasets that provide pre-generated time series, which cannot be modified after release. Others are script-based generators, where each script fixes one combination of mechanism, confounding, and missingness, so any new combination requires writing new code. Neither supports the controlled experiments that method development demands, where a single property such as sparsity, noise distribution, or missingness mechanism is varied while others are held fixed. A further gap concerns counterfactual data. Prior work in time-varying treatment-effect estimation has relied on per-paper simulators with hardcoded mechanisms, while general-purpose time-series generators produce observational data only, leaving evaluation of conditional average treatment effect estimators reliant on conditional expectations rather than ground-truth individual effects. We introduce DTCD (Datasets and Toolkit for Causal Discovery), a data generation library that produces multivariate time series together with their ground-truth contemporaneous and lagged adjacency matrices. Topology, mechanism, noise, observational mask, and intervention policy are independent components that compose freely, enabling single-axis sensitivity studies through configuration alone. Paired counterfactual twins are produced by caching the stochastic realization during the factual rollout and re-injecting it during the interventional rollout, recovering individual treatment effects to numerical precision. To demonstrate the library, we present a reference benchmark of eight discovery algorithms across eighty-one discovery configurations and four treatment-effect estimators across eighteen CATE configurations, exposing failure modes not visible in aggregate metrics, including the inability of standard discovery methods to handle missing values regardless of whether the missingness mechanism is informative.
Show more
View full details
Poster

JARVIS-Bench: Benchmarking Personal Intelligence Agents on Long-Horizon Real-User Daily Traces

Weizhi Zhang ⋅ Wei-Chieh Huang ⋅ Yueqing Liang ⋅ Liwei Jiang ⋅ Zhengxiang Wang ⋅ Liangwei Yang ⋅ Zechen Li ⋅ Yuchen Wu ⋅ Haozhen Zhang ⋅ Yu Wang ⋅ Yuanchen Bei ⋅ Yue Zhou ⋅ Siqi Zhu ⋅ Henry P Zou ⋅ Jiahong Liu ⋅ Xinni Zhang ⋅ Paul Martin ⋅ Joseph Marvin Imperial ⋅ Yuyang Luo ⋅ Xiongxiao Xu ⋅ Baixiang Huang ⋅ Shanglin Wu ⋅ Lucas Resck ⋅ Yibo Wang ⋅ Yuqing Liu ⋅ Langzhou He ⋅ Chengze Li ⋅ Yuxin Tian ⋅ Kening Zheng ⋅ Enze Ma ⋅ Boi Huynh ⋅ Zheng Hui ⋅ Rui Yang ⋅ Tao Feng ⋅ Jie Yang ⋅ Shanghao Li ⋅ Haoran Wang ⋅ Wooseong Yang ⋅ Hanrong Zhang ⋅ Huanhuan Ma ⋅ Daye Yoon ⋅ Yaozu Wu ⋅ Shikan Lian ⋅ Xiaoqian Ruan ⋅ Fangxin Wang ⋅ Hyeonjeong Park ⋅ Hins Hu ⋅ Wenzhe Fan ⋅ Chen Wang ⋅ Yifan Gu ⋅ Dongyuan Li ⋅ Song Wang ⋅ Aylin Caliskan ⋅ Jindong Wang ⋅ Xiao Luo ⋅ Yankai Chen ⋅ Kai Shu ⋅ Steve Liu ⋅ Philip S Yu
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A918
Personal intelligence agents aim to support users across daily-life tasks by learning their preferences, routines, needs, and emotional states from long-term interaction. They are expected to anticipate user needs, provide proactive assistance, and infer preferences that users have never explicitly articulated. However, existing benchmarks for LLM personalization lack real user data spanning extended periods and diverse daily-life domains, and typically reduce personalization to simplified settings such as recommendation or fact retrieval. We introduce JARVIS-Bench, a new benchmark for evaluating personal intelligence agents, curated from real users’ daily traces and augmented with synthetic trajectory expansions grounded in implicit-preference insights from surveys and structured personas. JARVIS-Bench is built from participants who contributed 28 to 44 days of multimodal self-tracking logs, structured personas, and a 100-question implicit-preference survey spanning eight domains. It supports four downstream task surfaces: implicit preference reasoning, proactive-help prediction, emotion-aware modeling, and cross-domain transfer, all evaluated through a unified event iterator and long-horizon user-memory pipeline. We benchmark 28 LLMs from 10 model families, including Gemini, GPT, Qwen, Gemma, and Claude, across a five-condition input ladder ranging from no memory to lifelong memory. Results show that memory and model scaling improve personalization, yet current frontier models remain far from achieving robust real-world personal intelligence.
Show more
View full details
Poster

MPCI-Bench: A Benchmark for Multimodal Pairwise Contextual Integrity Privacy Evaluation of Language Model Agents

Shouju Wang ⋅ Haopeng Zhang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1001
As language-model agents evolve from passive chatbots into proactive assistants that handle personal data, evaluating their adherence to social norms becomes increasingly critical, often through the lens of Contextual Integrity (CI). However, existing CI benchmarks are largely text-centric and primarily emphasize negative refusal scenarios, overlooking multimodal privacy risks and the fundamental trade-off between privacy and utility. In this paper, we introduce MPCI-Bench, the first Multimodal Pairwise Contextual Integrity benchmark for evaluating privacy behavior in agentic settings. MPCI-Bench consists of paired positive and negative instances derived from the same visual source and instantiated across three tiers: normative Seed judgments, context-rich Story reasoning, and executable agent action Traces. Data quality is ensured through a Tri-Principle Iterative Refinement pipeline. Evaluations of state-of-the-art multimodal models reveal systematic failures to balance privacy and utility and a pronounced modality leakage gap, where sensitive visual information is leaked more frequently than textual information.
Show more
View full details
Poster

HearSayBench: Can LLMs Navigate from Abstract Human Rights to Lived Lives?

Sobhan Lotfi ⋅ Ava Iranmanesh ⋅ Ali Iranmanesh ⋅ Liwei Jiang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1013
"Hearing is never like being.'' —Persian Proverb Large language models have become the default advisors for life-critical human problems. While they democratize access to personal counseling, they suffer from a silent foundation bias: the internet is a record of people with the freedom to act. Current benchmarks assume users have this same agency, ignoring the reality of those in war zones or navigating statelessness. These long-tail experiences are not only missing from training data, but authentic evaluation data to measure them is equally scarce. We introduce HearSayBench, a human-verified dataset of 400 scenarios from respected archives like the United Nations, covering 80 regions across three specific barriers: social, personal, and environmental. Our work uses Capabilities Approach to test if a model can distinguish between what a person is legally promised and what they are actually free to do in their specific environment. While these models may have "heard'' about global inequality during training, we find that this knowledge is merely hearsay. Across 11 frontier and open-weight models, we identify a systemic %37 performance drop between situational comprehension and structural reasoning. When faced with the most vulnerable users, models consistently offer a "Checklist of Impossible Things'': polite, fluent advice that is physically impossible or legally suicidal to follow. Ultimately, we show that the true digital divide is no longer about access to technology, but about whether an AI can recognize the reality of your life.
Show more
View full details
Poster

RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities

Tarek Elsayed ⋅ Shiping Yang ⋅ Eunsong Koh ⋅ Sanika Goyal ⋅ Vincent Huang ⋅ Paul Ngo ⋅ Nathan Young ⋅ Mohammad Omidvar Tehrani ⋅ Alvyn Kang ⋅ Arnell Kang ⋅ Zeyu Chen ⋅ Angelica Moreira ⋅ Xuan Feng ⋅ Angel Chang ⋅ Nick Sumner ⋅ Steve Ko
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1014
LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace. They typically rely on small non-compilable snippets, focus on binary classification (vulnerable or not), and do not account for the risk that publicly-released datasets are part of model training corpora. We introduce RustMizan, a benchmarking framework for Rust vulnerability analysis that addresses these gaps. RustMizan contains compilable code variants at the crate, file, and function levels, with annotations for binary vulnerability detection, CWE classification, and function- and line-level localization. A paired mutation framework produces semantics-preserving code mutants for contamination testing and robustness probing. Across four frontier models in an agentic setup with command-line access, binary classification sits in the 56--65\% range, but line localization F1 stays near 20\%, and adversarial cues drop line F1 by about 27\%.
Show more
View full details
Poster

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

Bohan Lyu ⋅ Yucheng Yang ⋅ Siqiao Huang ⋅ Jiaru Zhang ⋅ Qixin Xu ⋅ Xinghan Li ⋅ Xinyang Han ⋅ Yicheng Zhang ⋅ Huaqing Zhang ⋅ Runhan Huang ⋅ Kaicheng Yang ⋅ Zitao Chen ⋅ Wentao Guo ⋅ Junlin Yang ⋅ Xinyue Ai ⋅ Wenhao Chai ⋅ Yadi Cao ⋅ Ziran Yang ⋅ Kun Wang ⋅ Dapeng Jiang ⋅ Huan-ang Gao ⋅ Shange Tang ⋅ Chengshuai Shi ⋅ Simon Du ⋅ Max Simchowitz ⋅ Jiantao Jiao ⋅ Dawn Song ⋅ Chi Jin
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1122
Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. As large language models demonstrate advanced capabilities in reasoning, coding, and engineering tasks, it is increasingly important to understand whether they can discover such methods rather than only apply existing ones. We introduce MLS-Bench, a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods. MLS-Bench contains 140 tasks across 12 domains, each requiring an agent to improve one targeted component of an ML system or algorithm and demonstrate that the improvement generalizes across controlled settings and scales. We find that current agents remain far from reliably surpassing human-designed methods, and that engineering-style tuning is easier for them than genuine method invention. We further study the effects of test-time scaling, adaptive compute allocation, and context provision on agents' discovery performance, together with case studies of their behavior. Our analyses suggest that the bottleneck is not only in proposing new methods, but also in the scientific insight needed to plan, validate, and scale claims about them. More search, compute, or context alone does not remove this bottleneck. We build and maintain a community platform for cumulative and comparable iteration, and release the data and code at https://mls-bench.com.
Show more
View full details
Poster

SiliconBench: Speed, Memory, and Fidelity for LLM Inference on Apple Silicon

Ranran Haoran Zhang ⋅ Aysa Fan ⋅ David M Correia ⋅ Alex Cheema ⋅ Rui Zhang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A418
Apple Silicon is a major consumer platform for local LLM inference, with at least ten serving engines competing for users. Speed-only leaderboards on this platform mislead: the fastest stack often claims memory the rest of the machine needs, or silently produces wrong output. We introduce \ourbench, a benchmark that evaluates ten Apple Silicon inference stacks through three lenses: throughput and latency at concurrency 1, 8, and 16; peak memory under contention with the operating system; and fidelity as weighted F1 on a classification task against an NVIDIA reference run on identical weights. The benchmark covers three model releases and two workload types (short chat and multi-turn agent prompts averaging ${\sim}$4K input tokens). A maintainer agent re-runs the full benchmark weekly; in its first five runs it caught a single upstream library bump that broke three of ten stacks through three distinct failure modes and diagnosed a chat-template error that corrupted one stack's fidelity scores. No audited stack performs well across all three lenses, and the stack rankings shift when memory and fidelity are read alongside speed. The candidate set, stacks without disqualifying gaps under any lens, reduces from ten to four. A separate multi-node comparison shows that tensor parallelism over RDMA scales decode $1.3\times$ on two nodes, while pipeline parallelism over TCP regresses $16$--$19\%$. In every case the gaps are in the engines, not the hardware: Apple Silicon provides sufficient memory, compute, and interconnect; the engines have not yet exploited it. We release the harness, per-run results, and weekly journals.
Show more
View full details
Poster

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

David Restrepo ⋅ Chenwei Wu ⋅ Luis Nakayama ⋅ Miguel L Martins ⋅ Stergios Christodoulidis ⋅ Maria Vakalopoulou ⋅ Enzo Ferrante
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A806
Progress in AI-based retinal image analysis has accelerated with the adoption of foundation models, yet evaluating their real-world reliability remains challenging. Performance reported on a single dataset does not capture how models behave under dataset shift, across clinical definitions, or for different patient subgroups. This limitation is particularly critical in in medical imaging analysis, where robustness, calibration, and fairness are essential for safe deployment. We introduce FOCUS (Foundation Ophthalmic Cross-Dataset Understanding under Shift), a cross-dataset benchmark for evaluating retinal fundus models that considers vision-only encoder models (VM), vision-language dual-encoder models (VLM), and multimodal large language models (MLLM). FOCUS harmonizes binary diabetic retinopathy, referable diabetic retinopathy, and glaucomatous optic neuropathy tasks across ten public datasets spanning diverse geographies, acquisition conditions, and label protocols. The benchmark evaluates models through a unified analysis layer that measures ranking performance, calibration, subgroup disparities, and image-quality robustness. We present a large-scale evaluation covering 532 base configurations and 228 low-rank adaptation LoRA-adapted MLLMs trained with supervised fine-tuning (SFT). Results show that no model family consistently dominates across tasks and datasets: general VM encoders achieve the strongest average ranking performance, medical MLLMs are competitive but variable, and dual encoder VLMs benefit substantially from lightweight adaptation. Fine-tuning improves in-domain performance but exhibits heterogeneous transfer to external datasets, particularly in calibration. These findings demonstrate that retinal model evaluation is inherently multidimensional. FOCUS provides a practical framework and public benchmark to assess generalization, reliability, and robustness beyond single-dataset leaderboards.
Show more
View full details
Poster

Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models

Aditi Sarker ⋅ Nazreen Shah ⋅ Rafi Ibn Sultan ⋅ Rhongho Jang ⋅ Dongxiao Zhu ⋅ Prashant Khanduri
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A715
Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks; however, they often exploit spurious object-background correlations, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence. Despite growing interest in hallucination and robustness evaluation, existing benchmarks provide limited control over whether model predictions are grounded in the target object or induced by correlated background cues. In this work, we introduce PURGE (Partition-aware Unlearning for Removing spurious-correlation Generated Errors), a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs. The framework consists of two components: (1) Structured dataset construction, wherein we develop three complementary structured data construction strategies that partition examples by object-relevant evidence and spurious background cues, enabling controlled diagnosis of shortcut reliance; and (2) Partition-aware unlearning, which uses these partitions to selectively remove spurious object-background associations while preserving object-based reasoning. We evaluate the PURGE framework across multiple LVLMs, including LLaVA-1.6-7B, Qwen3-VL-8B-Instruct, and Qwen3.5-9B, together with CLIP as a vision-language encoder, on a diverse suite of benchmarks, including CHAIR, POPE, Causal-HalBench, MM-SpuBench, AMBER, MMHal, and Waterbirds. Our results show that PURGE consistently reduces hallucinations and spurious-correlation-driven errors while maintaining or improving overall performance, providing both a reusable evaluation protocol and an effective mitigation framework for more reliable LVLMs.
Show more
View full details
Poster

TorchUMM: A Unified Multimodal Model Codebase for Evaluation, Analysis, and Post-training

Yinyi Luo ⋅ Wenwen Wang ⋅ Haoyue Bai ⋅ Hongyu Zhu ⋅ Hao Chen ⋅ Pan He ⋅ Marios Savvides ⋅ Sharon Li ⋅ Jindong Wang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1121
Recent advances in unified multimodal models (UMMs) have led to a proliferation of architectures capable of understanding, generating, and editing across visual and textual modalities. However, developing a unified framework for UMMs remains challenging due to the diversity of model architectures and the heterogeneity of training paradigms and implementation details. In this paper, we present TorchUMM, the first unified codebase for comprehensive evaluation, analysis, and post-training across diverse UMM backbones, tasks, and datasets. TorchUMM supports a broad spectrum of models covering a wide range of scales and design paradigms. Our benchmark encompasses three core task dimensions: multimodal understanding, generation, and editing, and integrates both established and novel datasets to evaluate perception, reasoning, compositionality, and instruction-following abilities. By providing a unified interface and standardized evaluation protocols, TorchUMM enables fair and reproducible comparisons across heterogeneous models and fosters deeper insights into their strengths and limitations, facilitating the development of more capable unified multimodal systems. Code is available at: https://anonymous.4open.science/r/TorchUmm-1E7E.
Show more
View full details
Poster

OpenBrain: An Auditable Generated-Label Release for Whole-Brain MRI Parcellation

Qizhen Lan ⋅ Yu-Chun Hsu ⋅ YUXIANG WEI ⋅ Lijing Zhu ⋅ Zenan Sun ⋅ Liang He ⋅ Lishan Yu ⋅ Xiaoqian Jiang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A708
Automatic tools can now generate whole-brain MRI parcellations at cohort scale, but generated-label releases often provide little evidence about which cases are reliable enough to reuse. We introduce OpenBrain, a confidence-aware release of 35{,}838 provenance-verified OpenNeuro T1w cases from 607 source datasets. OpenBrain is organized as an auditable case-record release rather than a directory of standalone segmentation files: each record links a primary parcellation, two committee parcellations, source/license provenance, raw QC-7 measurements, source-local percentile risks, a combined risk score R, and a confidence tier. QC-7 is a reference-free reliability panel over label geometry, image-label consistency, and committee disagreement, so users can filter, weight, audit, or re-threshold generated labels without rerunning inference. We evaluate this released evidence under a frozen protocol. Across external cohorts, lower QC-7 risk corresponds to higher generated-label quality; in fixed-budget training, lower-risk selections improve paired macro Dice over random selection by +$3.2$ to $+3.7$ percentage points and substantially outperform high-risk-tail selection. OpenBrain also exposes raw axes, committee outputs, provenance, and datasheet documentation, so confidence claims remain inspectable after download. It does not turn generated labels into manual annotations or clinical-grade segmentations. Its contribution is a large open generated-parcellation substrate, a reusable case-level risk-evidence layer, and an external validation protocol for release-time QC.
Show more
View full details
Poster

StereoPep: Do Molecular Models Understand Stereochemistry? A Benchmark on Synthetic Diastereomeric Peptides

Michael Desgagné ⋅ Amirabbas Kazeminia ⋅ Kübra Kaygisiz ⋅ Bradley Pentelute ⋅ Marinka Zitnik
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A701
Existing peptide models saturate natural-sequence proteomics benchmarks, but whether they capture the underlying physicochemical dynamics or merely pattern-match on sequence statistics remains unclear. Public peptide datasets offer no axis along which to test this: non-canonical residues and stereochemical variation are essentially absent. We introduce StereoPep, a benchmark of 48,788 chemically synthesized peptides spanning 11 libraries, produced in the laboratory via high-throughput flow synthesis and characterized experimentally by reverse-phase liquid chromatography retention time. Unlike model-generated synthetic data, every peptide in StereoPep is a real molecule with a real chromatographic measurement. The dataset is constructed around matched diastereomeric pairs (sequences identical in primary structure but differing at a single stereocenter), providing a direct test of whether models capture three-dimensional stereochemical information. StereoPep enables progress across multiple fields: it serves as a benchmark for stereochemistry-aware molecular representations and geometric deep learning, supports adapting protein language models to non-canonical residues, and underpins emerging proteomics multiplexing strategies that encode samples through retention-time shifts. Across specialized retention-time models, graph-based architectures, and protein language models, all approaches predict overall retention reasonably well (Pearson $r \approx 0.80$) but collapse on diastereomer pair discrimination (AUC $\approx 0.55$), revealing that current representations encode atomic composition far better than 3D structure.
Show more
View full details
Poster

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Miaosen Chai ⋅ Wang Bill Zhu ⋅ Shangshang Wang ⋅ Yejia Liu ⋅ Song Bian ⋅ Honghua Dong ⋅ Willie Neiswanger ⋅ Robin Jia
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A920
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often hack unit tests by regenerating correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce Precise Debugging Benchmarking (PDB), a general, dataset-agnostic framework that converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measure how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single on single-line bugs and PDB-Wild on multi-line and repository-level bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging on single-line bugs. Iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
Show more
View full details
Poster

StarCraft Motion: A Dataset for Agent Simulation in Adversarial and Partially Observable Scenarios

Yi-Chung Chen ⋅ Mingyu Kim ⋅ Ruqi Bai ⋅ James Z Hare ⋅ Jing Gao ⋅ David Inouye
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A511
Agent simulation aims to model the future behavior of interacting agents, but existing benchmarks have largely focused on structured domains such as autonomous driving. Adversarial and partially observable settings remain less explored, despite their importance for modeling strategic multi-agent behavior. To address this gap, we introduce StarCraft Motion, a large-scale dataset for agent simulation built from human StarCraft~II replays. Our processing pipeline converts raw replays into standardized simulation scenarios by extracting continuous unit trajectories, deriving command-based unit-level intent labels, and aligning game states with player-specific fog-of-war observations. This enables direct study of behavior, intent, and opponent responses under competitive partial observability. As an initial study, we evaluate autoregressive simulation methods on StarCraft Motion and introduce HMART, a hierarchical baseline that extends SMART with learnable player-level query aggregation for efficient global-context modeling in large-agent-count scenarios. Experiments show that global context improves observer-unit simulation and intent prediction, while closed-loop fine-tuning further improves rollout quality. However, accurately capturing intent initiation and transitions remains difficult. We also find that opponent modeling from partial observations is substantially more challenging, and that additional global information can be misleading when it does not match the opponent's perspective. These results highlight StarCraft Motion as a useful dataset for studying scalable agent simulation in adversarial, partially observable environments.
Show more
View full details
Poster

SiliciclasticReservoirs: A Million-Reservoir Dataset and Flow-Matching Foundation Model for 3D Siliciclastic Reservoir Generation

Ilgar Baghishov ⋅ Elnara Rustamzade ⋅ Graeme Henkelman ⋅ John Foster ⋅ Michael J Pyrcz
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A921
3D subsurface geological models underpin hydrocarbon exploration, geothermal energy, underground gas storage, and geological carbon storage. A decade of deep generative facies modeling has yet to yield a model that generalizes across geological settings, scales to realistic reservoir extents, and conditions naturally on well data, bottlenecked by the absence of a large-scale, multi-environment, property-rich dataset. We release SiliciclasticReservoirs, the largest and most diverse public 3D dataset of siliciclastic (sandstone-bearing) subsurface reservoirs to date: $10^6$ voxelized reservoirs at $64{\times}64{\times}32$ resolution spanning eight depositional settings (deep-water turbidite lobes, deltas, and six fluvial channel architectures from the Alluvsim family), each annotated with binary and six-class facies labels, per-voxel porosity and permeability, and a natural-language caption describing the reservoir's geological setting and petrophysical attributes. To our knowledge, it is the first siliciclastic dataset to pair 3D geological models with petrophysical properties at this scale, the first to span this breadth of settings in a unified release, and the first to ship text captions, supporting future multimodal and language-conditioned training for text-to-reservoir generation and retrieval. Building on SiliciclasticReservoirs, we train ResFlow, a flow-matching foundation model that, from a single set of weights, generates any environment under continuous parametric control, conditions on arbitrary well configurations via classifier-free guidance, and assembles reservoirs at arbitrary spatial extent through overlapping-tile MultiDiffusion, with no retraining. SiliciclasticReservoirs (CC-BY 4.0), ResFlow (MIT), and the data-generation engine ResMill (MIT) are publicly released.
Show more
View full details
Poster

CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging

Shiyang Li ⋅ Haoyang Chen ⋅ Mattia Fazzini ⋅ Caiwen Ding
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A201
Debugging CUDA programs has long been challenging because failures often arise from subtle interactions among hardware behavior, compiler decisions, memory hierarchy, and asynchronous execution. More importantly, with the rapid expansion of GPU usage across scientific computing, machine learning, graphics, and systems workloads, CUDA debugging has become more challenging than ever. Current evaluations of LLM-based CUDA programming largely miss this setting: a model can pass correctness tests with repair by degeneration, simplifying the CUDA code into a safer but slower program that abandons the original optimization structure. We introduce CUDABEAVER, a benchmark for CUDA debugging from real failing workspaces produced during LLM-based CUDA generation. Each task provides the broken candidate, native build/test commands, raw error evidence, and a single editable file. CUDABEAVER evaluates whether a fixer truly repairs the failing CUDA code or merely finds a slower test-passing replacement, reporting results by failure category, debugging trajectory, stagnation mode, and performance preservation. We further propose pass@k(M,C,A), a protocol-conditional CUDA debugging metric by making the fixer M, corpus C, and protocol axes Aexplicit. Using this metric across 213 tasks and seven frontier LLMs, we show that protocol-aware evaluation gives a more faithful view of CUDA debugging ability: when performance-loss tolerance is high, fixers appear much stronger, but even a minor stricter performance requirement can sharply reduce measured success, shifting scores by up to 40 percentage points.
Show more
View full details
Poster

Evaluating Depth and Breadth in Test-Time Scaling for Compositional Visual Generation

Shantanu Jaiswal ⋅ Mihir Prabhudesai ⋅ Nikash Bhardwaj ⋅ Zheyang Qin ⋅ Amir Zadeh ⋅ Chuan Li ⋅ Katerina Fragkiadaki ⋅ Deepak Pathak
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A619
Test-time scaling is increasingly used to improve visual generation, yet it remains unclear how different scaling strategies should be evaluated when they trade off accuracy, latency, throughput, compute cost, and visual quality in different ways. We study this question through a systematic comparison of prominent test-time strategies including sampler-depth scaling, breadth-oriented parallel sampling, step-by-step construction, iterative refinement, and hybrid breadth--depth for compositional text-to-image generation. Our results show that the preferred strategy depends on prompt complexity and cost assumptions: parallel sampling remains competitive for simpler prompts and high-throughput settings, while iterative refinement becomes increasingly useful as prompts require more object, spatial, numeric, and relational bindings. Surprisingly, step-by-step generation does not consistently inherit the benefits of step-by-step reasoning in language models, since early visual decisions about field of view, layout, scale, and resolution can constrain later edits. Across fixed budgets, hybrid policies often provide the strongest accuracy--cost tradeoff by combining parallel exploration with sequential correction. Finally, we show that refinement-based scaling is bottlenecked by verifier and editor reliability, and that adaptive routing can recover much of the benefit of fixed hybrid policies at lower average cost. Overall, our findings show that effective test-time scaling depends on matching the policy to the prompt and deployment bottleneck: breadth supports exploration and throughput, depth supports semantic repair, and hybrid/adaptive policies offer the strongest tradeoff when compositional correctness is the priority.
Show more
View full details
Poster

Lost on Campus: Evaluating Embodied Spatial Reasoning of Vision-Language Models in the Wild

Zehan Zheng ⋅ Yanyuan Chen ⋅ Deming Li ⋅ Yutao Tang ⋅ Jianwen Xie ⋅ Alan Yuille ⋅ Jieneng Chen ⋅ Cheng Peng
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A714
Embodied Spatial Reasoning (ESR) is central to deploying Vision-Language Models (VLMs) in real-world embodied tasks, yet remains inadequately evaluated. Existing benchmarks primarily assess spatial reasoning in static images or synthetic indoor environments, where semantic shortcuts often suffice, making it difficult to faithfully evaluate spatial reasoning capabilities in the wild. In light of this, we present Lost on Campus, a benchmark for evaluating ESR in large-scale real-world outdoor 3D environments reconstructed by 3D Gaussian Splatting. Our benchmark introduces a unified reasoning-action evaluation framework that seamlessly integrates diagnostic QA for isolated reasoning with closed-loop interactive navigation for active reasoning under multimodal instructions. To enable fine-grained diagnosis, we systematically decompose ESR into six fundamental capabilities: action grounding, spatial foresight, metric awareness, goal-directed planning, global localization, and spatio-temporal consistency. Extensive experiments reveal that: (i) Compared to indoor settings, real-world outdoor environments impose substantially higher demands on spatial reasoning, where existing models exhibit poor performance; (ii) VLMs still struggle with fine-grained visual alignment, with failures in spatial foresight and self-aware localization emerging as primary bottlenecks; (iii) Improving multimodal interaction capabilities and long-term spatial reasoning is crucial for advancing embodied intelligence. These findings highlight that faithful evaluation of ESR demands benchmarks tightly coupling perception, reasoning and action within realistic and continuous environments.
Show more
View full details
Poster

AgentAbstain: Do LLM Agents Know When Not to Act?

Xun Liu ⋅ Yi Evie Zhang ⋅ Vira Kasprova ⋅ Parisa Rabbani ⋅ Pardissadat Zahraei ⋅ Tianyu Zhang ⋅ Ali Ebrahimpour-Boroojeny ⋅ Varun Chandrasekaran
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A1012
Agent systems based on Large Language Models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain. This gap poses real risks: under ambiguity, conflicting constraints, or tool failures, agents may execute unintended and irreversible actions. To close this gap, we present the first systematic evaluation framework for agentic abstention: the ability to recognize when not to act. At its core, AgentAbstain is a paired-task benchmark that defines 8 abstention categories spanning pre-execution reasoning and runtime discovery, with 263 paired tasks across 42 executable sandbox environments, each pairing a should-act task with a should-abstain variant produced by a controlled perturbation to the instruction, tool, or environment state. Scaling such paired evaluations poses two practical challenges: manually authoring diverse tasks is expensive, and static benchmarks risk data contamination as models evolve. To address both, we propose AbstainGen, a fully automated pipeline that synthesizes sandbox environments and generates paired tasks end-to-end, validated by deterministic replay and semantic LLM judges. Its scalable design enables on-demand regeneration of fresh task instances, and human validation rates 96% of generated tasks as well-designed. Evaluating 17 frontier LLMs across 4 agent harnesses, the best model (Gemini 3.1 Pro) achieves only 59.5% paired accuracy (correct on both the act and abstain sides of each paired task). More importantly, abstention capability is largely independent of general task-solving capability, indicating that scaling task-solving alone will not close this gap. We identify failure modes such as Post-hoc Abstention, in which agents execute irreversible actions before recognizing abstention triggers. For instance, an agent may cancel a flight reservation before noticing contradictory rebooking instructions, leaving the user stranded. These findings underscore the need for rigorous abstention evaluation to develop more trustworthy LLM agents.
Show more
View full details
Poster

Do Joint Audio-Video Generation Models Understand Physics?

Zijun Cui ⋅ Xiulong Liu ⋅ Hao Fang ⋅ Mingwei Xu ⋅ Jiageng Liu ⋅ Zexin Xu ⋅ Weiguo Pian ⋅ Shijian Deng ⋅ Feiyu Du ⋅ Chenming Ge ⋅ Yapeng Tian
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A620
Joint audio-video generation models are rapidly approaching professional production quality, raising a fundamental question: do these models truly understand audio-visual physics, or do they merely generate plausible sounds and frames that violate real-world physical consistency? To answer this question, we introduce AV-Phys Bench, the first comprehensive benchmark for evaluating physical commonsense in joint audio-video generation. AV-Phys Bench systematically tests joint audio-video generation models across three scene categories that probe how physical commonsense holds as the scene evolves: (a) Steady State, (b) Event Transition, and (c) Environment Transition. Each scene category investigates three physics-grounded subcategories that reflect real-world scenes, along with an additional Anti-AV-Physics subcategory, where prompts deliberately violate audio-visual physics to probe whether models possess generative physics knowledge or merely encode physically consistent priors. Each generation is scored along five dimensions: semantic adherence and physical commonsense within each modality, and cross-modal physical commonsense, which tests whether the visual and audio streams agree on the same physical event. Across three proprietary and four open-source models, Seedance 2.0 leads on physical commonsense, with an overall pass rate of 0.660. The gap to open-source models remains pronounced; every model degrades by up to 67% on event-driven and environment-driven transitions, and proprietary leaders collapse by 45–69% on Anti-AV-Physics prompts. Beyond human evaluation, we introduce AV-Phys Agent, a ReAct-style agentic evaluator that pairs a multimodal language model with deterministic acoustic measurement tools and ranks generators in close alignment with human ratings, enabling scalable physical-commonsense evaluation without further human annotation. AV-Phys Bench identifies cross-modal physical consistency and transition-driven scene dynamics as the open frontier for joint audio-video generation. We release AV-Phys Bench and AV-Phys Agent to facilitate future research on joint audio-video generation and evaluation.
Show more
View full details
Poster

ForceBody: Force-Paired Parametric Body Motion with Torque Uncertainty

Joonwoo Kwon ⋅ Yufei Zhang ⋅ Xiaoming Liu ⋅ Zijun Cui
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A627
Forces and torques drive human motion, central to biomechanics, robotics, and physics-aware animation, yet they are often absent from learning-based motion pipelines for body reconstruction and generation. Some motion datasets provide force annotations, but the paired motion sequences are incompatible with the parametric body models used in modern learning pipelines. In addition, existing datasets typically obtain joint torques through inverse dynamics, which is inherently sensitive to the quality of the input kinematics; however, they do not quantify the reliability of these torque estimates. We close this gap with \textbf{ForceBody}, the first dataset to bring ground truth force supervision into a parametric body representation, shipped with per-sample torque uncertainty. ForceBody pairs the SKEL body model with ground reaction forces and inverse-dynamics joint torques across $10{,}389$ motion trials ($9.7$M frames, $27$ hours) from $140$ subjects. Each torque label is paired with a per-frame, per-joint uncertainty annotation obtained by Monte Carlo sampling through the inverse dynamics pipeline. We benchmark six architectures (MLP, Conv1D, LSTM, GRU, Mamba2, Transformer) on ground reaction force and joint torque prediction from SKEL motion. As one example use of the released uncertainty, finetuning a Transformer with per-sample uncertainty weighting reduces the MAE of joint torque estimation by $12\%$.
Show more
View full details
Poster

Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?

Ming Zhong ⋅ Stacy Li ⋅ Nicholas Carlini ⋅ Matthew Jagielski
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A821
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce **Agent4Sci**, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on [GitHub](https://github.com/maszhongming/agent4sci) and [Hugging Face](https://huggingface.co/datasets/MingZhong/agent4sci).
Show more
View full details
Poster

FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

Minwei Kong ⋅ Chonghe Jiang ⋅ Ao Qu ⋅ Wenbin Ouyang ⋅ Zhaoming Zeng ⋅ Xiaotong Guo ⋅ Zhekai Li ⋅ Junyi Li ⋅ Yi Fan ⋅ Xinshou Zheng ⋅ Xi Jing ⋅ Yikai Zhang ⋅ Zhiwei Liang ⋅ Seonghoo Kim ⋅ Runqing Yang ⋅ Sirui Li ⋅ Han Zheng ⋅ Wangyang Ying ⋅ Ou Zheng ⋅ Chonghuan Wang ⋅ Jinglong Zhao ⋅ Paul Liang ⋅ Jinhua Zhao ⋅ Hai Wang
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A302
Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research often requires a harder capability: designing scalable algorithms that exploit problem structure and outperform direct formulation-and-solve baselines. Existing benchmarks are limited to small or simplified examples far below real-world scale and complexity. We introduce **FrontierOR**, the first large-scale benchmark targeting LLM-based efficient algorithm design for realistic optimization problems. FrontierOR includes 110 tasks derived from methodologically diverse papers published in top-tier operations research venues, each with standardized instances and a hidden, human-verified evaluation suite. We evaluate seven frontier LLMs in one-shot and test-time evolution settings. The results reveal that frontier models still struggle to move from executable formulations to efficient optimization algorithms: the strongest one-shot model outperforms Gurobi in only **39\%** of cases in terms of both solution quality and runtime, and even strong coding agents with test-time evolution achieve only **50\%** on selected hard tasks. FrontierOR establishes a practical evaluation platform for LLM-based optimization algorithm design, which enables future LLMs and agents to be systematically tested on whether they can move beyond correct formulation toward feasible, high-quality, and runtime-competitive algorithms.
Show more
View full details
Poster

Differential Item Functioning as an Item-Level Diagnostic for LLM Benchmarks

Hansol Lee ⋅ Jason Cho ⋅ Nick Haber ⋅ Benjamin Domingue ⋅ Sanmi Koyejo
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A123
Comparative claims about large language model (LLM) performance are typically based on aggregate benchmark scores. This treats models with similar scores as comparable on the items the benchmark contains, an assumption that can fail if models from different populations succeed on systematically different items. We test this assumption using Differential Item Functioning (DIF), a psychometric diagnostic that flags items whose relative difficulty differs across groups after conditioning on overall performance. Applying Mantel--Haenszel DIF analyses to six widely-used LLM benchmarks across two widely-used open-source model families (Llama and Qwen), we find that many items behave differently across families even among models with the same total benchmark score, with some items favoring Llama-family models and others favoring Qwen-family models. The flagged items concentrate in benchmark-provided categorical content (subjects, subtasks, problem types), and choosing items based on which family they favor can substantially amplify, attenuate, or reverse aggregate family-level comparisons. Our results show that aggregate benchmark scores can summarize different item-level performance patterns across model populations, and that DIF offers a useful diagnostic for surfacing this variation.
Show more
View full details
Poster

CSI-TextBench: A Dataset and Benchmark for Language-Grounded Ambient Sensing Perception

Guozhen Zhu ⋅ Yuqian Hu ⋅ Sakila S Jayaweera ⋅ Wei-Hsiang Wang ⋅ Beibei Wang ⋅ K. Liu
Dec 10, 8:30 AM - 11:30 AM Hall C1 #A112
Indoor context deciphered by ambient sensing underpins many applications, from assisted living to smart buildings, yet remains weakly connected to the language-aligned paradigm of modern multimodal systems. Through billions of IoT devices, ambient WiFi Channel State Information (CSI) captures the indoor context of human activity and environmental dynamics passively via ubiquitous, always-on infrastructure. However, unlike vision or audio, CSI lacks naturally paired text like captions or transcripts, limiting its integration with language-driven AI systems. We present CSI-TextBench, a large-scale dataset and benchmark for aligning CSI data with natural language, unifying 3 sensing benchmarks into 285,790 samples across 7 tasks and pairing them with 2,291 hierarchically structured, label-derived descriptions via many-to-many assignment to construct 2.0 million CSI–text training tuples. Benchmarking 5 alignment methods from 4 paradigm families, we show that contrastive CSI--text alignment matches supervised classifiers without class labels, adapts to unseen classes with few samples, and responds to open-vocabulary natural-language prompts. A single unified model supports compositional scene understanding, answering joint queries over activity, proximity, and identity, achieving 86.9\% top-1 accuracy on 140 composite scenes, outperforming task-specific classifier ensembles. CSI-TextBench enables ubiquitous ambient sensing as a language-indexed perception layer, enabling indoor contextual awareness in language-driven AI systems.
Show more
View full details
Poster

SPATIALEPIBENCH: Benchmarking Spatial Information and Epidemic Priors in Forecasting

Ruiqi Lyu ⋅ Alistair Turcan ⋅ Bryan Wilder
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A802
Accurate epidemic forecasting is crucial for public health response, resource allocation, and outbreak intervention, but remains difficult with sparse, noisy, and highly non-stationary data. Because epidemics unfold across interacting regions, spatiotemporal methods are natural candidates for improving forecasts. Despite growing interest in spatial information, no standardized benchmark exists, and current evaluations often use simple chronological train-test splits that do not reflect real-time forecasting practice. We address this gap with SpatialEpiBench, a challenging benchmark for spatiotemporal epidemic forecasting in realistic public-health settings. SpatialEpiBench includes 11 epidemic datasets with standardized rolling evaluations and outbreak-specific metrics. We evaluate adjacency-informed forecasting models with widely used epidemic priors that adapt general models to epidemiology, but find that most methods underperform a simple last-value baseline from 1 day to 1 month ahead, even during outbreaks and with these priors. We identify three major failure modes: (1) poor outbreak anticipation, (2) difficulty handling sparsity and noise, and (3) limited utility of common geographic adjacency for epidemiological spatial information. We release benchmark data, code, and instructions at https://github.com/Rachel-Lyu/SpatialEpiBench to support development of operationally useful epidemic forecasting models.
Show more
View full details
Poster

GraphIP–Bench: How Hard Is It to Steal a Graph Neural Network, and Can We Stop It?

Kaixiang Zhao ⋅ Bolin Shen ⋅ Yuyang Dai ⋅ Shayok Chakraborty ⋅ Yushun Dong
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1020
Graph neural networks (GNNs) deployed as cloud services can be \emph{stolen} through \emph{model-extraction attacks}, which train a surrogate from query responses to reproduce the target's behaviour, and a growing line of ownership defenses tries to prevent or trace such theft. The title of this paper asks two questions: \emph{how hard is it to steal a GNN?}, and \emph{can we stop it?} Prior work cannot answer either, because experiments use inconsistent datasets, threat models, and metrics. We introduce \emph{GraphIP-Bench}, a unified benchmark which evaluates both sides under a single black-box protocol. It integrates twelve extraction attacks, twelve defenses spanning watermarking, output-perturbation, and query-pattern-detection families, ten public graphs covering homophilic, heterophilic, and large-scale regimes, three GNN backbones, and three graph-learning tasks, and it reports fidelity, task utility, ownership verification, and computational cost on shared splits, queries, and budgets. We further add a joint attack-and-defense track which runs every attack on every defended target and measures watermark verification on the resulting surrogate, which exposes the protection that a defense retains after extraction. The empirical picture is short: stealing a GNN is easy at medium query budgets and most defenses do not change this; several watermarks verify reliably on the protected model but lose most of their verification signal on the extracted surrogate, which exposes a gap that single-model evaluations miss; and heterophilic graphs are systematically harder to steal, while a cross-architecture mismatch between target and surrogate reduces but does not prevent extraction. Code: \href{https://anonymous.4open.science/r/GraphIPBench-C641}{anonymous.4open.science/r/GraphIPBench-C641}.
Show more
View full details
Poster

BrainTRACE: Tracing Longitudinal, Multimodal, and Volumetric Evidence in Brain MRI Clinical Reasoning

Qizhen Lan ⋅ Mengchen Fan ⋅ Hang Zhang ⋅ Jingwei Duan ⋅ Moule Lin ⋅ Jialin Chen ⋅ Baocheng Geng ⋅ Xiaoqian Jiang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A719
Brain MRI interpretation is a longitudinal clinical reasoning problem: radiologists compare serial studies, integrate information across MRI sequences, localize findings within volumetric anatomy, and translate this evidence into report-grounded assessments. Existing medical VQA and 3D imaging benchmarks capture important parts of this workflow, but often evaluate brain MRI through isolated images, static volumes, or ungrounded report-style answers, thereby obscuring failures in the evidence chain that support clinical validity. We introduce BrainTRACE, a report-grounded benchmark for evaluating whether vision-language models can trace the evidence structure required for longitudinal brain MRI interpretation. BrainTRACE contains 7,273 scored VQA instances derived from 1,778 longitudinal patients, 7,299 MRI studies, and approximately 29k co-registered 3D MRI sequence volumes. The benchmark is organized by five levels of clinical reasoning, from acquisition recognition to case-level synthesis, and by evidence demands covering longitudinal comparison, report-grounded references, multi-sequence integration, and volumetric spatial evidence. BrainTRACE supports rendered inputs compatible with standard VLM interfaces, a 3D-evidence condition, and a decomposed case-reasoning track that audits six steps in a longitudinal evidence chain. Evaluation of 20 VLM configurations shows that current systems can identify isolated visual cues but rarely compose them into grounded longitudinal interpretations. We release the benchmark specification, evaluation lists, scoring implementation, scoring rubrics, and audit-record format to support reproducible progress in brain MRI VLM evaluation.
Show more
View full details
Poster

CellularS2-Bench: A Staged, Evidence-Grounded Benchmark for Cellular Network Security Reasoning

Ke Xie ⋅ Xingyi Zhao ⋅ Yiwen Hu ⋅ Haotian Lin ⋅ Shuhan Yuan ⋅ Wen Li ⋅ Tian Xie
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A111
Cellular networks are critical infrastructure supporting billions of worldwide users and safety- and mission-critical services. Vulnerabilities in cellular networks can therefore cause service disruption, privacy breaches, and broad societal harm, motivating growing efforts to analyze 3GPP specifications that define required device and operator behavior. However, existing benchmarks for this domain share three systematic deficiencies: they rely on LLM-generated entries without expert verification, they do not require faithful interpretation of normative language, reasoning across cross-referenced clauses, and verifiable conclusions grounded in multimodal evidence such as tables and figures, and they do not evaluate security reasoning over end-to-end protocol behavior. To address these gaps, we present CellularS2-Bench, a staged benchmark. CellularS2-Bench covers core control-plane specifications, processing over 1,760 paragraphs, 13,800 sentences, 424 figures, and 340 tables. It comprises 7,851 questions and instances derived from 3GPP specifications and 43 real-world vulnerabilities across three progressive stages. Beyond newly constructed tasks, CellularS2-Bench integrates and corrects subsets from 7 prior datasets to broaden task coverage. To establish a reproducible baseline on CellularS2-Bench, we introduce CellularS2-ARI, a unified Adapt-Retrieve-Integrate framework for systematic understanding and specification-driven security analysis of 3GPP specifications. Together, they establish an accessible and reproducible foundation for quantifying progress in specification understanding and security reasoning in cellular network security.
Show more
View full details
Poster

MetaPI: Constructing Prompt Injection Benchmarks from Any Agent Benchmarks

Peiran Wang ⋅ Chong Xiang ⋅ Wenjie Qu ⋅ Ying Li ⋅ Jiaheng Zhang ⋅ G. Edward Suh ⋅ Yuan Tian
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1019
Agents powered by large language models (LLMs) are vulnerable to indirect prompt injection (IPI), in which malicious instructions embedded in external content can influence agent behavior at execution time. Progress in defending against IPI has largely been assessed using existing benchmarks. However, current IPI benchmarks are often limited by (1) oversimplified tasks and environments, (2) static attack objectives and strategies, and (3) non-adaptive attack payloads. To address these limitations, we present MetaPI, an extensible toolkit that turns any existing non-security agent benchmark into an agentic IPI benchmark, with built-in support for dynamic attack objective selection and adaptive attack payload optimization. We apply MetaPI to diverse agentic settings and evaluate a broad range of defenses across multiple backbone LLMs. Our results suggest that widely used benchmarks such as AgentDojo can substantially overstate both robustness and utility, especially when defenses are deployed in more realistic, complex environments. This further underscores the significance of MetaPI, the first toolkit of its kind to provide scalable, realistic, dynamic, and adaptive benchmarks for evaluating IPI defenses.
Show more
View full details
Poster

DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMs

Md Hasebul Hasan ⋅ Krity H Charu ⋅ Eshwara Prasad Sridhar ⋅ Shuchisnigdha Deb ⋅ Mohammad A Islam
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A815
Effective de-escalation is critical for law enforcement safety and community trust, yet traditional training methods lack scalability and realism. While Large Language Models (LLMs) enable dynamic, open-ended simulations, their substantial computational footprint renders them impractical for deployment on the lightweight, portable hardware required for immersive field training. Small Language Models (SLMs) offer a viable real-time alternative but suffer from a critical scarcity of high-quality, domain-specific training data. To bridge this gap, we present DeEscalWild, a novel benchmark dataset curated from a multi-stage pipeline of ``in-the-wild`` police-civilian interactions extracted from publicly available video repositories. Starting with 5,000 raw inputs, we employed a rigorous hybrid filtering process combining human-in-the-loop verification with ``LLM-as-a-Judge`` evaluation to distill 1,500 high-fidelity scenarios. The resulting corpus comprises 285,887 dialogue turns, totaling approximately 4.7 million tokens. Extensive experiments demonstrate that SLMs fine-tuned on this data significantly outperform their base counterparts across ROUGE-L, BLEU-4, METEOR, BERTScore, Realism Score, and human evaluation metrics. Notably, our fine-tuned Qwen 2.5 (3B-Instruct) surpasses the general-purpose Gemini 2.5 Flash model when evaluated under equivalent conditions, demonstrating that domain-optimized SLMs can achieve superior performance with a fraction of the computational cost.This work establishes the civilian-response simulation agent and supporting dataset needed for accessible, low-latency de-escalation training at the edge. Code and dataset are available at \url{https://github.com/Hasebul/DeEscalWild-Benchmark-Framework} (code) and \url{https://doi.org/10.7910/DVN/CWMCZI} (dataset).
Show more
View full details
Poster

SchemeArena: Factorized Stress Testing of Scheming in LLM Agents

Jie Ruan ⋅ Inderjeet Nair ⋅ Amy Liu ⋅ Muhammad Khalifa ⋅ Yusheng Zhou ⋅ Lu Wang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1008
We study scheming in LLM agents, in which agents covertly pursue misaligned goals. Our focus is to understand how scheming arises from the interaction of key factors, such as instrumental goals, environmental affordances, oversight conditions, and perceived consequences. Prior work examines only a small number of scenarios, limiting the ability to isolate how these conditions shape an agent's propensity or capability to scheme. This limited scale and task diversity also restrict coverage of realistic deployment settings and the range of scheming strategies that can be observed. To this end, we introduce SchemeArena, a 400-scenario benchmark for scalable scheming stress testing, constructed through a factorized scenario synthesis framework spanning diverse safety-relevant tool domains, instrumental goals, oversight conditions, and pressure mechanisms. To enable scalable and reliable monitoring, we further propose SCOUT, a scheming monitor that grounds multi-criteria judgments in evidence drawn from agents' reasoning and actions. Across controlled stress tests on five LLM agents, we find that explicit instrumental goals are the strongest driver of scheming propensity. Strategic hints play a distinct role by helping agents translate scheming reasoning into concrete covert behavior. Oversight has mixed effects: in several closed models, action-only monitoring increases scheming, suggesting that partial oversight can act as an optimization constraint rather than a deterrent. CoT is a useful but incomplete monitoring signal: it can reveal latent scheming before execution, yet action-only scheming shows that covert behavior may occur without explicit reasoning evidence.
Show more
View full details
Poster

Rethinking SAE Evaluation: An Atomic Interpretable Unit Framework Reveals Hidden Polysemanticity and Redundancy

Riteng Zhang ⋅ Xiaoqian Wang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1000
Sparse autoencoders (SAEs) were originally motivated by discovering monosemantic features, a property that characterizes individual features but does not specify what the learned feature set should look like as a whole. As SAEs have increasingly been adopted for attribution, circuit discovery, and intervention, researchers have begun evaluating metrics that describe global properties of the feature set, such as sparsity, dead feature rates, reconstruction fidelity, and disentanglement, a diverse set that extends far beyond monosemanticity. However, this implicit shift beyond monosemanticity has not been formalized as a clear, well-defined objective that comes before these evaluation metrics. Without such an objective, existing metrics remain ad hoc, with their necessity unjustified. In this work, we formalize this ultimate objective by introducing a new paradigm for SAE evaluation: requiring that SAE features, and more generally dictionary-based decompositions of model representations, form an atomic interpretable unit basis, a coordinate system for semantically meaningful explanation. Under this paradigm, we organize existing and new measurements such as DAME (Dominance-Aware Monosemanticity Evaluation) as proxy metrics and introduce a comprehensive benchmarking framework. Evaluating hundreds of SAEs across multiple architectures and model families, we uncover failure modes in which some SAEs appear monosemantic under existing evaluation metrics while remaining functionally polysemantic: the most dominant features, those contributing most to model computation, are systematically less interpretable and more redundant. These failure modes warrant attention from SAE researchers as targets for improved training and architectures, and motivate dominance-aware evaluation as a standard check for practitioners before SAE features are deployed for attribution, circuit tracing, or intervention.
Show more
View full details
Poster

Theory-Scale Auto-Formalization of Logics for Computer Science

Yuming Feng ⋅ Frederick Pu ⋅ One An ⋅ Osbert Bastani ⋅ Li Zhang ⋅ Jiani Huang ⋅ Xujie Si ⋅ Ziyang Li
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A911
Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-formalization, which coherently translates hundreds of interdependent definitions, lemmas, and theorems, remains open due to challenges in consistency, faithfulness, scalability, and correctness. In this paper, we introduce LCS-Bench, a stand-alone, theory-scale benchmark based on Logics for Computer Science. LCS-Bench is built through a novel semi-automated agentic pipeline that leverages dependency graphs, concept tagging, issue tracking, and iterative refinement, complemented by significant faithfulness review from human experts. The resulting artifact covers 327 textbook items, over 4,076 Lean declarations, and more than 85K lines of Lean code. The dataset supports broad evaluation through a data engine that automatically derives five tracks of evaluation benchmarks, measuring different aspects of auto-formalization and theorem-proving capabilities. We also introduce a novel evaluation protocol featuring definitional equivalence checkers, enabling more fine-grained and faithful assessment. Through extensive evaluation on 14 models, we demonstrate that (1) LCS-Bench is of high quality, consistent, and faithful; (2) the benchmark is challenging, with state-of-the-art models achieving only 20.1% on auto-formalization tasks; and (3) our analysis reveals key findings regarding theory-scale auto-formalization and suggests promising directions for future work.
Show more
View full details
Poster

Visual Robustness and Neural Alignment in a Shared Foraging Task: The Mouse vs. AI Benchmark

Marius Schneider ⋅ Joe Canzano ⋅ Yuchen Hou ⋅ Jing Peng ⋅ Anjali Deepu ⋅ Utsab Karan ⋅ Phu-Hoa Pham ⋅ Chi-Nguyen Tran ⋅ Dao S Minh ⋅ Phu-Quy Nguyen-Lam ⋅ Trung-Kiet Huynh ⋅ Simone Azeglio ⋅ Spencer Smith ⋅ Michael Beyeler
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A826
Visual robustness under real-world conditions remains a critical bottleneck for modern reinforcement learning agents. In contrast, biological systems such as mice show remarkable resilience to environmental changes, maintaining stable performance even under degraded visual input with minimal exposure. Inspired by this gap, we established the Mouse vs. AI: Robust Foraging Competition, a novel bioinspired visual robustness and neural alignment benchmark in which agents and mice perform the same naturalistic 3D foraging task. The benchmark consists of two tracks: (1) a robustness track evaluating generalization to unseen visual perturbations, and (2) a neural alignment track measuring how well model representations predict large-scale neural recordings from mouse visual cortex. We provide a public benchmark suite comprising a Unity-based environment for training and evaluating virtual agents, centralized evaluation protocols, and an extensive neural dataset with approximately 178 minutes of multimodal recordings of behavior, visual input, and neural activity from more than 50,000 neurons in mice performing the task. As part of the NeurIPS competition, we received submissions from 22 teams proposing diverse architectures and training strategies. By embedding evaluation in a shared sensorimotor task performed by both biological and artificial agents, the benchmark enables a class of evaluative claims not supported by existing robustness or neural alignment benchmarks in isolation: namely, whether robust, brain-like visual representations emerge spontaneously from behavior-driven learning, and whether robustness and neural alignment reinforce or trade off against one another. Access to the dataset as well as the benchmarking infrastructure will remain online at https://robustforaging.github.io.
Show more
View full details
Poster

TopoGraphRAG-Bench: Evaluating Multimodal GraphRAG on Layout-Grounded Evidence Reasoning

Ruochi Li ⋅ peter lin ⋅ Haoxuan Zhang ⋅ Haihua Chen ⋅ Junhua Ding ⋅ Edward Gehringer ⋅ Yang Zhang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A112
Real-world documents distribute evidence across text, tables, figures, captions, and page layouts. Answering complex questions over such documents therefore requires more than retrieving relevant passages: systems must recover the evidence topology that connects heterogeneous evidence units. Existing GraphRAG evaluations focus primarily on text-derived chunks, entities, and relations, while multimodal document RAG benchmarks evaluate cross-modal retrieval and generation without directly testing whether systems recover the intended reasoning structure. We introduce \textsc{TopoGraphRAG-Bench}, a layout-grounded benchmark for multimodal evidence reasoning in GraphRAG. The benchmark contains 2,024 questions over 201 long, visually rich documents, with evidence grounded in text, figures, and tables. Each instance is constructed bottom-up from layout-grounded evidence units and assigned one of three controlled topologies: single-hop retrieval, bridge-chain reasoning, or multi-source synthesis. To ensure that questions preserve their intended structure, we apply counterfactual validation for shortcut resistance, modality necessity, and evidence necessity. We evaluate text-only GraphRAG, page-level visual retrieval, and multimodal GraphRAG systems using retrieval, generation, and topology-aware reasoning metrics. Multimodal GraphRAG systems achieve the strongest overall performance, but still fail when visual-textual evidence alignment or multi-unit composition is incomplete. Text-only GraphRAG struggles when key dependencies are grounded in figures or tables, while page-level visual retrieval lacks the fine-grained structure needed for topology recovery. These results suggest that future GraphRAG systems must model not only textual entities and relations, but also layout structure, modality alignment, and the reasoning roles of evidence units. Code and data are available at \url{https://anonymous.4open.science/r/Benchmark-B1DE}.
Show more
View full details
Poster

Your Benchmark Is an Empirical Measure Over Difficulty

Yifan Sun ⋅ Naicheng Yu ⋅ Jiawen Gong ⋅ Jingyan Shen ⋅ Huan Zhang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A816
A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: \emph{measure mismatch}, where the benchmark difficulty distribution differs from the intended difficulty distribution, and \emph{coverage gaps}, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze \textbf{25} popular benchmarks spanning \textbf{7} domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor model improvement process. Finally, we turn diagnosis into action by introducing \textsc{BenchFill}, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps.
Show more
View full details
Poster

From Detection to Understanding — A Multi-Task Dataset for Traffic Anomaly Reasoning

Han Zhang ⋅ yilin zhao ⋅ Zaid Pervaiz Bhat ⋅ Zheng (Thomas) Tang ⋅ Varun Praveen ⋅ Vidya Nariyambut Murali ⋅ David Anastasiu ⋅ Tomasz Kornuta
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A113
We present TAR (Traffic Anomaly Reasoning), a large-scale multi-task dataset for training and evaluating vision-language models on traffic anomaly understanding. While video anomaly detection (VAD) has traditionally focused on binary classification or temporal localization, TAR moves beyond detection to reasoning. TAR contains 44,040 training annotations with explicit chain-of-thought reasoning traces across 10 task types, covering 3,670 CCTV transportation videos (~26 hours) sourced from eight public datasets. TAR provides a structured progression of tasks organized into three groups — Question Answering, Temporal Reasoning, and Scene Understanding — reflecting our finding that reliable anomaly understanding demands capabilities far beyond detection. Annotations are produced by a hierarchical auto-labeling pipeline using state-of-the-art VLMs, with existing human annotations incorporated as supplementary context for 25% of videos. Every answer is paired with a step-by-step reasoning trace, enabling both supervised training with chain-of-thought supervision and fine-grained evaluation of model reasoning. We evaluate eight vision-language models on a held-out test set of 84 videos (1,008 annotations) with human-reviewed and corrected annotations and find that current models struggle with Temporal Reasoning and Scene Understanding tasks even when they perform well on basic Question Answering — revealing that detection accuracy is a poor proxy for genuine understanding. A multi-task ablation study shows that progressively adding task groups during fine-tuning yields consistent gains, with the full 10-task model achieving +44.5 points on Binary QA accuracy and +21.4 points in aggregate score over the zero-shot baseline, demonstrating the dataset's value as a training resource. TAR is released under CC-BY-4.0 and serves as the official training data for AI City Challenge 2026 Track 3: Anomalous Events in Transportation. The annotations and video download scripts are publicly available at https://huggingface.co/datasets/nvidia/PhysicalAI-Traffic-Anomaly-Reasoning.
Show more
View full details
Poster

GlucoFM-Bench: Benchmarking Time-Series Foundation Models for Blood Glucose Forecasting

Baiying Lu ⋅ Zhaohui Liang ⋅ Ryan Pontius ⋅ Shengpu Tang ⋅ Temiloluwa Prioleau
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A803
Blood glucose forecasting models are foundational for modern diabetes management systems, as reliable short-term predictions can enable proactive interventions, support automated insulin delivery, and reduce the risk of hypo- and hyperglycemic events. From a modeling perspective, glucose forecasting poses unique challenges due to heterogeneous physiological dynamics across different diabetes populations. Traditional machine learning and deep learning models have been extensively evaluated for glucose prediction. However, there is significantly less research on evaluating and benchmarking recent time-series foundation models (TSFMs) for blood glucose forecasting. To bridge this gap, we present GlucoFM-Bench, a comprehensive benchmark evaluating state-of-the-art TSFMs alongside supervised deep learning models for blood glucose forecasting. We assess eight representative architectures, including pre-trained TSFMs, time-series large language models, and task-specific deep learning models, across 15 publicly available diabetes-relevant datasets comprising 1,117 individuals with type 1 diabetes, type 2 diabetes, prediabetes, and no diabetes. Models are evaluated under zero-shot, few-shot, and full-shot protocols, with systematic variation in context length and prediction horizon. Across datasets, pre-trained TSFMs, especially Chronos-2 and TimesFM, show strong zero-shot and few-shot transfer, with the best zero-shot model performing within 5% of the best full-shot supervised model. Yet, when task-specific data is abundant, a lightweight LSTM remains strongest, outperforming TSFMs by 4-21% under full-shot training. Stratified analyses reveal persistent challenges in T1D cohorts and hypo-/hyperglycemic ranges, highlighting the need for evaluation beyond aggregate error metrics. Together, GlucoFM-Bench provides a standardized and reproducible foundation for evaluating, comparing, and improving foundation models for blood glucose forecasting. The dataset and code are publicly available at https://huggingface.co/datasets/glucofmbench/GlucoFM-Bench and https://anonymous.4open.science/r/GlucoseML_benchmark-B50C, respectively.
Show more
View full details
Poster

ShadowTransfer: A Geographic Transfer Benchmark for Overhead Shadow Detection

Shagun Mittal ⋅ Raymond A. Yeh ⋅ Satish Ukkusuri
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A717
Overhead aerial imagery feeds dense-prediction pipelines that are routinely trained in one geographic region and deployed in another, yet no benchmark measures how shadow detection models transfer across regions. Existing overhead shadow datasets either lack a geographic transfer protocol or rely on automatically generated labels. We introduce \shadow, a benchmark of $4{,}500$ human-verified shadow masks across three U.S.\ cities at two native resolutions, paired with a leave-one-city-out evaluation protocol. Across three architectures and seven adaptation methods, all adaptation methods degrade under geographic transfer. We propose a three-probe framework that diagnoses where and how the transfer breaks: the failure operates in feature space (not pixel space), shows up as a confidence collapse on shadow pixels while background confidence stays intact, and is recoverable by retraining different architectural components depending on the model: encoder for some, decoder for others. These findings replicate on INRIA building footprint extraction under the same protocol, suggesting they describe geographic transfer broadly rather than something specific to shadows. Guided by the diagnosis, we build Spectral Information Bottleneck (SIB), a plug-in module with one component per finding. Across the $9$ (architecture, held-out-city) cells, \sib{} attains the highest mean mIoU ($70.92$) and the smallest worst-case gap to the within-city upper bound ($7.18$ mIoU). The benchmark, LOCO protocol, code, and trained models are released to support reproducible evaluation of geographic transfer in overhead dense prediction.
Show more
View full details
Poster

Conditional Multi-Event Temporal Grounding in Long-Form Video

Yuanhao Zou ⋅ Arthad Kulkarni ⋅ Lucas Toñanez ⋅ Lincoln Spencer ⋅ Guangyu Sun ⋅ Tianxingjian Ding ⋅ Andong Deng ⋅ Yi Li ⋅ Shuangjun Liu ⋅ Yuan Li ⋅ Dashan Gao ⋅ Ning Bi ⋅ TAOTAO JING ⋅ Shuai Zhang ⋅ Chen Chen
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A718
Multimodal large language models have made rapid progress in video temporal grounding, yet real-world applications routinely require localizing every event that satisfies compositional temporal and spatial conditions. Existing benchmarks fall short: they localize only a single moment per query, count without temporal conditions, or treat grounding and counting as disjoint tasks. We introduce CoMET-Bench for Conditional Multi-Event Temporal Grounding in long-form video, comprising 2,789 queries over 600 videos averaging 33.8 minutes across five real-world domains, with each query composed from 4 temporal conditions, 3 spatial conditions, and a dedicated negative-query subset. We further propose a unified evaluation protocol jointly measuring counting, grounding, and negative-query recognition, including a new Rejection-F1 metric that prevents trivial gaming by lazy "always-empty'' models. Benchmarking a broad suite of MLLMs, agent-based, and grounding-specialized methods reveals that existing approaches remain far from solving this task. Building on these findings, we propose CoMET-Agent, a training-free agentic framework that reformulates the task as structured search-and-aggregate, improving F1@0.5 by 6.1% over GPT-5 purely through structural reasoning. Failure analysis further surfaces three open directions: fine-grained entity tracking, position-uniform retrieval, and causal event pairing.
Show more
View full details
Poster

AndroidReality: How Far Are Mobile Agents from the Real World?

Xiaoou Liu ⋅ Longchao Da ⋅ Hanyang Chen ⋅ Yuan Ling ⋅ Hua Wei
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A720
Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real-world deployment due to environmental variations and imperfect interface conditions. In this work, we introduce AndroidReality, a perturbation-based framework for evaluating and improving the robustness of mobile agents. Through a Markov Decision Process (MDP) perspective, we organize real-world interface variability into a principled taxonomy of perturbations along three axes: state, transition, and action. Guided by this taxonomy, we build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections, enabling systematic robustness evaluation of mobile agents. Our evaluation reveals substantial robustness gaps and four recurring error categories, motivating a simple training-free Test-Time Introspective Recovery (TTIR) mechanism that mitigates these failures on both perturbed and clean settings. Together, these results position robustness as a missing dimension in mobile agent evaluation and establish benchmark perturbation as an effective tool for both stress testing and surfacing latent weaknesses of mobile agents. Benchmark and implementation are available.
Show more
View full details
Poster

Argus: A Cross-Regime Benchmark for the Transferability of Uncertainty Quantification in Computer-Use Agents

DIVAKE KUMAR ⋅ Sina Tayebati ⋅ Devashri Naik ⋅ Amanda Rios ⋅ Nilesh Ahuja ⋅ Omesh Tickoo ⋅ Ranganath Krishnan ⋅ Amit Trivedi
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1018
Computer-use agents turn visual predictions into executable GUI clicks, so uncertainty estimates must support rejection, calibration, miss-severity ranking, and spatial safety regions. Existing post-hoc uncertainty quantification (UQ) evidence is fragmented across isolated model and dataset pairs, leaving unclear whether UQ rankings remain stable when the agent, benchmark, or observable interface changes. We present a cross-regime benchmark for post-hoc UQ in single-step executable GUI grounding, covering a 27-method, seven-family open-weight matrix over 4 GUI-grounding VLM agents and 4 datasets, plus an 8-method API-compatible closed-source matrix across 3 frontier vendors where logits, hidden states, and attention maps are unavailable. The main finding is selective transfer: UQ rankings are stable across datasets for a fixed model, but degrade across model classes and observable interfaces. Hidden-state and density methods form the most stable open-weight family, while CoCoA-1MCA, Focus, sampling-based scores, and verbalised self-assessment win in specific regimes. Ranking transfer is strongest within a fixed model across datasets, reaching Spearman $\rho=0.969$ and averaging $\rho=0.705$ over 120 open-weight pairs. In contrast, open-weight to closed-source transfer is the strongest break: cross-tier Spearman $\rho$ averages only $+0.08$ over 12 vendor x dataset pairs on the shared 8-method intersection, indicating that UQ recommendations do not survive removal of internal model signals. Model-class transitions further reshape UQ preferences: attention, verbalised, and VLM-native families lose AUROC on every dataset, while density methods remain stable; a scale-only baseline shows that logit-family degradation on ScreenSpot-Pro is partly confounded by scale rather than grounding fine-tuning alone. Finally, conformal click regions show that score-level discrimination is not enough for deployment: locally weighted disks can shrink radii by 40-60% when the plug-in UQ is calibrated, but coverage can degrade under calibration-test or interface mismatch. We release per-item records, calibration/test splits, UQ scores, and analysis scripts as a reproducible basis for regime-aware UQ selection in GUI agents.
Show more
View full details
Poster

Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors

Shuhaib Mehri ⋅ Philippe Laban ⋅ Sumuk Shashidhar ⋅ Marwa Abdulhai ⋅ Sergey Levine ⋅ Michel Galley ⋅ Dilek Hakkani-Tur
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A817
As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behaviors of real users. While existing works train user simulators to generate human-like responses, whether they capture the broad and heterogeneous distribution of real user behaviors remains an open question. In this work, we introduce a method to measure the distributional gap between real and simulated user behaviors, validated through a human study and ablations. Given a dataset of real and simulated conversations, our method extracts representations of user behavior from each conversation, quantizes them into discrete distributions via clustering, then computes divergence metrics. We provide the first systematic evaluation of 24 LLM-based user simulators on coding and writing tasks, and reveal a large distributional gap from real users that varies across model families, scales, and behavioral facets. Pairwise comparisons show that most simulators behave similarly, while a few stand apart. Combining behaviorally complementary simulators brings the resulting distribution closer to real users compared to either simulator on its own. Finally, a TF-IDF analysis of the clusters surfaces interpretable patterns of behaviors that simulators capture, miss, and hallucinate.
Show more
View full details
Poster

SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

Xiangyi Li ⋅ Yimin Liu ⋅ Wenbo Chen ⋅ Shenghan Zheng ⋅ Yifeng He ⋅ Xiaokun Chen ⋅ Bingran You ⋅ Haotian Shen ⋅ Yubo Li ⋅ Kyoung Whan Choe ⋅ Binxu Li ⋅ Shuyi Wang ⋅ Chris Kong ⋅ Xinyi Liu ⋅ Jiankai Sun ⋅ Runhui Wang ⋅ Xuanqing Liu ⋅ Jiachen Li ⋅ Xin Lan ⋅ Yuanli Wang ⋅ Xuandong Zhao ⋅ Yueqian Lin ⋅ Wengao Ye ⋅ Junwei He ⋅ Di Wang ⋅ Roey B Chaim ⋅ Songlin Li ⋅ Hanwen Xing ⋅ Yue Zhang ⋅ Yipeng Gao ⋅ Yijiang Li ⋅ Ze Ma ⋅ Liqiang Jing ⋅ Qunhong Zeng ⋅ Tianyu Wang ⋅ Kaixin Li ⋅ Yiqi Xue ⋅ Zonglin Di ⋅ Yizhuo He ⋅ Yuchen Tian ⋅ Yixuan Gao ⋅ Hejia Geng ⋅ Bo Chen ⋅ Litong Liu ⋅ Sikai Cheng ⋅ Zelin Tan ⋅ Jiajun Bao ⋅ Junwei Zhou ⋅ Shuaicheng Tong ⋅ Shuwen Xu ⋅ Penghao Jiang ⋅ Terry Yue Zhuo ⋅ Tinghan Ye ⋅ Chujun Tao ⋅ Qi Qi ⋅ Miao Li ⋅ Longtai Liao ⋅ Zhikang Dong ⋅ Chang Shi ⋅ Steven Dillmann ⋅ Han-chung Lee
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1113
Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a paired benchmark of 84 tasks across 8 expertise-heavy domains; every task runs under three matched conditions (no Skills, curated Skills, self-generated Skills) on 7 agent--model configurations and 7,326 trajectories with deterministic verifiers. Curated Skills raise pass rate by 16.4\,pp on average, but the lift is heterogeneous: per-domain $\Delta$ spans +7.2 to +24.8pp and 18 of 84 tasks regress. Self-generated Skills net -1.3pp, indicating that current models cannot author the procedural knowledge they benefit from consuming. SkillsBench establishes paired evaluation as standard practice for measuring agent augmentations.
Show more
View full details
Poster

A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts

Claudio César Claros-Olivares ⋅ Austin Brockmeier
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A306
We present a systematic benchmark of out-of-distribution (OOD) detection CSFs through a representation-centric lens. Our study spans CNN and ViT backbones, multiple training paradigms, four image-classification source datasets (CIFAR-10, CIFAR-100, SuperCIFAR-100, and TinyImageNet), and OOD datasets grouped into near, mid, and far regimes using CLIP-derived semantic distances. To compare CSFs across these settings, we employ a multiple-comparison-controlled rank pipeline that identifies top cliques of statistically indistinguishable winners under threshold-free ranking metrics (AURC and AUGRC). The main empirical finding is that the competitive detector family depends more on the learned representation than on score design alone. For both CNNs and ViTs, simple probabilistic scores dominate misclassification detection. On CNNs, margin-based scores are strongest in near-OOD regimes, while geometry-aware scores such as NNGuide, fDBD, and CTM become more competitive as shift severity increases. On fine-tuned ViTs, the top cliques are led mainly by reconstruction- and residual-based scores. To interpret these ranking shifts, we analyze the last-layer representation using Neural Collapse (NC) metrics. The resulting picture is consistent across architectures: prototype- and boundary-aware scores become stronger when the representation is more collapsed and better aligned with classifier weights, whereas weaker-collapse regimes favor gradient- and manifold-based scores. Building on these insights, we propose two contributions: a simple PCA-based projection-filtering procedure that improves detector performance, and an approach that uses NC measurements computed from a trained classifier to predict its competitive out-of-distribution detector shortlist, without requiring any additional OOD data.
Show more
View full details
Poster

Stress-Testing Neural Network Verifiers with Provably Robust Instances

David Troxell ⋅ Yulia Alexandr ⋅ Sofia Hunt ⋅ Stephanie Lei ⋅ Guido Montufar
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A307
Neural network verifiers aim to provide formal guarantees on model behavior, but existing verification benchmarks are fundamentally limited by their lack of ground-truth labels. As a result, verifier evaluation relies on indirect heuristics, which prevents exact scoring and systematic studies of verifier failure modes. We address this gap by introducing a reusable framework for generating verification instances whose ground-truth robustness labels are known a priori through analytic construction. Our framework led to the discovery of multiple numeric tolerance concerns and implementation bugs in popular verifiers, highlighting the need for ground-truth labels. Additionally, to systematically study verifier failure modes, we introduce the verification Difficulty Profile, a collection of estimable quantities capturing distinct sources of instance hardness. Using our framework and these profiles, we evaluate five state-of-the-art verifiers and show that different instances stress distinct aspects of the verification pipeline. We show that these results can aid the future development of verifiers as they provide actionable targets for improving numerical reliability, relaxation quality, and search behavior. Our code is publicly available: https://github.com/NeurIPSAnon12345/VeriStressGT.git.
Show more
View full details
Poster

Bottleneck Benchmark: Sequence Compression Under Controlled Difficulty and Fixed Latent Size

Emilio Munoz ⋅ Felix Mohr
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A314
One key application of autoencoders is to learn how to compress token sequences. Since the difficulty of compression depends on factors such as sequence length, vocabulary size, structural regularity, and input noise, it would be valuable to understand how these factors affect the performance of different architectures. Surprisingly, however, there is no benchmark for comparing autoencoder architectures that allows control over these aspects of difficulty. We introduce the Bottleneck Benchmark (BB), an open experiment framework in which encoder and decoder communicate only through a fixed-size latent and that allows to configure the previously mentioned difficulty aspects for any customizable data generator. As a case study, we evaluate BiLSTM, Transformer, and Bidirectional SSM encoders on copy, sort, and set tasks, finding that performance differences arise from inductive biases under noise and length rather than raw capacity.
Show more
View full details
Poster

Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation

Garvin Kruthof
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A205
When researchers iteratively refine ideas with large language models, do the models preserve fidelity to the original objective? We introduce DriftBench, a benchmark for evaluating constraint adherence in multi-turn LLM-assisted scientific ideation. Across 2,146 scored benchmark runs spanning seven models from five providers (including two open-weight), four interaction conditions, and 38 research briefs from 24 scientific domains, we find that iterative pressure reliably increases structural complexity and often reduces adherence to original constraints. A restatement probe reveals a dissociation between declarative recall and behavioral adherence, as models accurately restate constraints they simultaneously violate. The knows-but-violates (KBV) rate, measuring constraint non-compliance despite preserved recall, ranges from 8\% to 99\% across models. Structured checkpointing partially reduces KBV rates but does not close the dissociation, and complexity inflation persists. Human validation against blind raters confirms that the LLM judge under-detects constraint violations, making reported constraint adherence scores conservative. Sensitivity analyses confirm the findings are robust to temperature (0.7 vs.\ 1.0) and pressure type (novelty vs.\ rigor). We release all briefs, prompts, rubrics, transcripts, and scores as an open benchmark.
Show more
View full details
Poster

Evaluating Compositional Generalization in Transformers: The Role of Composition Equivalence and Module Coverage

Purva Pruthi ⋅ Andrew Yuan ⋅ Alexander D'Amour ⋅ David Jensen
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A100
Compositional generalization is an important form of out-of-distribution generalization, defined as the ability to train on some combinations of modules (e.g., functions such as `sort`, `filter`) and then generalize to unseen combinations. Despite substantial empirical work evaluating compositional generalization in transformers, findings have been inconsistent because evaluations were often conducted on a limited set of train-test splits, defined in an ad hoc manner. Through a systematic evaluation, we identify two key aspects of the train-test distribution that strongly affect performance: **composition equivalence** and **module coverage**. We show that the apparent performance of direct models (trained only on final outputs) results primarily from exploiting composition equivalences—different sequences of modules that reduce to identical end-to-end functions. When benchmarks eliminate these equivalences, performance drops significantly, showing their inability to generalize to compositions of known modules that produce novel end-to-end functions. We further demonstrate that performance degrades monotonically as module coverage shifts increase—when modules appear in different contexts between train and test—affecting both direct and step-by-step models (trained on intermediate and final outputs). These findings have concrete implications for benchmark design: benchmarks that include compositional equivalences can lead to significant overestimation of the generalization abilities of direct models, and benchmarks that include both module coverage violations and compositional equivalences can exacerbate shortcut learning. We further demonstrate the effect of compositional equivalence on the generalization performance of the pre-trained model Llama3-8B on the code-output prediction task in the coding benchmark [LiveCodeBench](https://livecodebench.github.io/) and discuss its real-world implications.
Show more
View full details
Poster

Trimming the Long-Tail of Visual World Modeling Evaluation

Bingxuan Li ⋅ Yining Hong ⋅ Cheng Qian ⋅ Hyeonjeong Ha ⋅ Jiateng Liu ⋅ Zhenhailong Wang ⋅ Yue Guo ⋅ Yunzhu Li ⋅ Heng Ji
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1114
Physical interactions follow a long-tailed distribution: a set of common and regular interactions dominates human experience and visual data, while a broad spectrum of rare and irregular interactions remains underrepresented. Although recent world models achieve impressive realism on existing benchmarks, they primarily focus on simulating common physical interactions. This raises a central question: Do world models internalize and generalize the physical principles? In this work, we introduce Tailor-Bench, a benchmark to systemically evaluate this question through the lens of visual world modeling on tool-using interactions. Our benchmarks features three scenario modes (regular, unconventional and impossible), and two settings (predictive and descriptive) under a unified evaluation protocol. Our experimental results reveal a clear long-tail gap in visual world modeling: Performance consistently degrades from Regular to Unconventional and Impossible scenarios, indicating limited generalization beyond common interactions. Failure analysis further shows that models rely on superficial visual patterns, where image models fail to realize correct state changes, and video models additionally suffer from temporal inconsistencies. These limitations persist even with explicit outcome descriptions: Models often revert to familiar patterns in descriptive settings and fail to infer correct outcomes in predictive settings. By exposing systematic failures under long-tail scenarios, Tailor-Bench highlights the need for world models that understand object attributes, causal dynamics, and constraint-aware interactions. We further introduce Tailor-Evaluator-8b, a specialized auto-evaluator trained with structured reasoning supervision, which achieves 0.88 human correlation for scalable evaluation of long-tail visual world modeling.
Show more
View full details
Poster

DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

Yuxuan Jiang ⋅ Chenglin Liu ⋅ John Rose ⋅ Mason Miller ⋅ Ruthesh Thavamani ⋅ Juntao Wu ⋅ Yinwei Huang ⋅ Jiahao Xu ⋅ Peng Huang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A114
LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents’ ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope.
Show more
View full details
Poster

K9-Bench: Evaluating Multimodal LLMs on Canine-Centric Videos

Khush Attarde ⋅ Yusuf Ali ⋅ Megha Thukral ⋅ Divye Bhutani ⋅ Thomas Ploetz ⋅ Zsolt Kira
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A1115
Multimodal Large Language Models (MLLMs) have demonstrated remarkable zero-shot capabilities across diverse inputs such as images, video, audio, and text. A crucial, yet underexplored, application of these models lies in understanding and modeling animal-centric scenarios, especially domesticated animals. As animals are integral to millions of households, benchmarking next-generation AI models on pet-focused tasks, ranging from recognizing distress signals in pets to enabling responsive robotic companions, is essential for building AI systems that can live and work alongside us. We introduce K9-Bench, a novel benchmark focused on real-world videos of domestic dogs, specifically targeting canine action and interaction understanding via $\approx$5000 question-answer pairs across 907 videos spanning 5 distinct task categories that test long-form, canine-centric multimodal reasoning in MLLMs. To create this dataset, we propose a scalable, VLM/LLM-powered data generation pipeline that automatically mines canine-centric videos from open web sources and curates QA pairs requiring fine-grained, multi-hop reasoning over canine actions and temporally extended interaction sequences. We further propose bias mitigation strategies designed to eliminate biases introduced by VLMs during dataset curation. Through extensive experimentation, we find that frontier MLLMs exhibit limited zero-shot performance on canine-centric tasks: although state-of-the-art closed-source models outperform open-source counterparts, they still struggle with compositional reasoning over subtle posture and interaction cues spread over long horizons. We further observe that generic chain-of-thought prompting provides only modest performance for such long-horizon reasoning. We also conduct human evaluations and checks on a subset to validate the overall dataset quality. Beyond a novel dataset for canine activity analysis, K9-Bench provides a general-purpose dataset construction pipeline that can be adapted to other low-data domains for quantitative analysis. Our dataset and evaluation suite will be made publicly available.
Show more
View full details
Poster

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

Zhenyu Zhao ⋅ Xinrui Ruan ⋅ Waverly Wei ⋅ Yueshan Zhang ⋅ Zeyu Zheng ⋅ Sui Huang ⋅ Jingshen Wang
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A115
Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold'' labels are expensive to collect and limited in quantity. Organizations thus often turn to collecting vast but noisy "silver'' labels from crowdsourced workers or vendor annotators as proxies for gold labels. Because gold remains the evaluation target, naively aggregating noisy silver labels may introduce bias, and estimators built on sparsely observed gold labels may have high variance to resolve the model performance gaps that guide practical decisions. Model evaluation has become an ongoing operational practice rather than a one-time exercise, with evaluation rounds repeating across model versions, releases, and content domains. A natural question is whether the previous historical evaluation data can be used to improve each new round of evaluation. We introduce HERO (History Enhanced RObust model evaluation), a novel framework that uses historical data to reduce both bias and variance in model performance evaluation. HERO calibrates silver labelers' performance learned from historical gold annotations, and stabilizes the resulting estimator by anchoring it to covariate information measured with high precision in the historical data. HERO can be broadly applied across multiple common evaluation tasks, and remains valid when only a subset of historical labelers appears in the current round. We establish conditions under which the bias and variance reductions hold, showcase HERO's performance in simulation studies, and demonstrate its effectiveness on real-world model evaluation benchmarking datasets.
Show more
View full details
Poster

RoboWits: Unexpected Challenges for Robotic Creative Problem Solving

Chunru Lin ⋅ Hongxin Zhang ⋅ Fenghao Yu ⋅ Zhehuan Chen ⋅ Tom Griffiths ⋅ Yejin Choi ⋅ David Held ⋅ Chuang Gan
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A920
The ability to reason, adapt, and creatively solve problems under unexpected challenges is essential for robots operating in real-world environments. However, current robotic benchmarks primarily emphasize skill-level execution and provide limited insight into such cognitive reasoning capabilities. We introduce RoboWits, a bi-manual robotic benchmark designed to systematically evaluate cognitive reasoning, creative tool use, and robustness to unexpected conditions. To enable scalable construction of high-quality reasoning-centric unexpected scenarios, we propose an automated task generation pipeline formulated as a multi-agent cooperative framework, comprising agents for seed task generation and verification, metric generation, scene generation, and task mutation. Using the pipeline, we curated 30 diverse seed tasks and 208 tasks with mutations and graded difficulty across geometry, material, and assembly-based reasoning. We benchmark popular robot policies, pre-trained VLAs, and oracle-state planners. Our results reveal a significant performance gap: while pre-trained VLAs exhibit preliminary success on seed tasks after single-task fine-tuning, they struggle to perform on mutated tasks, implying their brittleness in manipulation tasks requiring reasoning, strategy adaptation, and robustness to deceptive or constrained environments.
Show more
View full details
Poster

Beyond Task Success: Probing Cognitive Primitives in Web Agents

Xunjian Yin ⋅ Tianchen Guan ⋅ Jinao Wang ⋅ Weili Cao ⋅ Royce Cheng-Yue ⋅ Daisy Lin ⋅ Keagan Long ⋅ Kyle Wong ⋅ Bhuwan Dhingra ⋅ XJ Wang ⋅ Shuyan Zhou
Dec 11, 2:00 AM - 5:00 AM Hall C1 #A120
Cognitive primitives such as planning, exploration, and backtracking are widely regarded as core to competent web agents. Yet existing studies are largely retrospective: agents are evaluated on domain-based tasks, and trajectories are inspected post hoc for evidence of these behaviors. Because such tasks are not designed to require specific primitives, observed behaviors do not reliably reflect true capability demands. This raises a key question: Can we design realistic tasks where success provably depends on a chosen cognitive primitive? WEBSTRESS addresses this by casting capability evaluation as a controlled comparison. For each task, we construct a paired variant in which a target capability becomes necessary (e.g., transforming a direct path into an obstructed one requiring exploration). The performance gap between the pair measures the agent’s reliance on that capability. Across seven environments and seven capabilities, this yields 519 paired tasks. Evaluating six frontier agents reveals consistent weaknesses: all suffer 18-28\% drops on capability-demanding variants, with 75\% of failures being belief failures, where agents incorrectly declare success. In contrast, humans exhibit only a 5.7\% performance drop under the same paired evaluation. These results highlight that, despite strong overall performance, substantial gaps remain between human and agent cognitive behavior, and that our approach provides a principled way to measure them.
Show more
View full details
Poster

The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks

Shuo Wen ⋅ Beixi Du ⋅ Edwin Meriaux ⋅ Junming Shi ⋅ Chloe Si ⋅ David Meger ⋅ Doina Precup ⋅ Gregory Dudek
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A715
With the rapid development of Vision-Language Models (VLMs), it is increasingly critical to evaluate their ability in accurate and robust reasoning based solely on visual information. However, existing video understanding benchmarks often overlook the deep cognitive processes inherent in visual reasoning and frequently rely on multiple-choice formats, implicitly encouraging reasoning shortcuts. To address these limitations, we introduce \emph{The Prestige}, a novel video understanding benchmark designed to isolate VLM reasoning under deliberate cognitive misdirection through professional magic performances. The dataset consists of a curated collection of high-quality magic videos paired with a diverse distribution of strictly open-ended questions targeting causal state tracking, paradoxical reasoning, and resistance to adversarial questions. These questions need the models to perform long-horizon, multi-step commonsense reasoning with cognition. Through extensive evaluations of leading open-weight and proprietary VLMs, using a combination of machine and human grading, we find a substantial gap between model and human performance. \emph{The Prestige} establishes a rigorous new frontier for evaluating temporal, causal, and cognitive-flow reasoning in multimodal foundation models.
Show more
View full details
Poster

DEBATE: A Large-Scale Benchmark for Evaluating Opinion Dynamics in Role-Playing LLM Agents

Yun-Shiuan Chuang ⋅ Ruixuan Tu ⋅ Chengtao Dai ⋅ You Li ⋅ Binwei Yao ⋅ Michael Tessler ⋅ Sijia Yang ⋅ Dhavan Shah ⋅ Robert Hawkins ⋅ Junjie Hu ⋅ Timothy T Rogers
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A1027
Modeling opinion change through social interactions is important for understanding polarization, misinformation, and societal conflict. Recent work simulates opinion dynamics with role-playing LLM agents (RPLAs), but multi-agent simulations often display unnatural group behavior, such as premature convergence, and lack empirical benchmarks for assessing alignment with real human group interactions. We introduce DEBATE, a benchmark for evaluating the authenticity of opinion dynamics in multi-agent RPLA simulations. DEBATE contains multi-round public messages and private Likert-scale beliefs from U.S.-based participants across 107 topics; the cleaned benchmark used in our experiments contains 2,788 participants in 697 groups, enabling evaluation at the utterance and group levels. We instantiate “digital twin” RPLAs with seven LLMs and evaluate across two settings: next-message prediction and full dynamics simulation, using stance-based opinion-dynamics metrics. In zero-shot settings, RPLA groups exhibit strong opinion convergence relative to human groups. On the held-out group split, supervised fine-tuning (SFT) for Llama-3.1-8B-Instruct improves auxiliary stance alignment and reduces group-level convergence error, though discrepancies in opinion change and belief updating remain. DEBATE provides a benchmark for simulated opinion dynamics; reviewer-accessible code and dataset URLs are provided at submission.
Show more
View full details
Poster

SarcBench: A Bilingual Benchmark for Contextual Sarcasm Understanding, Response, and Generation

Dihong Huang ⋅ Zhuoyue Chang ⋅ Yuhao Liu ⋅ Zhouting Mo ⋅ Jianxing Yu ⋅ Wenqing Chen ⋅ Jingping Liu
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A827
Sarcasm understanding and response are important for large language models (LLMs) to engage naturally in real-world conversations. However, existing sarcasm benchmarks primarily focus on detection or related subtasks in isolation, lacking diagnostic power and failing to reflect the complexity of real conversational interactions. To address this gap, we introduce SarcBench, a bilingual benchmark with 30,083 aligned samples. It defines three interconnected tasks, Intent Recognition, Sarcasm Response, and Sarcasm Generation, enabling diagnostic evaluation of sarcasm understanding and conversational behavior. We evaluate 12 contemporary language models on SarcBench and find that even the top-performing model achieves only 61.21%. We also identify a clear understanding–action gap: models are consistently better at recovering intended meanings than at producing socially appropriate responses or generating sarcasm that faithfully realizes a specified rhetorical mechanism. In addition, performance varies substantially across models and task types, and controlled sarcasm generation often collapses rhetorical diversity into a narrow set of default templates. Taken together, these findings indicate that current models remain limited in handling sarcasm as an interactive social behavior rather than merely a recognition task.
Show more
View full details
Poster

SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics

Yicheng Zhu ⋅ Yang Chen ⋅ Tao Li ⋅ Zilin Bian
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A601
Autonomous-driving simulators typically trade physical fidelity for scalable parallelism. Physics-based platforms such as CARLA and MetaDrive provide articulated vehicle dynamics and contact, but their non-vectorized control interfaces make large batched training difficult. GPU-batched systems such as Waymax and GPUDrive scale to hundreds of scenarios by replacing rigid-body physics with simplified kinematics models, omitting tire--road interaction, suspension, contact dynamics, and road-condition-dependent friction. We introduce SceneFactory, a GPU-vectorized platform for procedural scene construction, physics-based multi-agent simulation, and reinforcement learning in autonomous driving environments. Built on NVIDIA Isaac Sim and Isaac Lab, SceneFactory represents worlds and agents as batched tensors: vehicle control, observations, rewards, resets, and policy inference are executed as GPU tensor operations over the Isaac Lab tensor API. SceneFactory converts Waymo Open Motion Dataset road topologies into simulation-ready USD (Universal Scene Description) worlds. SceneFactory runs many worlds concurrently on one GPU, populates each with multiple articulated PhysX vehicles, and maps precipitation and road-surface type to PhysX material friction coefficients. Thanks to the GPU vectorization, SceneFactory achieves up to 127$\times$ higher throughput than a non-vectorized PhysX baseline on the same GPU and physics solver, reaching 19,250 controlled-agent simulation steps per second (CASPS) at 256 worlds $\times$ 16 agents. Cross-simulator transfer reveals an asymmetric dynamics gap: physics-grounded RL driving policies transfer to a simplified kinematic bicycle model with 99.5\% success, whereas the reverse transfer success rate drops to 47.3\%. Under wet-road friction, friction-aware policies reduce mean peak deceleration rate to avoid crash (DRAC) from 58.7 to 27.8\,m/s$^2$ without sacrificing goal reach. SceneFactory shows that scalable autonomous-driving training need not discard articulated rigid-body dynamics or physically grounded road-condition variation.
Show more
View full details
Poster

More Is Not More: What Matters for Diversity in LLM Opinions?

Qiyang Yao
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A825
Large language models are increasingly used to simulate diverse human opinions in open-ended tasks such as synthetic surveys, focus group modeling, and public opinion prediction. However, LLM outputs exhibit systematic opinion homogenization. Practitioners have explored various interventions to increase diversity, but the landscape remains fragmented: different methods are evaluated in isolation with incomparable metrics, and in practice they are typically deployed and upgraded simultaneously, making it difficult to attribute gains to specific components. To advance a more scientific understanding of LLM output diversity, we design a factorial experiment that separates two primary intervention dimensions: input conditioning (operationalized through persona depth) and interaction architecture. We evaluate all conditions on 100 real-user open-ended questions across 7 models, measuring diversity with multiple complementary metrics. Our findings challenge several common assumptions. First, more persona detail does not monotonically increase diversity. The initial step of persona conditioning already captures the majority of the gain, while further elaboration with demographic detail does not consistently improve and can reduce diversity on some models. Second, rather than seeking a single best interaction architecture, we find that different architectures explore largely non-overlapping opinion regions. Combining multiple architectures yields broader coverage than optimizing any one. Third, commonly attempted low-cost alternatives such as raising sampling temperature and adding diversity instructions produce negligible effects compared to structured interventions. Overall, our work demonstrates that diversity is not a product of scaling along any single dimension, but is highly sensitive to the structural form and combination of interventions. The field needs to move toward empirically grounded diversity strategy design over assumption-driven exploration.
Show more
View full details
Poster

\$OneMillion-Bench: How Far are Language Agents from Human Experts?

Michael Liu ⋅ Jiaqi Li ⋅ Jun Bai ⋅ Qianyu Yang ⋅ Xiaobo Hu ⋅ Tao Peng ⋅ Zaiyuan Wang ⋅ Jiayun Dong ⋅ Chun Zhang ⋅ Zixia Jia ⋅ Kaiyuan Chen ⋅ Yanglihong Xiao ⋅ Lingyue Yin ⋅ Tiliang Duan ⋅ Ge Zhang ⋅ Gang Yao ⋅ Hao Chen ⋅ Jianpeng Jiao ⋅ Zilong Zheng
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A117
As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks remain largely confined to structured or exam-style tasks that fall short of real-world professional demands. To this end, we introduce \\$OneMillion-Bench (\\$OMB), a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios. Unlike prior work, the benchmark requires retrieving authoritative sources, resolving conflicting evidence, applying domain-specific rules, and making constraint decisions, where correctness depends as much on the reasoning process as the final answer. We adopt a rubric-based evaluation protocol scoring factual accuracy, logical coherence, practical feasibility, and professional compliance, focusing on expert-level problems to ensure meaningful differentiation across agents. Together, \\$OMB provides a unified testbed for assessing agentic reliability, professional depth, and an indicator of practical readiness in domain-intensive scenarios.
Show more
View full details
Poster

VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation

Zichen Xie ⋅ Mrigank Pawagi ⋅ Yuxin Liu ⋅ Aaditi Rai ⋅ Lize Shao ⋅ John Berberian ⋅ Sicong Che ⋅ Wenxi Wang
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A116
Large language models can now generate useful code from natural language, but their outputs still come without correctness guarantees. Verifiable code generation offers a path beyond testing by requiring models to produce not only executable code, but also formal specifications and machine-checkable proofs. Progress in this direction, however, is difficult to measure: existing benchmarks are often small, focus on only one part of the pipeline, lack ground-truth proofs or rigorous specification validation, or target verification settings far from mainstream software development. We present VeriContest, a benchmark of 946 competitive-programming problems from LeetCode and Codeforces for verifiable code generation in Rust with Verus. Each problem pairs a natural language description with expert-validated formal specifications, judge-accepted Rust code, Verus-checked proofs, and positive and negative test suites. VeriContest is constructed through a three-phase pipeline that scales from manually verified seed problems to semi-automated expansion with human-in-the-loop review. To further strengthen benchmark quality, we use testing as an additional quality-assurance layer for validating postcondition completeness. VeriContest supports both isolated and compositional evaluation of specification generation, code generation, proof generation, and end-to-end verified program synthesis. Evaluating ten state-of-the-art models reveals a sharp gap between ordinary coding ability and verifiable code generation: the strongest model reaches 92.18% on natural-language-to-code generation, but only 48.31% on specification generation, 13.95% on proof generation, and 5.29% end-to-end. These results identify proof and specification generation as the central bottlenecks for current models and establish VeriContest as a rigorous platform for measuring and training future systems that generate code with machine-checkable correctness.
Show more
View full details
Poster

The Complexity Kink: LLM Rubric Instruments for Causal Inference on Code Generation Reliability

Michael Hernandez ⋅ Tian Zhao
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A115
Measuring complexity from a model's own code is biased by failure: a hard prompt can produce a short, broken program that looks simple. We instead score each prompt's structure on six dimensions from the prompt alone, never from generated code or test results. We draw 5,000 Python prompts across six bands of a preliminary single-rater score, and four out-of-panel LLM judges rescore every prompt (19,997 scores; composite ICC(2,1) = 0.872 on the 4,998 prompts with all four). Twenty-one models answer every prompt (105,000 generations), and an independent LLM auditor, checked against known-answer cases, reviews each unit-test verdict. Averaged over models, pass rate has a nonmonotone breakpoint at composite $\hat{\gamma}=14.0$: 83.4\% at or below, 90.1\% above. That jump is not a law of difficulty. With task-type fixed effects the breakpoint stays at 14.0 but the adjusted jump shrinks from 6.8 to 0.4 points, and neither of the benchmark's two prompt sources reproduces the jump on its own. Nineteen models pass more often above their own breakpoint and two less often, and one of those two declines, Gemini 3.1 Pro Preview's, comes from 156 API calls that returned nothing. In post hoc follow-up, Kimi K2.5 shows the clearest decline: with a prompt-source control, and within the later source alone, its break is at 14.25, where its later-source pass rate falls from 0.848 to 0.808 (95\% interval $[14.00,14.75]$). A 365-prompt audit-clean extension matches the original raw test outcomes at bins 15 and 16 but adds only 14 prompts above bin 16. Among failed generations with measurable output complexity, 25.0\% pair a prompt composite above 8 with output complexity of at most 10, twice the share among passing ones. The pooled breakpoint moves to 11.25 on raw test outcomes and 13.75 under the partial audit used at submission. Human agreement with the index is moderate and rater-dependent, paraphrase and cross-language rescoring preserve its ordering, and overidentification tests reject the six dimensions as instruments, so we treat the composite as an index and make no causal claim. The contribution is a prompt-side way to locate where each model's pass rate changes and to test how much that depends on the prompt mix and on how success is scored.
Show more
View full details
Poster

Self Driving Datasets: From 20 Million Papers to Nuanced Biomedical Knowledge at Scale

Haydn Jones ⋅ Yimeng Zeng ⋅ Alden Rose ⋅ Yifei Li ⋅ Yining Huang ⋅ Kaiwen Wu ⋅ Jiaming Liang ⋅ Maggie Huan ⋅ Yoseph Barash ⋅ Cesar de la Fuente-Nunez ⋅ Osbert Bastani ⋅ Zachary Ives ⋅ Mark Yatskar ⋅ Jacob Gardner
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A701
Manually curated biomedical repositories---spanning bioactivity, genomics, and chemistry---are expensive to maintain, lag behind primary literature, and often discard experimental context. The absence of contextual information obscures critical nuances, thereby complicating the assessment of data correctness and coverage, necessary criteria for building high-quality models. We show that PubMed itself can be turned into structured datasets---autonomously and cost-effectively---that are larger, more nuanced, and more accurate than the curated databases they would replace. We present three coupled contributions: (1) an LLM-based entity-tagging pipeline, grounded in nine biomedical ontologies, that tags 4.5 billion entities across 19 categories in a 22.5M-paper, 2.5-trillion-token PubMed corpus; (2) hybrid sparse–dense retrieval infrastructure supporting surgical entity-filtered semantic queries over the tagged corpus; and (3) Starling, a multi-agent deep research system that, given only a natural-language task description, autonomously designs precision- and recall-targeted retrieval filters, induces an extraction schema, and emits structured records with nuance-rich fields and supporting passages. Applied to six tasks---blood-brain barrier permeability, oral bioavailability, acute toxicity (LD50), gene-disease associations, protein subcellular localization, and chemical reactions---Starling produces ~7.7M records (per-task scale ranges from 131K to 3M); several of these are, to our knowledge, the largest public datasets for their respective properties. Frontier-model rejection of our kept extractions is 0.6–7.7% across tasks, surprisingly far below the error rates we measure on the widely used, manually curated counterparts (e.g., 16.5% on BBB_Martins, 7.3% on Bioavailability_Ma). Beyond scale and accuracy, the attached supporting passages carry nuance that tabular databases discard: for example, oral bioavailability of a molecule might depend on whether the patient is fed or fasted. Together, the corpus, retrieval layer, and agent establish a foundation for multimodal predictive and generative models in AI-driven therapeutic design. We release code and links to our datasets at https://anonymous.4open.science/r/starling-B026/.
Show more
View full details
Poster

MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing

Nishant Bhargava ⋅ Rodrigo S Barrento
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A321
Long-term LLM agents must decide what to write into persistent memory before future queries are known, yet existing evaluations measure only final QA accuracy, entangling memory writing with retrieval and reasoning. We introduce MEMAUDIT, a package-oracle evaluation protocol that turns memory writing into a finite, auditable optimization problem with a certified ground-truth optimum under a fixed storage budget. Each evaluation package specifies an experience stream, candidate memory representations, and future-query requirements, enabling exact measurement of how well a memory writer preserves task-relevant information. We instantiate MEMAUDIT with a semantic coverage objective under storage and exclusivity constraints and compute exact optima via certified optimization. Across controlled and naturalistic settings, MEMAUDIT disentangles representation quality, validity preservation, and budget-aware selection effects that end-to-end QA cannot isolate. This enables, for the first time, principled and reproducible evaluation of memory-writing policies independent of downstream retrieval and reasoning.
Show more
View full details
Poster

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

Jingbo Yang ⋅ Henry Lai ⋅ Xiaowen Wang ⋅ Shiyu Chang ⋅ Yaar Harari ⋅ Evgeniy Gabrilovich
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A826
Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.
Show more
View full details
Poster

Offline Preference-Based Trajectory Evaluation

Fernando Diaz
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A612
Offline evaluation of agentic systems often collapses trajectories to terminal success, discarding information about partial progress and inducing widespread ties. We argue that this trajectory collapse creates substantial statistical inefficiency by reducing effective sample size and weakening the ability to distinguish systems. We propose preference-based trajectory evaluation, which compares trajectories directly through temporal preferences over progress and time-to-return profiles. Across diverse agentic and interactive benchmarks, standard success-based metrics produce tied comparisons on roughly 75\% of instances, whereas trajectory-aware preferences reduce ties to roughly 30\%, improving discriminative power, ranking stability, and data efficiency. Our results suggest that benchmark saturation, often portrayed as the result of poor data collection or problem difficulty, may also be explained by the choice of evaluation measure.
Show more
View full details
Poster

AnyHand: A Large-Scale Synthetic Dataset for RGB(-D) Hand Pose Estimation

Chen Si ⋅ Yulin Liu ⋅ Bo Ai ⋅ Jianwen Xie ⋅ Rolandos Alexandros Potamias ⋅ Chuanxia Zheng ⋅ Hao Su
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A706
We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation. While recent works with foundation approaches have shown that scaling training data markedly improves hand pose estimation, existing real-world datasets are limited in coverage, and prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our proposed AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich geometric annotations. We show that extending the original training data recipes of existing RGB baselines with AnyHand yields significant gains on multiple benchmarks (FreiHAND and HO-3D), even when keeping the architectures and training schemes fixed. Together with extensive ablations on the scale and composition of the training data setups, these results suggest that training data diversity and quality are as critical as scale for advancing hand pose estimation. We further examine the utility of AnyHand's aligned depth maps in the appendix, showing that scaling RGB-D supervision with AnyHand allows a lightweight depth-fusion variant of existing RGB baselines to outperform prior RGB-D methods.
Show more
View full details
Poster

MammoGPS: A Benchmark for Visual Grounding, Perception, and Spatial Reasoning in Mammography

Yen Nhi Truong Vu ⋅ Dan Guo ⋅ Sripad Joshi ⋅ Harshit Kumar ⋅ Jason Su ⋅ Thomas P Matthews
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A811
Vision-language models (VLMs) can achieve strong image-level visual question-answer (QA) performance while relying on shortcuts rather than grounded visual understanding. This is especially concerning in medical imaging, where clinically meaningful interpretation often requires spatial reasoning, localization, and anatomical understanding. Yet in mammography, no benchmark directly assesses whether VLMs can localize findings or reason spatially. To address this, we introduce MammoGPS, a benchmark for evaluating visual grounding, perception, and spatial reasoning in mammography. MammoGPS comprises 56,638 QA pairs across 13 clinical tasks for anatomical and finding grounding, finding-type, quadrant prediction, and depth determination. MammoGPS also includes 94,119 QA pairs across 20 controlled diagnostic tasks designed to probe model failures and spatial biases: we design controlled difficulty ladders to isolate the effects of semantic cues, visual cues, clinical terminology, and background context, as well as spatial-bias tasks to test whether model performance depends on finding location. We evaluate 12 VLMs spanning proprietary, general-purpose, and medical-domain models. Our analysis shows that current systems struggle to localize suspicious findings, make limited use of spatial semantic cues, rely on priors and shortcut heuristics for categorical and grounding tasks, struggle with mammography-specific spatial terminology, and exhibit position-dependent biases. Together, our results underscore the need for more rigorous and fine-grained mammography VLM evaluations.
Show more
View full details
Poster

ShadowBench: Exposing Lexical Anchoring and the Illusion of Forgetting in Large Language Models

Sujan Maharjan ⋅ Lu
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A824
Large Language Models (LLMs) have emerged as primary interfaces for factual retrieval, yet current evaluation paradigms rely almost exclusively on explicit entity names. In this work, we demonstrate that LLM knowledge is fundamentally lexically anchored: models possess extensive factual information but find it significantly difficult to retrieve without explicit name tokens. To quantify this limitation, we introduce ShadowBench, a rigorously hardened, shortcut-resistant benchmark designed to evaluate Latent Entity Association via a novel Dual-Trait Association (DTA) task. Our evaluation reveals a pervasive "Shadow Gap" across all model scales – up to the frontier models GPT-5.4 and Claude-Sonnet-4.6 – where removing lexical anchors causes performance drops of over 20%. Furthermore, we demonstrate that this lexical dependency exposes a critical vulnerability in AI safety. Applying ShadowBench to Machine Unlearning, we find that state-of-the-art algorithms (e.g., Gradient Difference, NPO) achieve only superficial lexical erasure. While unlearned models fail on direct queries, our novel Latent Entity Leakage Rate (LELR) metric reveals that reasoning models explicitly reconstruct the "forgotten" entity in their internal reasoning traces in over 89% of cases, utilizing residual shadow knowledge to solve associative tasks. Ultimately, ShadowBench proves that current unlearning paradigms create an "Illusion of Forgetting," demonstrating that existing methods act as superficial output filters and are not yet reliable for achieving true parametric erasure.
Show more
View full details
Poster

CLAMP: A Sim-to-Real Benchmark for Closed-Loop Kinematic Pose Estimation and Assembly Reasoning

Kevin Murray ⋅ Randolph B Robert ⋅ Petar Z Duric ⋅ Zoran Duric
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A921
Existing benchmarks for known-object pose estimation either treat objects as rigid bodies or restrict articulation to serial-chain robot arms with revolute joints, and all assume that every part is always present. Real-world industrial and consumer equipment, however, exhibits closed-loop kinematic chains, prismatic joints, and structural assembly perturbations—challenges that current datasets and methods do not address. We introduce CLAMP (Closed-Loop Assembly and Mechanism Perception), a sim-to-real benchmark for kinematic pose estimation and assembly reasoning on mechanically complex known objects. The dataset spans seven diverse pieces of equipment—ranging from a delta 3D printer with 27 coupled DOFs to a hydraulic jack with closed kinematic loops—comprising over two million synthetic training images and approximately 21,000 real test images across 210 scenes, each annotated with ground-truth joint states and part-level assembly labels. To produce this data we build supporting infrastructure, released with the benchmark: (i) a graph-based kinematic representation that extends URDF's tree structure with closed-loop constraints and the topology changes induced by missing parts; (ii) a synthetic data generator that samples valid poses on the kinematic constraint manifold and stochastically removes parts with automatic kinematic restructuring; and (iii) a real-image labeling pipeline that jointly recovers camera alignment and equipment pose via multi-view constrained optimization. The pipeline requires only a CAD model and kinematic description for each new piece of equipment, with no per-equipment code, and the seven types serve as representative test cases of real-world mechanical complexity. We adapt three methods from different families as baselines: RoboPEPP, the state of the art on the DREAM robot pose benchmark, as well as HoRoPose and CRAVES. An ablation shows that our constrained pose sampling improves accuracy for most closed-loop equipment. The best method varies by equipment, but none performs consistently well: even the strongest on average, RoboPEPP, remains well below its accuracy on DREAM, and all three degrade on average when parts are missing. We release the full dataset ([https://huggingface.co/datasets/clamp-benchmark/clamp-benchmark](https://huggingface.co/datasets/clamp-benchmark/clamp-benchmark)) as well as the synthetic data generation pipeline, evaluation code, and baseline implementations ([https://github.com/overlab-kevin/clamp](https://github.com/overlab-kevin/clamp)).
Show more
View full details
Poster

RecoverBench: A Systematic Benchmark for Error Recovery in Robotic Manipulation

Zijia Tang ⋅ Ganlong Zhao ⋅ Xingping Chen ⋅ Junye Chen ⋅ Zihao Mo ⋅ Zedi Wang ⋅ Guanbin Li
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A920
Robotic manipulation policies have achieved remarkable success in controlled settings, but they catastrophically fail when execution errors occur---dropped objects, collisions, misaligned grasps. Despite the critical importance of error recovery for real-world deployment, no existing benchmark systematically evaluates manipulation policies' ability to recover from such errors. We introduce RecoverBench, the first systematic benchmark for error recovery in robotic manipulation. We propose a taxonomy of 12 Error Skills organized into 5 Recovery Behavior Groups (RBGs), spanning 24 error subtypes across 2 difficulty degrees. Our Error Skill pipeline automatically generates 1,360 reproducible error scenes from clean demonstration trajectories across 6 manipulation tasks. Each error scene includes complete simulation state snapshots, environment fingerprints, and RNG states for deterministic reproduction. Beyond evaluation, RecoverBench also supports recovery-demo collection and augmentation, and we show that this recovery supervision improves recovery performance. We evaluate 4 representative manipulation policies on RecoverBench. Even the best-performing policy reaches only 48.6% average recovery success, ranging from 32.0% on threading to 76.7% on stack, showing that current methods still lack robust recovery. These results highlight the need for dedicated recovery evaluation and position RecoverBench as a benchmark for improving recovery robustness in manipulation policies. Code, data, and evaluation protocols will be released.
Show more
View full details
Poster

VerifyThisBench: Joint Evaluation of Code, Specifications, and Proof

Xun Deng ⋅ Barış Bayazıt ⋅ Si Cheng Zhong ⋅ Andreas Veneris ⋅ Fan Long ⋅ Xujie Si
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A917
Large language models (LLMs) have demonstrated remarkable progress in code generation, but many existing benchmarks are approaching saturation and offer little guarantee on the trustworthiness of the generated programs. To improve visibility into model reasoning on formal correctness, we introduce \verify, a new benchmark that evaluates end‑to‑end program verification from natural language descriptions: models must (i) extract formal specifications, (ii) implement in a verification‑aware language, and (iii) construct machine‑checkable proofs. Our evaluation reveals a gap between formal verification and true correctness. While models can sometimes produce programs that pass verification, only a small fraction are actually aligned with the intended task: across 1,078 tasks, six SOTA models collectively produce just 36 human-verified correct solutions, with the best model achieving only 20. This discrepancy arises because verification only guarantees correctness with respect to the generated specification, not the original intent. To address this, we introduce a filtering methodology combining automatically generated test cases with an independent LLM-based semantic judge, requiring human inspection only when the two signals disagree. Further, we propose VerifyThisBench, a relaxed variant where partial specifications, implementations, or proofs are provided disentangle sources of difficulty. Together, We release VerifyThisBench with test suite, VerifyThisBenchXS, and a unified evaluation environment spanning seven verification tools to support future research on trustworthy program synthesis.
Show more
View full details
Poster

MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks

Xinkai Zhang ⋅ Zhipeng Wei ⋅ Huanli Gong ⋅ Jing Ting Zheng ⋅ Yuchen Zhang ⋅ Yue Dong ⋅ N. Benjamin Erichson
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A913
Multi-turn jailbreaks exploit the ability of large language models to accumulate and act on conversational context. Instead of stating a harmful request directly, an attacker can gradually steer the conversation toward an unsafe answer. Recent methods demonstrate this risk, but they are usually evaluated as black-box pipelines with different budgets, judges, retry rules, and strategy generation procedures. As a result, it is often unclear whether reported gains reflect stronger attack mechanisms or different experimental conditions. We introduce MT-JailBench, a modular evaluation framework for benchmarking multi-turn jailbreaks under fixed conditions. MT-JailBench implements each attack as five interacting modules: evaluation function, attack strategy, prompt generation, prompt refinement, and flow control. This design enables fair comparison across attack methods and component-wise analysis of what drives attack success. Using MT-JailBench, we find that resource budgets and evaluation functions are major confounders: controlling turns, retries, interactions, sampled strategies, and judges substantially change the ranking of attacks. At the component level, prompt generation accounts for most performance variation, while refinement and flow control provide moderate gains. We also find that explicit dynamic strategy generation is not always necessary; stochastic sampling from a fixed strategy can rival more elaborate diversification mechanisms. Finally, recomposing the best components yields a strong attack configuration that outperforms its source attacks and generalizes across diverse target LLMs. MT-JailBench therefore provides a modular framework for comparing multi-turn jailbreaks, understanding the impact of components, and guiding stronger red-teaming evaluations.
Show more
View full details
Poster

Evaluating an Evaluation: Membership Inference Attacks as Machine Unlearning Diagnostics

Umid Suleymanov ⋅ Laman Aliyeva ⋅ Nihat Abdullayev ⋅ Saida Zarbiyeva ⋅ Murat Kantarcioglu
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A1010
Evaluation practices in machine learning are increasingly objects of scientific study in their own right: their assumptions, statistical properties, and failure modes determine which research conclusions can be trusted. We audit one such practice - the use of aggregate-subset Membership Inference Attacks (MIAs) for evaluating machine unlearning - across 8 dataset-architecture pairs, 4 unlearning methods, and over 500 experimental runs, identifying five failure modes: (1) limited and regime-dependent identifiability between baseline and retrained models, with KS-test indistinguishability ($p > 0.05$) in $\approx$42\% of experiment--task pairs; (2) regime-dependent signal-to-noise collapse; (3) monotonicity violations in 5 of 8 experiments under graded forgetting; (4) inter-task rank inconsistency, and (5) regime confounding by model training quality. To quantify these failures, we develop the Membership Inference Attack Unlearning Score (MIAU), a normalized diagnostic framework integrating three MIA comparisons as gap closure fractions between baseline and retrained references. We formalize four validity properties any unlearning metric should satisfy. Our findings indicate that aggregate-subset MIAs are reliable unlearning diagnostics only in narrow high-memorization regimes, and motivate complementary evaluation paradigms, alongside MIA-based protocols.
Show more
View full details
Poster

Auditing Evidence Framing at NeurIPS: A Decade-Scale Study of Accepted Papers

Jason Godfrey
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A1026
NeurIPS now publishes a broad mix of algorithmic, theoretical, empirical, dataset, benchmark, systems, and evaluation-infrastructure work. Yet we lack systematic evidence on how the composition of this mix has changed over time. We audit titles, abstracts, and metadata for 19,361 accepted NeurIPS papers from 2015--2024. We define semantic axes for formal-theoretical evidence, benchmark/resource evidence, empirical-performance evidence, artifact-release orientation, scale/capability framing, claim strength, and hedging; score papers with contrastive embedding measures; and validate the constructs with blinded LLM-agent ratings, lexical probes, encoder triangulation, length checks, and topic controls. In raw decade trends, accepted papers shift toward benchmark/resource evidence (+0.842 SD), empirical-performance evidence (+0.750 SD), and scale/capability framing (+0.606 SD), and away from formal-theoretical framing (-0.872 SD). Topic composition explains much of this movement, but embedding-cluster residuals remain +0.246, +0.264, +0.247, and -0.300 SD, respectively. Independent lexical LDA controls preserve the same signs at smaller magnitudes. Artifact-release framing is a useful negative result: it rises modestly in the raw corpus (+0.226 SD) but is near zero or negative after topic adjustment. We release a reusable audit package that includes metadata, semantic and lexical features, validation artifacts, robustness scripts, and an interactive Evidence Norms Atlas for inspecting aggregate evidence-framing patterns by year, topic neighborhood, and track.
Show more
View full details
Poster

DopplerWild: A Doppler Dataset and Benchmark for Human Kinematic Understanding in the Wild

Shubo Yang ⋅ Aaron Wang ⋅ Narayan Schuetz ⋅ Jida Zhang ⋅ Edison R Altamirano ⋅ Kaitlyn Leitherer ⋅ Ehsan Adeli ⋅ Amin Arbabian
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A113
Understanding human kinematics in outdoor, real-world environments is critical for assistive robots and intelligent infrastructure. Many perception systems rely on cameras and LiDAR, which infer motion indirectly from spatial observations, making estimates sensitive to discontinuities and ambiguity under low lighting or at distance. In contrast, mmWave radar is robust to lighting and weather, operates at long range, and directly measures motion through Doppler signatures. However, Doppler-based kinematic understanding remains largely unexplored in the wild because existing datasets are primarily controlled and scripted. We introduce DopplerWild, a dataset and benchmark for evaluating Doppler-based kinematic understanding in the wild. DopplerWild comprises 447k radar frames across four outdoor locations, including an unlabeled subset for self-supervised representation learning and a labeled subset of 539 subjects with nearby-person interference (56\%) and occlusion (16\%). The benchmark spans coarse and fine-grained motion classification, as well as velocity estimation. Beyond aggregate metrics on controlled datasets, DopplerWild evaluation isolates real-world factors: subject variation, location shift, multi-person interference, occlusion, sensing geometry, and low-label regimes. DopplerWild reveals three findings. First, self-supervised pretraining matches supervised performance with less than half the labeled data and improves performance under interference, occlusion, and lateral motion. Second, we identify key challenges for single-view Doppler sensing, including physical limits in lateral motion, subtle asymmetric limb motion, and location distribution shifts. Third, we observe bidirectional transfer asymmetry between DopplerWild and an external controlled Doppler dataset, suggesting that real-world coverage is important for transferable Doppler representations. DopplerWild provides an evaluation framework for characterizing the generalization, limits, and transferability of Doppler-based kinematic representations in the wild.
Show more
View full details
Poster

Benchmarks as Measurement Instruments: Quantifying Signal and Noise for More Efficient AI Evaluations Under Distribution Shift

Michael Hardy ⋅ Anka Reuel-Lamparth ⋅ Jodi Casabianca ⋅ Hansol Lee ⋅ Benjamin Domingue ⋅ Sanmi Koyejo ⋅ Mykel J Kochenderfer
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A1024
AI progress is increasingly tracked through evaluations, such as AI benchmarks, yet small perturbations in evaluation pipelines can produce unstable scores and model rankings. We recast benchmark reliability as a measurement problem and formalize it as a signal-to-noise ratio derived from a crossed random-effects decomposition. This framework yields a direct mapping between variance components and expected rank stability, linking theoretical reliability to observable leaderboard concordance. Modern AI benchmarks operate in a high-dimensional regime with many items and relatively few evaluated models, where classical item-level reliability measures are ill-posed. We define a lower bound for benchmark item-level reliability classically, and we further introduce a tractable proxy, $\lambda^\bigstar_6$, that preserves the variance structure underlying reliability without modifying the estimand through sparsity or dimensionality reduction. Across diverse benchmarks, we show that selecting items via $\lambda^\bigstar_6$ produces smaller subsets that achieve higher ranking reliability than random subsampling at fixed evaluation budgets. Finally, we analyze reliability under positive distribution shift--as often observed in the current AI ecosystem--by partitioning models into lower- and higher-performing cohorts. We show that reliability is population-dependent and frequently declines as between-model variance shifts among stronger systems. Our results establish a principled framework for diagnosing and improving benchmark reliability and demonstrate that ranking stability is neither intrinsic nor static, but a measurable and optimizable property of the benchmark–population pairing.
Show more
View full details
Poster

Where Tabular Foundation Models Falter on Genetic Data: Datasets That Expose and Provide a Path to Address the Gap

ANIRBAN DAS ⋅ Yan Cui
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A804
Tabular foundation models are used as off-the-shelf predictors for heterogeneous tabular tasks, but it remains unclear how they will perform on real genotype-to-phenotype tabular datasets, which carry unique challenges. One such challenge is ancestry-dependent non-stationarity in allele effect sizes (the effects of genetic variation on disease risk): the predictive contribution of a given genetic variant may change significantly across the ancestry spectrum. This is especially consequential for patients from ancestry groups that are underrepresented in existing datasets, because models that are unaware of this non-stationarity are more likely to perform poorly for them. The battery of synthetic tasks used to pretrain tabular foundation models does not capture this structure, and we show that it leads to systematic performance degradation. Using controlled hierarchical Gaussian-process stress tests, we demonstrate that both off-the-shelf TabICL and TabPFN are robust when ancestry dispersion in the data is low, but as dispersion grows, the latent ancestry-dependent non-stationarity of effect sizes becomes a first-order problem, and the predictive performance of both models deteriorates. We confirm the same pattern on All of Us (AoU) (a large biobank containing whole-genome sequencing and electronic health record (EHR) data spanning the ancestry spectrum) across an extensive panel of cancer phenotypes evaluated with the two leading tabular foundation models: TabICL and TabPFN. Holding the in-context training-set size, phenotype, and feature set fixed, ancestry-specific in-context tables, which are less dispersed in ancestry space, consistently outperform size-matched meta-ancestry tables, isolating ancestry dispersion as the driver of degradation. To address the failure without changing the base architecture, we construct two synthetic task families that explicitly encode ancestry-dependent effect drift and instruction-tune an off-the-shelf TabICL model on tasks sampled across both families. On held-out AoU evaluations covering cancer and respiratory disease phenotypes, the tuned model yields more stable performance across ancestry-distance bins and is especially strong in the bins farthest from the center of the in-context exemplars in ancestry space, that is, on subjects whose ancestry is most underrepresented among the provided exemplars. The paper contributes an evaluation protocol, a failure analysis, and two synthetic task families targeted at ancestry-dependent non-stationarity in precision-medicine tabular modeling.
Show more
View full details
Poster

Verify0: Can AI Agents Build Formally Verified Software Repositories?

Zhe Ye ⋅ Hantao Lou ⋅ Yuechun Sun ⋅ Peiyang Song ⋅ Zhengxu Yan ⋅ Timothe Kasriel ⋅ Qingyang Zhang ⋅ Kaiyu Yang ⋅ Soonho Kong ⋅ Jingxuan He ⋅ Dawn Song
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A916
AI agents are increasingly used for programming, but do not provide any guarantee on the correctness of generated code. Verified code generation, in which an agent produces both an implementation and a machine-checked proof of its specification, offers a stronger path toward trustworthy AI-generated software. Existing benchmarks in this direction either focus on individual functions or only evaluate proof generation with provided implementations. It is still an open question whether agents can make coherent implementation and proof choices across real multi-module codebases. To bridge this gap, we introduce Verify0, the first benchmark to evaluate joint implementation and proof synthesis at the repository level. Verify0 contains 35 multi-module instances sourced from real-world repositories spanning Python, Dafny, Verus, and Coq, and covering diverse domains from cryptographic protocols to distributed systems. Each instance consists of a multi-module Lean 4 repository with predetermined API interfaces, manually curated formal specifications, and reference implementations, supporting both proof-only and code-and-proof evaluation modes. To improve benchmark reliability, Verify0 also includes an audit mechanism where agents are allowed to formally prove unsatisfiability of provided specification or incorrectness of reference code, which surfaces and corrects latent code and specification errors during curation. We evaluate frontier coding-agent configurations with Lean toolchain access. The strongest agent fully solves only 18 of 35 instances and closes no specifications on the hardest repositories. Verify0 provides a concrete testbed for measuring progress toward repository-scale verified software synthesis, where current agents still fall short. We release the benchmark, curation pipeline, and evaluation harness at https://anonymous.4open.science/r/verify0-neurips26-submission-82C2.
Show more
View full details
Poster

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

Aritra Mazumder ⋅ Shubhashis Roy Dipta ⋅ Nusrat Jahan Lia ⋅ Tanzila Khan ⋅ Kainat R Hossain ⋅ Nehaa Shri ⋅ Humayra Tasnim ⋅ Gour G Shawon ⋅ Shubhrangshu Debsarkar ⋅ Sumaiya A Rani ⋅ Al Jami Islam Anik ⋅ Debjoty Mitra ⋅ Al Nafeu Khan
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A114
Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system's final output may look correct even though the reasoning chain was quietly corrupted, and existing outcome-based evaluations are blind to such multi-hop process failures. To make these vulnerabilities measurable before deployment, we introduce AgentCollabBench, a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering. Each task isolates one of four behavioral risks: instruction decay (does a constraint survive peer pressure?), false-belief contagion (does a falsehood spread through consensus?), context leakage (does information bleed between tasks?), and tracer durability (does marked data reach the final agent?). Evaluating four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct), we expose model-specific vulnerability profiles invisible to outcome-only evaluation; Qwen-3.5-35B-A3B, for example, leads on tracer durability and instruction stability, while GPT 4.1 mini leads on leakage containment and false-belief resistance. Beyond per-model differences, communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival. The effect traces to a synthesis bottleneck specific to converging-DAG nodes: an agent weighing competing parent inputs discards constraints carried by a minority branch, a bottleneck structurally absent from linear chains. AgentCollabBench demonstrates that suboptimal topology can silently erase the safeguards of highly capable models, arguing that multi-agent reliability is fundamentally a structural problem and that scaling model intelligence alone is no substitute for architecture.
Show more
View full details
Poster

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

Hanqi Jiang ⋅ Junhao Chen ⋅ Yi Pan ⋅ Lifeng Chen ⋅ Weihang You ⋅ Haozhen Gong ⋅ Ruiyu Yan ⋅ Jinglei Lv ⋅ Lin Zhao ⋅ Hui Ren ⋅ Quanzheng Li ⋅ Tianming Liu ⋅ Xiang Li
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A1025
Medical vision–language models (VLMs) are usually evaluated on intact image–question pairs, but trustworthy clinical use requires a stronger property: a model must recognise when the evidential basis for an answer has failed. We study this through silent failures under perturbed evidence, where a vision-required medical question is paired with a false premise, wording perturbation, knowledge-only rewrite, or ROI-corrupted image, yet the model returns a fluent non-refusal answer. We introduce MedVIGIL, a 300-case evaluation suite drawn from four public medical VQA sources, supervised end to end by four board-certified radiologists: every gold answer, refusal option, candidate-answer set, paraphrase, false-premise trap, ROI box, and clinical risk tier is clinician-authored. Two attending radiologists annotate every case in parallel, a senior radiologist consolidates the released manifest, and a separate fourth radiologist independent of construction answers every probe to provide the human reference baseline. The release contains 2,556 MCQ probes, 240 counterfactual triplets, physician-adjudicated risk-tier and answerability flags, ROI boxes, and a paired open-ended variant. We report seven correctness-conditioned audit metrics that summarise into the MedVIGIL Composite Score (MCS), and audit 16 vision-capable models plus two text-only baselines. The independent radiologist scores MCS 83.3 at silent-failure rate 5.8%, leaving a 14.1-point composite headroom above the strongest audited model (Claude Opus 4.7 at 69.2). The benchmark and evaluation harness are publicly released.
Show more
View full details
Poster

Acting without Knowing: Planning, Prediction, and Transfer Dissociate in Interactive Visual Physics

Alison Gopnik ⋅ Eunice Yiu ⋅ Kevin Murphy ⋅ Kelsey Allen ⋅ Shiry Ginosar
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A716
Evaluations of large multimodal models (LMMs) often conflate behavioral task success with true physical understanding. However, classical accounts of ``world models'' dictate that physical understanding requires the unified operation of three core capacities: planning, prediction, and positive transfer. Because current benchmarks typically test these three abilities in isolation, they often obscure whether models actually possess a cohesive internal world representation. In this paper, we propose a joint evaluation protocol to test whether LMMs acquire a more complete and consistent understanding of the physical world. Using a suite of virtual physics-based puzzles, models are tasked with placing a tool in a scene to alter its dynamics, iteratively revising their placements based solely on visual feedback. By testing the same models on the exact same tasks, we uncover a stark dissociation between acting and knowing. While LMMs can sometimes solve diverse physical goals under an interactive action-feedback protocol, their success is not accompanied by robust feedback-guided replanning, accurate prediction, or human-like predictive transfer to new actions. Models show weak and inconsistent prediction of the consequences of their own proposed actions, and their predictive transfer to new actions is partial, inconsistent, and less selectively goal-directed than in humans. Ultimately, we demonstrate that the capacities a world model should wholly unify remain only loosely connected. We therefore argue that benchmarks for LMMs should not measure task success alone, but should jointly evaluate planning, prediction, and transfer.
Show more
View full details
Poster

VFIG: Vectorizing Complex Figures in SVG with Vision-Language Models

Qijia He ⋅ Xunmei Liu ⋅ Hammaad Memon ⋅ Ziang Li ⋅ Zixian Ma ⋅ Jaemin Cho ⋅ Jason Ren ⋅ Daniel Weld ⋅ Ranjay Krishna
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A714
Scalable Vector Graphics (SVG) are essential for technical illustration and digital design, offering resolution independence and semantic editability. In practice, original vector files are frequently lost, leaving only rasterized versions (e.g., PNG, JPEG) that resist modification, while manual reconstruction is prohibitively expensive. Progress on automating raster-to-SVG conversion has been bottlenecked by two gaps: existing SVG datasets are dominated by icons and decorative graphics that lack the complexity of professional diagrams, and existing benchmarks rely on pixel- or embedding-level similarity that fails to capture structural correctness (e.g., broken connectivity, misplaced arrows). We close both gaps with paired contributions targeting diagram-centric figures (e.g., model architectures, flowcharts, schematics). For training, we introduce VFIG-Data, the largest figure-to-SVG dataset of its kind at 66K pairs, combining real paper figures converted via a describe-and-generate pipeline with programmatic diagrams that supply noise-free supervision over arrow styles, fonts, and geometry. For evaluation, we introduce VFIG-Bench, a structure-aware evaluation suite, paired with VFIG-Bench-OOD, an out-of-distribution set of figures manually curated from highly cited arXiv papers. Beyond pixel and embedding similarity, our protocol reports rubric-based VLM-Judge scores and Elo ratings from pairwise human preference evaluation. Built on these contributions, VFIG is a VLM family trained with a simple-to-complex SFT curriculum followed by RL with rendering-aware rewards. VFIG achieves state-of-the-art open-source performance, outperforming the best open-source VLM baseline by over 30\%, and matches Claude Sonnet 4.6 on VFIG-BENCH: Gemini-Judge 78.2% vs. 76.7% and GPT-Judge 87.5% vs. 87.4%. It remains slightly behind the strongest proprietary models GPT-5.2 and Gemini-3.
Show more
View full details
Poster

When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents

Xiaolin Zhou ⋅ Aojie Yuan ⋅ Zheng Luo ⋅ Zipeng Ling ⋅ Xixiao Pan ⋅ Yicheng Gao ⋅ Haiyue Zhang ⋅ Jiate Li ⋅ Shuli Jiang ⋅ Prince Z Wang ⋅ Zixuan Zhu ⋅ Jinbo Liu ⋅ Ryan Rossi ⋅ Hua Wei ⋅ Xiyang Hu
Dec 11, 8:30 AM - 11:30 AM Hall C1 #A503
Tool-use language agents are evaluated on benchmarks that assume clean inputs, unambiguous tool registries, and reliable APIs. Real deployments violate all these assumptions: user typos propagate into hallucinated tool names, a misconfigured request timeout can stall an agent indefinitely, and duplicate tool names across servers can freeze an SDK. We study these failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP), where deployment noise enters through the observation, action space, reward-relevant metadata, or transition dynamics. We introduce RobustBench-TC, a benchmark with 22 perturbation types organized by these four POMDP components, each grounded in a verified GitHub issue or documented tool-calling failure. Across 21 models from 1.5B to 32B parameters (including the closed-source o4-mini), the robustness profile is sharply uneven: observation perturbations reduce accuracy by less than 5%, while reward-relevant and transition perturbations reduce accuracy by roughly 40% and 30%, respectively; scale alone does not close these gaps. We then propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories spanning the three statically encodable POMDP components. On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate perturbed accuracy comparable to open-source 14B function-calling baselines while substantially narrowing the gap to o4-mini. It closes approximately 27% of the Transition gap despite never seeing transition perturbations in training, suggesting that RL on adversarial static tool-use inputs induces a more persistent retry policy that transfers to unseen runtime failures. The code and benchmark dataset are available at https://anonymous.4open.science/r/robustbench-tc-release-3AB7/.
Show more
View full details
Poster

TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraints

Zabir Al Nazi ⋅ Shubhashis Roy Dipta
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A429
Deploying language models as autonomous agents requires more than per-task accuracy: when an agent faces a queue of problems under a finite token budget, it must decide which to attempt, in what order, and how much compute to commit to each, all before any execution feedback is available. This is the prospective form of metacognitive control studied for decades in human cognition, yet whether language models possess it remains untested. We introduce TRIAGE, an evaluation framework in which a model receives a task pool and a token budget calibrated to its own baseline cost, and commits to a single ordered plan that jointly encodes selection, sequencing, and per-problem allocation. Plans are scored against an oracle with full knowledge of the model's solvability and cost on each problem, yielding a triage efficiency ratio on a common scale. We evaluate frontier and open-source models, with and without reasoning enabled, across competition mathematics, graduate-level science, code generation, and expert multidisciplinary knowledge, and find that current language models exhibit substantial gaps in prospective metacognitive control, revealing a previously unmeasured capability dimension with direct implications for resource-efficient agent deployment.
Show more
View full details
Poster

TimeWarp: Evaluating Web Agents by Revisiting the Past

Farhan Ishmam ⋅ Kenneth Marino
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A309
As web agents close the gap with humans on benchmarks, it raises the question: *Do today's agents perform just as well on tomorrow's web?* We introduce TimeWarp, a benchmark that emulates the evolving web. TimeWarp consists of three web environments, each with six UI versions spanning UI design and frontend code from different eras of the internet. We pair TimeWarp with a set of complex, realistic tasks covering different forms of web navigation. Our experiments reveal vulnerabilities of web agents, especially visual ones, to changes and the limitations of behavior cloning (BC) on complex trajectories from a single version. To address this, we propose TimeTraj, a simple yet effective algorithm that uses plan distillation to collect trajectories across multiple versions. By training agents on teacher rollouts using our BC-variant, we achieve substantial performance gains: $20.4$ %$\rightarrow37.7$% for Qwen-3 4B and $0$ % $\rightarrow27.0$% for Llama-3.1 8B models. Our work helps study generalization across web designs and enables a new paradigm for collecting plans rather than trajectories to improve the robustness of web agents.
Show more
View full details
Poster

ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers

Jiayi Wu ⋅ Robert Joseph George ⋅ Animashree Anandkumar
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A919
Formal theorem proving has emerged as a frontier challenge for machine learning, yet the ecosystem is fragmented: proofs remain siloed across incompatible systems, limiting both training data for learning-based provers and the portability of verified results. We present **ITPEval**, the first benchmark for evaluating automated formal proof translation across four major ITPs (Lean 4, Rocq, Isabelle, and HOL Light), spanning two distinct logical foundations (the Calculus of Inductive Constructions and higher-order logic). Our benchmark comprises 1,560 source files spanning 6,848 theorems and lemmas across four systems (390 aligned items per ITP), organized into two tiers: a *controlled* tier of self-contained, axiomatized files (64 files, 660 lemmas), and an *ecosystem* tier of 1,496 files drawn from existing libraries and community formalizations. We release `itpeval`, a unified multi-ITP verification infrastructure with state-isolated warm backends that amortize prover startup while preserving per-artifact native checking semantics. We evaluate both *statement* and *proof* translation across five frontier and open-weight LLMs on 12 directed translation pairs: statement translation peaks at 29.1% pass@1 and proof translation at 10.5%; controlled theorems reach 29.7% proof pass@1 versus 5.2% for ecosystem-level translations, confirming that library mismatch is the dominant bottleneck. In an autoformalization/auto-informalization round-trip study, multi-ITP context substantially improves Lean 4 formalization success (4.8% to 10.6%), showing that aligned cross-ITP corpora serve not only as evaluation benchmarks but also as generation-time evidence. Our benchmark, verification infrastructure, and evaluation pipelines are publicly released.
Show more
View full details
Poster

VeruSAGE-Bench: A Benchmark Suite for Rust System Verification

Chenyuan Yang ⋅ Natalie Neamtu ⋅ Chris Hawblitzel ⋅ Jacob R Lorch ⋅ Shan Lu
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A1123
Large language models (LLMs) have shown impressive capability to understand and develop code. However, their capability to rigorously reason about and formally prove the correctness of software systems remains in question. Current benchmarks for formal proof generation focus on isolated tasks, typically algorithmic-style problems utilizing basic data structures. We curate a new benchmark suite for system-level proof generation, VeruSAGE-Bench, which consists of 849 proof tasks extracted from eight open-source Verus-verified Rust systems, covering operating systems, storage systems, memory allocators, and more. These tasks are much more complicated than those in previous benchmarks, containing on average more than 20x the formal specification (in Lines of Code) and requiring a much broader array of verification techniques. We evaluate five LLMs on VeruSAGE-Bench (o4-mini, GPT-5, Sonnet 4, Sonnet 4.5, Opus 4.5), and demonstrate that different models and agentic setups lead to vastly different success rates and costs. We also show that the best models possess impressive capability in writing correctness proofs and hence should inspire new ways of developing trustworthy software. Meanwhile, some difficult tasks in VeruSAGE-Bench stress even the best model, forcing long reasoning time (over an hour on the hardest tasks, 15.3 min per task on average), high token cost ($13.65 per task on average), and proof failures, which we hope will guide future improvement in model and agents.
Show more
View full details
Poster

OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries

Diane Tchuindjo ⋅ Devavrat Shah ⋅ Omar Khattab
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A203
Retrieval benchmarks are increasingly saturating, but we argue that efficient search is far from a solved problem. We identify a class of queries we call *oblique*, which seek documents that *instantiate a latent pattern*, like finding all tweets that express an implicit stance, chat logs that demonstrate a particular failure mode, or transcripts that match an abstract scenario. We study three mechanisms through which obliqueness may arise and introduce OBLIQ-Bench, a suite of five oblique search problems over real long-tail corpora. OBLIQ-Bench exposes an overlooked asymmetry between retrieval and verification, where reasoning LLMs reliably recognize latent relevance whenever relevant documents are surfaced, but even sophisticated retrieval pipelines fail to surface most relevant documents in the first place. We hope that OBLIQ-Bench will drive research into retrieval architectures that efficiently capture latent patterns and implicit signals in large corpora.
Show more
View full details
Poster

ColdDDI: Evaluating Knowledge Utilization in Cold-Start Drug-Drug Interaction Prediction

Jiheng Liang ⋅ Chen Zhao ⋅ Di Wu ⋅ Chenyang Bu ⋅ Yunpeng Hong ⋅ Xingquan Zhu ⋅ Yi He
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A707
Cold-start drug-drug interaction (DDI) prediction tests whether models can identify clinically significant interactions for drugs without training-time interaction history. Existing benchmarks mostly report aggregate edge-prediction scores, leaving a key evaluation question unanswered: when models receive molecular, textual, or knowledge-graph (KG) evidence, do they actually use the evidence that pharmacologically supports the interaction? We introduce ColdDDI, a reconstructible evaluation benchmark built from the latest DrugBank 5.1.13, with 1,900 approved small-molecule drugs and 565,731 positive DDI pairs. ColdDDI evaluates three levels of drug novelty, namely test pairs whose two drugs were both seen during training, pairs with one unseen drug, and pairs with two unseen drugs. It also annotates each interaction by whether it changes drug exposure or drug effect, and by whether the biomedical knowledge graph contains shared enzymes, transporters, or targets that can plausibly mediate the interaction. These annotations allow ColdDDI to separate two factors that aggregate metrics conflate, namely whether mechanistic evidence is available, and whether a model prediction depends on that evidence. We evaluate eight conventional DDI methods and 13 LLMs; for open-weight LLMs, we test five prompt patterns and use masking, drug replacement, and channel-sensitivity metrics to probe knowledge utilization. ColdDDI exposes that, in the hardest split where both drugs are unseen, the main performance divide is mediator availability. A fine-tuned 1B LLM recovers 89-93% of interactions with a shared enzyme, transporter, or target, but only 40-62% without such a mediator. More importantly, KG-provided evidence is not always used; several KG-augmented baselines change little when the shared mediator is masked or disrupted, whereas fine-tuned LLMs respond strongly to this intervention. Thus, ColdDDI evaluates knowledge utilization rather than knowledge access alone, showing where cold-start DDI models rely on mechanistic evidence and where they fail despite receiving it. We release derivative annotations, splits, prompts, diagnostics, evaluation code, and a DrugBank reconstruction pipeline.
Show more
View full details
Poster

CFC26: Building Evaluations for Deployment in Sonar-Based Fish Counting

Madison Van Horn ⋅ Suzanne Stathatos ⋅ Sevan Brodjian ⋅ Justin Kay ⋅ Kai Van Brunt ⋅ Erik Young ⋅ Sara Beery ⋅ Pietro Perona ⋅ Michael Hobley
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A713
Accurate counts of fish populations are essential for monitoring ecosystems and making decisions about conservation, harvest limits, and river management. In many field deployments, these counts are obtained from sonar videos that experts must inspect manually, making large-scale monitoring slow, expensive, and difficult to reproduce. Automated counting systems have been proposed as a scalable alternative, but so far this promise has been difficult to realize in deployment. Developing evaluations that better reflect real world deployment conditions will close this gap, and drive improvements to counting methods that translate to reality. To achieve this, we propose Caltech Fish Counting 2026 (CFC26), a new dataset with improved evaluation protocols that simulate deployment to current and new rivers, testing in-distribution and out-of-distribution performance. We expand the CFC22 dataset to eleven deployment locations across nine Pacific salmon river systems from Alaska to Northern California, capturing a broader geographical and ecological range of data. We additionally introduce two new evaluation metrics for counting, nMNE and nMANE, that mirror the operational workflow that experts use and help surface errors that frame-level detection metrics do not capture. Using these metrics, we reveal previously-concealed directional bias in upstream vs downstream counts, which would have direct and significant ecological implications if undetected.
Show more
View full details
Poster

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

Zhiyu Huang ⋅ Johnson Liu ⋅ Rui Song ⋅ Zewei Zhou ⋅ Ruining Yang ⋅ Yun Zhang ⋅ Tianhui Cai ⋅ Hanyin Zhang ⋅ Mingxuan Gao ⋅ Valeria Xu ⋅ Jiali Chen ⋅ Yishan Shen ⋅ Yiluan Guo ⋅ Tony Qi ⋅ Jiaqi Ma
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A926
Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions, and make safe decisions. However, existing AD datasets and benchmarks mainly target perception, prediction, or planning, and provide limited supervision for reasoning over realistic long-tail driving scenes. We introduce nuReasoning, a large-scale real-world dataset and benchmark for reasoning-centric AD. Following the lineage of nuScenes and nuPlan, nuReasoning advances real-world AD datasets and benchmarks toward structured reasoning in long-tail driving scenarios. It contains 20K 20-second clips collected across multiple cities, with synchronized multi-camera images, LiDAR, HD maps, object annotations, and complementary forms of human-verified reasoning annotations: Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning. Unlike prior datasets that focus primarily on visual question answering, nuReasoning supports both reasoning evaluation and planning evaluation, enabling a direct study of how reasoning supervision affects driving performance. Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific question answering, while incorporating reasoning supervision into VLA training improves planning performance even when textual reasoning outputs are disabled at inference time. These results establish nuReasoning as a foundation for evaluating and improving robust, interpretable, reasoning-driven AD systems in realistic long-tail settings.
Show more
View full details
Poster

DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy

Erchi Wang ⋅ Pengrun Huang ⋅ Eli Chien ⋅ Om Thakkar ⋅ Kamalika Chaudhuri ⋅ Yu-Xiang Wang ⋅ Ruihan Wu
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A1003
Differential privacy (DP) has a wide range of applications for protecting data privacy, but designing and verifying DP algorithms requires expert-level reasoning, creating a high barrier for non-expert practitioners. Prior works either rely on specialized verification languages that demand substantial domain expertise or remain semi-automated and require human-in-the-loop guidance. In this work, we investigate whether large language models (LLMs) can automate DP reasoning. We introduce DPrivBench, a benchmark in which each instance asks whether a function or algorithm satisfies a stated DP guarantee under specified assumptions. The benchmark is carefully designed to cover a broad range of DP topics, span diverse difficulty levels, and resist shortcut reasoning through trivial pattern matching. Experiments show that while the strongest models handle textbook mechanisms well, all models struggle with advanced algorithms, revealing substantial gaps in current DP reasoning capabilities. Through further analytic study and failure-mode analysis, we identify several promising directions for improving automated DP reasoning. Our benchmark provides a solid foundation for developing and evaluating such methods, and complements existing benchmarks for mathematical reasoning.
Show more
View full details
Poster

A$^3$Bench: A Benchmark for Compositional Reasoning over Aggressive Interactions in Videos

Austin C Baker ⋅ Raiyaan Abdullah ⋅ Daniel Z Yoffe ⋅ Shruti Vyas ⋅ Yogesh Rawat
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A714
Recent Multimodal Large Language Models (MLLMs) have achieved strong performance on general video understanding benchmarks, yet their ability to reason about participant roles in aggressive interactions remains largely unexplored. Existing violence and crime datasets primarily evaluate whether aggression occurs, without testing whether models can identify who performed the action, who received it, and how participants are related within the scene. To address this gap, we introduce **A$^3$Bench**, a benchmark to study *aggressive action and association* in videos. The benchmark evaluates whether models can jointly recognize aggressive actions, distinguish aggressors from victims and bystanders, and correctly bind actions to participants under challenging distractor settings. We benchmark 15 recent MLLMs and find that current models struggle substantially on fine-grained aggressive-scene understanding. Performance degrades consistently as reasoning complexity increases, with models frequently confusing aggressors with victims or bystanders, revealing a systematic failure in role-action association rather than coarse aggression recognition alone. We further observe that increasing model scale or applying generic reasoning prompts provides limited improvement. Finally, we explore a simple role-graph prompting strategy that encourages structured reasoning over participant interactions, leading to consistent gains on compositional reasoning. Our findings highlight that reliable aggressive-scene understanding remains a major challenge for current MLLMs and motivate future research on role-aware video reasoning beyond coarse violence detection.
Show more
View full details
Poster

DDBench: A Benchmark for Agentic Debugging on Distributed Systems

Yibo Yan ⋅ Huijuan Wang ⋅ Junzhou He ⋅ Yizhuo Liang ⋅ Shaoyu Wang ⋅ Huanchen Sun ⋅ Seo Jin Park
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A122
LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified. Distributed-system debugging, however, remains an under-explored regime: bugs span processes, nodes, and protocol interactions, with root causes rarely recoverable from source alone and brute-force exploration intractable across non-deterministic interleavings. This leaves two gaps in LLM and agent evaluation: no code-repair benchmark targets distributed-system bugs, and no controlled study isolates how much externally provided debugging context changes agent success on them. We introduce DDBench, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems, partitioned into three difficulty tiers. DDbench evaluates every case under two matched conditions: a symptom-only condition where the agent receives only the bug symptom and repository, and a context-augmented condition where it additionally receives a bounded debugging context (logs, traces, runtime state, and targeted code-investigation notes), isolating the effect of debugging context from model capability. The evaluation of ten LLMs on DDBench reveals several findings. First, distributed debugging exercises a reasoning dimension that single-process benchmarks do not surface: models’ pass rates span 61 pp, and pairwise bootstrap separates 9 of 15 top-tier model pairs at p < 0.05 on DDBench's hardest case-set. Second, bounded debugging context lifts aggregate pass rate by +18.1 pp, and the lift is asymmetric: weaker models gain pass rate, while stronger models gain efficiency. Third, debugging context requires careful curation, as even faithful debugging context can sometimes mislead LLMs.
Show more
View full details
Poster

Contextualized Evaluation of Vision Language Models through Dynamic Interviews

Yijiang Li ⋅ Huiqi Zou ⋅ Bingyang Wang ⋅ Ziang Xiao
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A722
Multi-modal Large Language Models (MLLMs) have made substantial advances on benchmarks, yet their real-world effectiveness remains uncertain. This gap stems from the fundamental misalignment between benchmarks in controlled, static settings and the dynamic, interactive, and contextualized nature of real-world applications. To bridge this gap, we propose CEDI (Contextualized Evaluations of MLLMs through a Dynamic multi-round Interview, a framework that recasts evaluation as a three-party interaction between an evaluatee model, an automated examiner, and a grader. The examiner conducts multi-turn, semi-structured interviews guided by a graph-based representation of the task. By navigating state-space transitions, CEDI deploys diverse strategies—from clarification requests to adversarial probes—to elicit performance evidence.
Show more
View full details
Poster

PBT-Bench: Benchmarking AI Agents on Property-Based Testing

Guohao Jing ⋅ Xinqi Wang ⋅ Liao Zhang ⋅ Simon Du
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A123
Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue. Neither isolates the distinct skill of property-based testing: deriving a semantic invariant from documentation, and then constructing an input-generation strategy precise enough to make a random search reveal the violation. We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries. Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them; the agent must read the library's documentation, identify the relevant invariant, and specify a Hypothesis @given strategy that concentrates mass in the trigger region. Bugs are stratified across three difficulty levels (L1–L3) spanning single-constraint boundary bugs to stateful, cross-function protocol violations. We evaluate eight contemporary LLMs under two prompting regimes (open-ended baseline vs. explicit Hypothesis scaffolding) for three independent runs per configuration. Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%. Hypothesis scaffolding lifts mid-capability models by over 20 percentage points, but yields smaller gains for the strongest models, with two exceptions showing degradation, suggesting the structured prompt can interfere with certain model behaviours rather than complementing them. The hardest bugs prove model-specific: different architectures fail on different problems, leaving persistent gaps that no single model closes. We release the benchmark, harness, and full evaluation corpus to support downstream work on documentation-grounded semantic reasoning.
Show more
View full details
Poster

AdShot: Benchmarking Multimodal Large Language Models for Video Advertisement Clipping

Wen Xie ⋅ Om Rastogi ⋅ Sai S Rangoju ⋅ Gijs Overgoor ⋅ Yakov Bart ⋅ Sarah Ostadabbas
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A726
Advertisers routinely create multiple duration variants of ads to meet marketing budgets and viewer preferences, a labor-intensive and costly process. While multimodal large language models (MLLMs) excel at general video understanding, their ability to perform specialized editorial tasks remains unexplored. We introduce \textbf{AdShot}, a benchmark for evaluating MLLMs on shot selection for ad clipping. It contains 823 professionally edited ad pairs (30-sec sources and 15-sec edits) across 194 brands and 17 industries, annotated with shot-level boundaries and content features (action, emotion, information, imagery). We evaluate 13 state-of-the-art video-only MLLMs and 2 audio-visual MLLMs (7B–72B parameters) on shot selection accuracy (Precision, Recall, F1), duration constraints (15s), and content preservation. We find: (1) model architecture matters more than scale, with smaller models often matching or exceeding larger ones; performance is strongest for target ads with 6–10 shots; (2) models exhibit a mean absolute duration error of 5.39s, with most tending to overshoot the 15s target, though they can still substantially reduce human editing effort; (3) all models exhibit systematic content biases, favoring information-rich and emotion-focused shots while under-selecting action and imagery. We publicly release our benchmark to facilitate research at the intersection of multimodal AI and computational advertising.
Show more
View full details
Poster

SourceBench: Can AI Answers Reference Quality Web Sources?

Hexi Jin ⋅ Xurui Liu ⋅ Yuheng Li ⋅ Simran Malik ⋅ Yiying Zhang
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A124
Large language models (LLMs) increasingly answer queries by citing web sources, but existing evaluations emphasize the final answer rather than evidence quality. We introduce SourceBench, a benchmark for measuring the quality of cited web sources across 100 real-world queries spanning informational, factual, argumentative, social, and shopping intents. SourceBench uses an eight-metric framework covering content quality (content relevance, factual accuracy, objectivity) and page-level signals (e.g., freshness, authority/accountability, clarity), and includes a human-labeled dataset with a calibrated LLM-based evaluator that matches expert judgments closely. We evaluate eight LLMs, Google Search, and three AI search tools over 3996 cited sources using SourceBench and conduct further experiments to understand the evaluation results. Overall, our work reveals four key new insights that can guide future research in the direction of GenAI and web search.
Show more
View full details
Poster

TDBench: Benchmarking Vision Language Models on Top-Down Image Understanding

Kaiyuan Hou ⋅ Minghui Zhao ⋅ Lilin Xu ⋅ Yuang Fan ⋅ Xiaofan Jiang
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A727
Top-down images are critical in applications such as autonomous navigation and aerial surveillance, whereas Vision-Language Models (VLMs) are primarily trained and evaluated on front-view benchmarks, leaving their performance in top-down settings largely unexplored. Moreover, conventional evaluation protocols based on single-pass accuracy or repeated testing on identical questions can overestimate model capability due to hallucinations or chance correctness. To address these limitations, we introduce a 2{,}000-question benchmark over ten task categories for drone-altitude image understanding, and RotationalEval (RE), an evaluation protocol that leverages rotational invariance to measure answer consistency across multiple orientations of the same scene. Human evaluators lose almost nothing going from single-pass accuracy to RE, whereas across the 60 open-source and proprietary VLMs we evaluated, accuracy drops by 15 to 26 points. We analyze per-question rotation-outcome distributions across the models we evaluated with an Empirical Mixture Model that compares each model's distribution over the number of correctly-answered rotations (0 to 4) to a difficulty-adjusted Poisson-binomial null, splitting the residual into a mastery component and a consistent-failure component. The benchmark is integrated into an open-source evaluation toolkit and will be released upon publication.
Show more
View full details
Poster

Towards Visual Query Segmentation in the Wild

Bing Fan ⋅ Minghao Li ⋅ Hanzhi Zhang ⋅ Shaohua Dong ⋅ NAGA PRUDHVI MAREEDU ⋅ Xiaoqiong Liu ⋅ Weishi Shi ⋅ Yunhe Feng ⋅ Yan Huang ⋅ Heng Fan
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A719
In this paper, we introduce visual query segmentation (VQS), a new paradigm of visual query localization (VQL) that aims to segment all pixel-level occurrences of an object of interest in an untrimmed video, given an external visual query. Compared to existing VQL locating only the last appearance of a target using bounding boxes, VQS enables more comprehensive (ie, all object occurrences) and precise (ie, pixel-level masks) localization, making it more practical for real-world scenarios. To foster research on this task, we present VQS-4K, a large-scale benchmark dedicated to VQS. Specifically, VQS-4K contains 4,111 videos with more than 1.3 million frames, and covers a diverse set of 211 object categories. Each video is paired with a visual query defined by a frame outside search video and its target mask, and annotated with spatial-temporal masklets corresponding to the queried target. To ensure high quality, all videos in VQS-4K are manually labeled with meticulous inspection and iterative refinement. To the best of our knowledge, VQS-4K is the first benchmark specifically designed for VQS. Furthermore, to stimulate future research, we present a simple yet effective method, named VQ-SAM, which extends SAM 2 by leveraging target-specific and background distractor cues from the video to progressively evolve the memory through a novel multi-stage framework with an adaptive memory generation (AMG) module for VQS, significantly improving the performance. In our extensive experiments on VQS-4K, VQ-SAM achieves promising results and surpasses all existing approaches, demonstrating its effectiveness. With the proposed VQS-4K and VQ-SAM, we expect to go beyond current VQL paradigm and inspire more future research and practical applications on VQS. Our benchmark, code, and results will be made publicly available.
Show more
View full details
Poster

An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation

Haoyu Han ⋅ Li Ma ⋅ Hanbing Wang ⋅ Bingheng Li ⋅ Daochen Zha ⋅ Chun How Tan ⋅ Huiji Gao ⋅ Xin Liu ⋅ Stephanie Moyerman ⋅ Sanjeev Katariya ⋅ Hui Liu ⋅ Jiliang Tang
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A125
Sequential recommendation is a central task in recommender systems, and recent research has increasingly shifted toward generative recommenders that leverage both sequential patterns and semantic item information. However, these methods are often evaluated on a small set of widely used benchmarks. This raises a natural question: do these benchmarks actually require the advanced modeling capabilities of modern generative recommenders? We conduct a benchmark audit using an intentionally simple graph heuristic: starting from only the most recent one or two interacted items, it retrieves candidates from a few-hop item-transition graph and ranks them with item-feature similarity. Surprisingly, despite its simplicity, this heuristic matches or outperforms a broad set of modern baselines on a variety of popular sequential recommendation benchmarks. For example, it achieves relative NDCG@10 improvements of 38.10\% and 44.18\% over the best competing baseline on the widely used Amazon Review Sports and CDs datasets, respectively. We further show that this phenomenon is not merely an artifact of a particular heuristic, but reflects shortcut solvability in existing benchmarks. Specifically, we identify three potential shortcut structures that could make next-item prediction easier than expected: low-branching local transition structure, feature-smooth transitions, and limited dependence on long user histories. These shortcuts do not need to appear simultaneously. Depending on the dataset, even one or two strong shortcut signals can make simple local retrieval highly competitive, while weakening the relevant signals allows more sophisticated models to show clearer benefits. Our broader evaluation across 14 diverse datasets further shows that model rankings change substantially with dataset properties, while the simple graph heuristic remains competitive on 10 out of 14 datasets. These findings suggest that strong performance on several standard sequential recommendation benchmarks may not faithfully reflect whether recent methods achieve the advanced modeling capabilities they aim to demonstrate. Rather than treating datasets as interchangeable leaderboards, we argue for more careful dataset selection and dataset-level diagnostic analysis when using benchmarks to support claims about the benefits of new recommendation models.
Show more
View full details
Poster

StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

Daeun Lee ⋅ Subhojyoti Mukherjee ⋅ Branislav Kveton ⋅ Ryan Rossi ⋅ Viet Lai ⋅ David Seunghyun Yoon ⋅ Trung Bui ⋅ Franck Dernoncourt ⋅ Mohit Bansal
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A928
Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realistic applications such as Augmented Reality (AR) glasses. While prior streaming benchmarks evaluate temporal reasoning, none measure whether Multimodal Large Language Models (MLLMs) can interpret or leverage human gaze signals within a streaming setting. To fill this gap, we introduce StreamGaze, the first benchmark designed to evaluate how effectively MLLMs utilize gaze for temporal and proactive reasoning in streaming videos. StreamGaze introduces gaze-guided past, present, and proactive tasks that comprehensively assess streaming video understanding. These tasks evaluate whether models can use real-time gaze signals to follow shifting attention and infer user intentions based only on past and currently observed frames. To build StreamGaze, we develop a gaze–video Question Answering (QA) generation pipeline that aligns egocentric videos with raw gaze trajectories through fixation extraction, region-specific visual prompting, and scanpath construction. This pipeline produces spatio-temporally grounded QA pairs that reflect human perceptual dynamics. Across all StreamGaze tasks, we observe substantial performance gaps between state-of-the-art MLLMs and human performance, highlighting key limitations in gaze-based temporal reasoning, intention modeling, and proactive prediction. We further provide detailed analyses of gaze prompting strategies, reasoning behaviors, and task-specific failure modes, offering insights into current limitations and directions for future research. All data and code are publicly available to support research in gaze-guided streaming video understanding.
Show more
View full details
Poster

Be CARE-ful with Text-to-SQL Benchmarks

Haonan Wang ⋅ Jiaxiang Liu ⋅ Elaine Ang ⋅ Tiancheng Ge ⋅ Wanting You ⋅ Austin S Wijaya ⋅ Reya Vir ⋅ Tianle Zhou ⋅ Mihir Agarwal ⋅ Yusen Zhang ⋅ Eugene Wu
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A808
Enterprise text-to-SQL systems powered by LLM-based agents require more than correct final SQL queries to be production-ready. In realistic deployments, an agent may query databases that are changing concurrently, consult access-control and safety policies, respect business rules and integrity constraints, and iteratively refine SQL before producing a final answer. Therefore, beyond returning correct answers, enterprise agents must remain reliable, policy-compliant, rule-compliant, and efficient throughout the SQL-generation trajectory. We introduce CAREFUL, a benchmark and executable evaluation environment for production-oriented text-to-SQL agents. CAREFUL is grounded in 590 practitioner articles, incident reports, standards, and source blocks, from which we identify four CARE criteria: Concurrency Control, Access Control and Safety Policies, Rules and Integrity, and Efficiency. The benchmark contains 1,058 manually annotated tasks across the four CARE dimensions, distilled from practitioner reports and instantiated on databases from Spider 2.0 and BIRD-Interact. CAREFUL was constructed by a nine-person annotation team, including four PhD-level annotators who led canonical scenario design and review and five experienced undergraduate annotators who supported task instantiation and verification, with three stages of rigorous quality control. Experiments with frontier LLM agents show that CAREFUL is challenging: GPT-5.5 obtains only a 33.9% Binary Pass Ratio, where a task passes only if all required operational criteria are satisfied, and a 73.3% Composite Score across the four CARE dimensions. These results show that strong text-to-SQL ability does not directly translate into production-ready database agency. Overall, CAREFUL provides a realistic testbed for evaluating and developing enterprise text-to-SQL agents that are reliable, policy-compliant, rule-compliant, and efficient in operational settings. Code and benchmark artifacts are provided for review and will be released publicly upon acceptance.
Show more
View full details
Poster

MaRiO: Multi-agent Collaborative Reasoning via Shared Observations in MLLMs

Nathaniel Redmond ⋅ Fidel Omar Tito Cruz ⋅ Devansh Sharma ⋅ Shehreen Azad ⋅ Sirshapan Mitra ⋅ Shruti Vyas ⋅ Yogesh Rawat
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A723
Multimodal Large Language Models (MLLMs) show strong visual perception capabilities, but existing evaluations largely focus on single-view or multi-view settings with shared camera parameters, leaving the challenge of {multi-agent collaborative} scenarios underexplored. In such settings, multiple agents observe a shared environment from independent viewpoints, and reliable decisions require reasoning across these independent observations of the same environment. To study this problem, we introduce **MaRiO**, a benchmark for evaluating multi-agent collaborative reasoning, and evaluate 23 state-of-the-art MLLMs on it. Our analysis reveals three key findings: (1) spatial tasks are generally easier than geometric reasoning tasks; (2) most models struggle to resolve cross-agent correspondences, with errors increasing with scene complexity; and (3) models show limited ability to leverage geometric cues present in the scene. As an initial step toward addressing these challenges, we explore geometry-guided synthetic view augmentation that generates intermediate transition frames between agent observations, providing auxiliary spatial context that improves performance in certain categories. Overall, our results highlight that multi-agent collaborative reasoning remains an open challenge and an important direction for multimodal embodied systems.
Show more
View full details
Poster

VeriWorld: A Verifiable Visual SWE-Bench for Spatial Reasoning in 3D Environments

Yan Zheng ⋅ Anthony Peng ⋅ XUYAO LIANG ⋅ Xiaoyan Cong ⋅ Nuo Chen ⋅ Bangya Liu ⋅ Zihan Wang ⋅ Wenyan Cong ⋅ Zhiwen Fan ⋅ Zhangyang &quot;Atlas&quot; Wang
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A724
Vision-language models (VLMs) are increasingly used for spatial tasks such as navigation and manipulation, where they must infer geometry and object relationships from visual input in order to act correctly. Existing benchmarks usually test perception and reasoning in isolation, making it hard to tell whether failures come from visual understanding, downstream reasoning, or the bridge between them. How can we evaluate whether a model can recover the right spatial structure from visual input and use it to make correct decisions? We introduce VeriWorld, a benchmark for spatial reasoning in interactive 3D environments with executable actions and deterministic verification. Each task is evaluated under three matched input settings -- visual-only, structured, and combined -- while the environment and objective remain fixed. This design isolates whether failures arise from perception, reasoning, or their interaction. VeriWorld brings together interactive 3D environments, code-based actions, deterministic verification, parameterized task generation, and controlled diagnostic evaluation in a single framework. Using this setup, we uncover two findings. First, models often succeed when given structured spatial information but fail when the same information must be inferred from visual input, revealing a recurring visual-to-structure gap. Second, action-space and harness design can change outcomes even under the same task and information condition, showing that interaction protocol is itself a confounding variable in spatial evaluation. VeriWorld is open-sourced to support reproducible and fine-grained evaluation of spatial reasoning in VLMs.
Show more
View full details
Poster

PhysicianBench: Evaluating LLM Agents in Real-World EHR Environments

Ruoqi Liu ⋅ Imran Mohiuddin ⋅ Austin J Schoeffler ⋅ Kavita Renduchintala ⋅ Ashwin Nayak ⋅ Prasantha L Vemu ⋅ Shivam Vedak ⋅ Kameron C Black ⋅ John L Havlik ⋅ Isaac Ogunmola ⋅ Stephen P. ⋅ Roopa Dhatt ⋅ Jonathan H Chen
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A701
We introduce PhysicianBench, a benchmark for evaluating LLM agents on physician tasks grounded in real clinical setting within electronic health record (EHR) environments. Existing medical agent benchmarks primarily focus on static knowledge recall, single-step atomic actions, or action intent without verifiable execution against the environment. As a result, they fail to capture the long-horizon, composite workflows that characterize real clinical systems. PhysicianBench comprises 100 long-horizon tasks adapted from real consultation cases between primary care physicians and specialists, with each task independently reviewed by a separate panel of physicians. Tasks are instantiated in an EHR environment with real patient records and accessed through the same standard APIs used by commercial EHR vendors. Tasks span 21 specialties (e.g., cardiology, endocrinology, oncology, psychiatry) and diverse workflow types (e.g., diagnosis interpretation, medication prescribing, treatment planning), requiring an average of 27 tool calls per task. Solving each task requires retrieving data across encounters, reasoning over heterogeneous clinical information, executing consequential clinical actions, and producing clinical documentation. Each task is decomposed into structured checkpoints (670 in total across the benchmark) capturing distinct stages of completion graded by task-specific scripts with execution-grounded verification. Across 12 proprietary and open-source LLM agents, the best-performing model achieves only 46% success rate (pass@1), while open-source models reach at most 19%, revealing a substantial gap between current agent capabilities and the demands of real-world clinical workflows. PhysicianBench provides a realistic and execution-grounded benchmark for measuring progress toward autonomous clinical agents.
Show more
View full details
Poster

ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videos

Wenliang Guo ⋅ Yu Kong
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A729
Procedural activities are fundamentally driven by object state transitions, yet existing instructional video benchmarks remain action-centric and cannot evaluate whether models reason about how objects evolve toward task completion. In this work, we introduce \textsc{ProcObject-10K}, the first benchmark that jointly evaluates object-centric reasoning and temporal evidence grounding in instructional videos, across both egocentric and exocentric views. It comprises 10,522 open-ended VideoQA pairs grounded in 1,799 video clips, spanning 137 tasks across 9 domains and five reasoning types covering preconditions, state evolution, counterfactuals, mistakes, and readiness. Benchmarking 13 leading MLLMs reveals a substantial answering-grounding gap: models produce plausible answers while failing to localize the supporting evidence (mIoU $<$ 45\%), exposing their reliance on linguistic priors rather than fine-grained object dynamics. As a step toward closing this gap, we further provide an object-centric supervised fine-tuning baseline with pseudo object-level supervision and spatial-temporal constraints. Models fine-tuned on \textsc{ProcObject-10K} not only improve on the benchmark itself, but also transfer effectively to other grounded VideoQA and embodied planning tasks. The dataset, annotations, and evaluation toolkit will be publicly released to support future research on object-centric procedural understanding.
Show more
View full details
Poster

VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models

Alex S Huang ⋅ Jiahui Zhang ⋅ Shiqing Tang ⋅ Yu Xiang
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A927
Vision-Language-Action (VLA) models have shown strong promise for general-purpose robotic manipulation, but their real-world evaluation remains limited by a lack of accessible, reproducible, and consistent benchmarks. Simulation benchmarks fail to capture real-world complexity, while existing real-world benchmarks often require expensive hardware, centralized evaluation, or are limited in task diversity. We introduce *VLA-REPLICA*, a low-cost, easily reproducible real-world benchmark for evaluating VLA models. Built from off-the-shelf components, our system can be quickly assembled and replicated across laboratories, providing a consistent environment for policy evaluation anywhere in the world. *VLA-REPLICA* includes a diverse suite of manipulation tasks and a small-scale demonstration dataset for target-domain adaptation, with real-world evaluation protocols for both in-distribution and out-of-distribution settings. Experiments with imitation learning and state-of-the-art VLA models reveal model strengths and limitations, while consistent results across independently constructed setups demonstrate the reproducibility of our benchmark.
Show more
View full details
Poster

BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces

Liangwei Yang ⋅ Jielin Qiu ⋅ Zixiang Chen ⋅ Ming Zhu ⋅ Juntao Tan ⋅ Zhiwei Liu ⋅ Wenting Zhao ⋅ Zhujun Lan ⋅ Akshara Prabhakar ⋅ Silvio Savarese ⋅ Huan Wang ⋅ Shelby Heinecke
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A711
Many decision-support systems require adapting to individual users, yet evaluating this capability remains challenging because user preferences and beliefs are rarely stated explicitly. Instead, they are revealed through sequences of real-world behavior. Existing benchmarks for user understanding often rely on textual personas, constructed preferences, or simulated users, which may not faithfully reflect how people actually decide.We introduce \textsc{BehaviorBench}, a benchmark for modeling user decisions from real-world behavioral traces. Built from public prediction-market and on-chain records, the benchmark reconstructs longitudinal decision histories and evaluates whether models can infer how a particular user believes and acts from prior behavior. It defines two complementary task layers: \emph{Belief prediction}, which infers a user's final revealed stance and confidence, and \emph{Trade prediction}, which predicts the direction and magnitude of individual actions. We evaluate frontier and open-weight generative models under multiple history representations, including direct behavioral history, generated user profiles, and retrieved cross-user evidence. Results show that personalization is not a single capability: models benefit differently from different representations depending on the task, and performance remains limited when reasoning over implicit behavioral signals. \textsc{BehaviorBench} provides an evaluation setting for studying personalized decision modeling grounded in real-world behavioral evidence rather than simulated users alone.
Show more
View full details
Poster

MaterialsSaddles: 34 Million Transition States and a Flow-Matching Saddle-Point Predictor for Materials

Ilgar Baghishov ⋅ Sung Hoon Jung ⋅ Graeme Henkelman
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A708
Transition state (TS) search is the rate-limiting step in high-throughput kinetics screening. Generative models have been used to accelerate TS prediction for molecules, but extending them to periodic materials has been blocked by the absence of a large-scale materials TS dataset. We release MaterialsSaddles, the first such dataset: 34.14 M (reactant, saddle, product) triplets generated with the state-of-the-art UMA-S-1.2 universal interatomic potential across four subsets, comprising 31.35 M triplets over LeMat-Bulk's 5.34 M-structure deduplication of Materials Project, OQMD, and Alexandria; 2.59 M triplets over OC20 surfaces; 167 k triplets over OC22 oxide electrocatalysts; and 35 k climbing-image nudged elastic band (CI-NEB) paths over Materials Project battery cells. The release totals 102.4 M structures (687 GB) in ASE-LMDB shards with stratified 90/5/5 splits, generated on a 500,000 GPU-hour HPC pipeline. Every triplet is connected by construction via double-minimization from the converged saddle, and every frame carries per-frame metadata including eigenmodes, curvatures, bond-change diffs, and source identifiers. As a working demonstration of the dataset's intended use, we release SaddleFlow, a flow-matching reactant-and-product-conditional saddle-point generator built on a UMA-S-1.2 backbone with FiLM-based time conditioning and an SO(3)-equivariant velocity head, trained on the mp20bat subset of MaterialsSaddles. In a pilot DFT study, we show that dimer searches initialized from a sample of mp20bat test triplets converge substantially faster and more reliably than those initialized from the standard reactant-product midpoint guess, indicating that MaterialsSaddles and SaddleFlow together offer a practical path to scaling DFT-level transition-state searches. MaterialsSaddles (CC-BY 4.0), SaddleFlow (MIT), and the data-generation engine SaddleMill (MIT) are publicly released.
Show more
View full details
Poster

EVALUATION CARDS: An Interpretive Layer for AI Evaluation Reporting

Avijit Ghosh ⋅ Anka Reuel-Lamparth ⋅ Wm. Matthew Kennedy ⋅ Jenny Chim ⋅ Yilin Huang ⋅ Srishti Yadav ⋅ Anastassia Kornilova ⋅ Damian Stachura ⋅ Kevin Klyman ⋅ Jennifer Mickel ⋅ Felix Friedrich ⋅ Max Lamparth ⋅ Jan Batzner ⋅ Anoop Mishra ⋅ Jeba Sania ⋅ Eliya Habba ⋅ Nathan Heath ⋅ Shalaleh Rismani ⋅ Usman Gohar ⋅ Andrea Loehr ⋅ David Manheim ⋅ Ruchira Dhar ⋅ Sree Harsha Nelaturu ⋅ Aarush Sinha ⋅ Leshem Choshen ⋅ Yixiong Hao ⋅ Yanan Long ⋅ Andrew Tran ⋅ Drishti Sharma ⋅ Ishan Khire ⋅ Amit Saha ⋅ Subramanyam Sahoo ⋅ Michael Hardy ⋅ Michael A. Riegler ⋅ Kabir Manghnani ⋅ Michelle Lin ⋅ Nancy Jiang ⋅ Asaf Yehudai ⋅ Nuno Moniz ⋅ Yacine Jernite ⋅ Zeerak Talat ⋅ Stella Biderman ⋅ Mubashara Akhtar ⋅ Jessica Ji ⋅ Aris Hofmann ⋅ Mykel J Kochenderfer ⋅ Sanmi Koyejo ⋅ Irene Solaiman
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A914
AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. This fragmentation is borne at the level of interpretation: readers cannot reliably compare results across sources, identify what is missing from a given report, or trace an aggregate claim to the evidence behind it. Recent efforts address isolated components of this problem but leave three gaps unresolved: They cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record. They specify static representations that do not differentiate between the questions technical and policy readers bring to the same evidence. And they remain proposals on paper, without the extraction infrastructure required for adoption at scale. We present EVALUATION CARDS, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured literature review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and policy audiences, and (3) provide a deployed monitoring tool that applies EVALUATION CARDS across 5,498 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.
Show more
View full details
Poster

CogBench: Evaluating Cognitive-Level Control in LLM Question Generation

Mourya Teja Kunuku
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A809
Educational LLM systems generate assessment questions on demand and depend on the assumption that a model can be controlled to a specified cognitive level. We test the assumption. CogBench is a benchmark for cognitive-level control in LLM question generation, grounded in the Revised Bloom's Taxonomy. Its central design choice is a two-mode protocol that separates cognition from imitation: in standard mode the model is asked to generate at level L_t; in adversarial mode it must do so while restricted to verbs from a different level L_v, demonstrating the requested level structurally rather than lexically. CogBench spans 8 subjects, 6 Bloom levels, and 120 CC-BY OpenStax passages, with ~17,000 graded generations across 13 models (six frontier closed APIs and seven open-source local models), each scored by a deliberately dual evaluator: a 28-rule constraint checker, and the Cognitive Complexity Score (CCS), a fine-tuned BERT classifier at 82.0% exact accuracy that outperforms a four-model LLM-as-judge panel by 7.8 points. We find that (1) adversarial vocabulary collapses strict adherence by 24-56 pp across the 12 models above floor, with the strongest standard performer suffering the largest collapse; (2) Remember-level adherence under adversarial vocabulary is essentially 0% in all 13 models, suggesting Remember lives almost entirely in its surface verbs; (3) the two evaluators rank models differently, including on the leader, a divergence we argue benchmarks should report rather than hide; and (4) the most adversarially robust models on CCS are not frontier flagships but gpt-4o-mini (46.8%) and llama3.1-8b (45.6%). What current LLMs offer educational systems looks like cognitive control and is largely verb-level imitation that resembles it. We release the dataset (CC-BY 4.0; Croissant 1.0 with Responsible-AI fields), the evaluation harness, and a 13-model leaderboard at https://huggingface.co/spaces/cogbench-anonymous/cogbench-leaderboard.
Show more
View full details
Poster

LIBERO-PeRM: Benchmarking Personalized Robotic Manipulation

Zhixu Li ⋅ Keqian Tang ⋅ Litian Gong ⋅ Jingyu Yao ⋅ Wenqian Zhang ⋅ Tianze Xu ⋅ Zehao Wang ⋅ Yuping Wang ⋅ Junge Zhang ⋅ Jiachen Li
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A925
Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic manipulation has been extensively studied, personalized robotic manipulation (PeRM) lacks a unified formulation and remains significantly underexplored. In this paper, we identify user preference as the cornerstone of PeRM and introduce LIBERO-PeRM, a novel benchmark designed for personalized preference learning in embodied agents. Specifically, LIBERO-PeRM highlights five key research topics in personalization: 1) how policies follow preferences in zero-shot settings; 2) how to learn single preferences from demonstrations; 3) how efficiently policies acquire preferences as demonstrations increase; 4) how learned preferences generalize across layouts and preference settings; and 5) how policies compose multiple preferences and resolve conflicts. To this end, we propose a unified formulation of personalized manipulation with three preference levels that jointly cover common daily personalization scenarios. Building on this formulation and the LIBERO framework, we develop an extensible procedural generation pipeline that generates paired demonstrations with and without preferences. For benchmarking purposes, we create five task suites (312 tasks in total) with corresponding high-quality demonstrations, rich metadata, and preference satisfaction metrics to probe the aforementioned research topics. Experimental results show that current VLA policies struggle with zero-shot personalization, while SFT makes preferences learnable; preference learning benefits from mixed training, broader layout diversity, and same-layout no-preference data, but generalization, decomposition, and conflict resolution remain difficult.
Show more
View full details
Poster

PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection

Cheng Wan ⋅ Yusu Qian ⋅ Qingyu Zhao
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A728
Multimodal large language models (MLLMs) have achieved remarkable progress on vision–language tasks, yet their reasoning processes remain sometimes unreliable. We introduce PRISM-Bench, short for Puzzle Reasoning with In-Sequence Mistakes, a benchmark of puzzle-based visual challenges designed to evaluate not only whether models can solve problems, but how their reasoning unfolds. Unlike prior evaluations that measure only final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error, models must identify the first incorrect step. This setting enables fine-grained assessment of logical consistency, error detection, and visual reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric, and analogical reasoning, resisting shortcuts based on superficial pattern matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap between fluent generation and faithful reasoning: models that produce plausible CoTs often fail to locate simple logical faults. By disentangling answer generation from reasoning verification, PRISM-Bench offers a sharper lens on multimodal reasoning competence and underscores the need for diagnostic evaluation protocols in the development of trustworthy MLLMs.
Show more
View full details
Poster

SALART-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images

Xiaoxiao Sun ⋅ Ruotian Zhang ⋅ Junzhe Huang ⋅ James Burgess ⋅ Serena Yeung-Levy
Dec 12, 2:00 AM - 5:00 AM Hall C1 #A725
Vision-language models (VLMs) are increasingly used to detect whether AI-generated images contain visible artifacts, yet their ability to analyze such artifacts remains poorly understood. A correct image-level decision can still hide important failures: a model may correctly flag an artifact while relying on the wrong visual cue, selecting the wrong region, or describing a defect that the image does not support. To evaluate these behaviors directly, we introduce SalArt-VQA, a diagnostic benchmark for fine-grained salient artifact understanding in AI-generated images. SalArt-VQA contains 950 images and 3,681 human-authored multiple-choice questions spanning artifact images, matched real reference images, and paired generated reference images. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, while the reference splits test calibration and abstention when the annotated defect is absent. Across 20 VLMs, SalArt-VQA reveals failures that image-level detection accuracy hides: the strongest model reaches 99.37% detection recall on artifact images but answers all four artifact-side questions correctly on only 53.26% of images. Comparing artifact images with defect-absent references reveals a sensitivity–calibration tradeoff: sensitive models often make unsupported artifact claims, while conservative models avoid false alarms largely by missing real artifacts. These results show that high artifact detection accuracy alone does not imply grounded artifact understanding. SalArt-VQA exposes these hidden failure modes and provides a fine-grained evaluation of whether VLM artifact claims are supported by local visual evidence.
Show more
View full details
Poster

Semantic Search over 9 Million Mathematical Theorems

Luke Alexander ⋅ Eric Leonen ⋅ Sophie Szeto ⋅ Artemii Remizov ⋅ Ignacio Tejeda ⋅ Jarod Alper ⋅ Giovanni Inchiostro ⋅ Vasily Ilin
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A120
Searching for mathematical results remains difficult: most existing tools retrieve entire papers, while mathematicians and theorem-proving agents often seek a specific theorem, lemma, or proposition that answers a query. While semantic search has seen rapid progress, its behavior on large, highly technical corpora such as research-level mathematical theorems remains poorly understood. In this work, we introduce and study semantic theorem retrieval at scale over a unified corpus of 9.2 million theorem statements extracted from arXiv and seven other sources, representing the largest publicly available corpus of human-authored, research-level theorems. We represent each theorem with a short natural-language description as a retrieval representation and systematically analyze how representation context, language model choice, embedding model, and prompting strategy affect retrieval quality. On a curated evaluation set of theorem-search queries written by professional mathematicians, our approach substantially improves both theorem-level and paper-level retrieval compared to existing baselines, demonstrating that semantic theorem search is feasible and effective at web scale. The project page, search tool, dataset, REST API, and MCP server are available at [Redacted]
Show more
View full details
Poster

JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

Russell Yang ⋅ Ruishi Chen ⋅ Pierce Kelaita ⋅ Riya Ranjan ⋅ Sibo Ma ⋅ Charles Dickens ⋅ Matthew Guillod ⋅ Megan Ma ⋅ Julian Nyarko
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A121
Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys—including at major U.S. law firms—with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts on the same items. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics (mean Spearman's rank correlation of 0.908 vs. 0.150, $\widehat{D}=0.758$ [0.494, 1.021]) while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.
Show more
View full details
Poster

Cross-Dialect Generalization Without Retraining: Benchmarks and Evaluation of Schema-Derived Constrained Decoding for MLIR

Plawan Kumar Rath
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A118
Multi-Level Intermediate Representation (MLIR) underlies modern ML compiler infrastructure, including TensorFlow, JAX via StableHLO, PyTorch's Inductor backend, and IREE. Yet it appears only in trace amounts in code-LM pretraining corpora. MLIR is also extensible by design: new dialects ship per application domain, so maintaining a fine-tuned model per dialect does not scale. We ask whether inference-time priors derived mechanically from each dialect's Operation Definition Specification (ODS) can substitute for gradient-based adaptation. We make two contributions. First, we release four natural-language-to-MLIR benchmarks across three dialects (MLIR-Spec-150, Linalg-Spec-30, StableHLO-Spec-30, and StableHLO-Held-Out-200), totaling 410 in-scope NL→MLIR pairs, plus a 25-program StableHLO-OutOfGrammar stress set (435 benchmark instances). Thirty of the Spec prompts, ten per dialect, also carry hand-authored input and expected-output references for functional testing (MLIR-Functional-Reference-30). All artifacts ship under Apache-2.0 with Gebru datasheets and Croissant 1.1 metadata. Second, on top of these benchmarks we build a three-layer schema-derived constraint stack: a context-free grammar over op signatures (C1), type-domain splits from an ODS-extracted type lattice (C2), and an SSA-scope validator driving five-retry rejection sampling (C3). Porting the stack from arith+func+memref+linalg to StableHLO required no new constraint-layer code. Empirically, on dialects whose verifier semantics are dominated by structural constraints, SmolLM2-1.7B with token-level schema-derived constraints matches or exceeds 15B–34B code LMs that run the strongest constraint enforcement their serving stack admits (post-hoc rejection), at a matched retry budget and 8–25× the per-generation speed: on linalg it reaches 80.0% verify-valid (three runs, each 80.0%, n=125), 21 to 44 pp above CodeLlama-34B, Granite-Code-34B, and StarCoder2-15B with non-overlapping CIs, and the lead survives a same-family fp16 precision control. The comparison is between systems, not models: under the baselines' post-hoc protocol, SmolLM2 itself falls from 52.0% to 24.5% on arith+func, a +27.5 pp masking contribution. On arith+func and on the templated parametric StableHLO-Held-Out-200, where verifier semantics turn on attribute values rather than structure, the same baselines match or beat the SLM; we scope these explicitly as non-win cells. We release benchmarks, decoder, every per-prompt generation, and a pinned reproducibility Dockerfile.
Show more
View full details
Poster

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

Chaithanya Bandi ⋅ Razvan-Gabriel Dumitru ⋅ Ben Hertzberg ⋅ Divyansh Agarwal ⋅ Geobio Boo ⋅ Tejas Polakam ⋅ Jeff Da ⋅ HiJae Kim ⋅ Vipul Gupta ⋅ Manasi Sharma ⋅ Andrew Park ⋅ Martin Villanueva Dimakis ⋅ Ernesto G Montoya ⋅ Rafael Cruz ⋅ MohammadHossein Rezaei ⋅ Chetan Rane ⋅ Daniel Y Zhang ⋅ Brad Kenstler ⋅ Bing Liu
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A115
The Model Context Protocol (MCP) is emerging as a standard interface through which large language model (LLM) agents discover and invoke external tools. However, existing MCP evaluations fall short along three key axes: realistic multi-step workflows with cross-server orchestration, breadth across authentic MCP servers rather than mocks, and structured, reproducible claim-level scoring disentangled from agent verbosity or style. We introduce MCP-Atlas, a benchmark for measuring tool-use competency against production MCP servers. MCP-Atlas contains 1,000 natural-language tasks written and verified by human experts spanning 36 real MCP servers and 220 tools. Prompts do not specify servers, tools, or parameters, requiring agents to identify relevant tools among semantically plausible distractors and to compose multi-step, cross-server workflows. Each task is scored with a claim-level rubric, where final answers are scored against atomic factual claims grounded in tool outputs. This answer-centric scoring permits valid alternative tool-call trajectories to receive credit. We pair this with an 11-category diagnostic taxonomy that disentangles tool-call failures from cognitive failures in task understanding, synthesis, parsing, and stopping. Evaluating 20 frontier models from six providers under matched task-level conditions, we find pass rates up to 82.2% at a 0.75 claim coverage threshold and a clear three-tier performance structure. Automated diagnostics show that 63.3% of diagnosed failures are cognitive rather than tool-call related. Notably, several high-performing models fail after successful tool execution due to premature stopping or incorrect synthesis. We release the task schema, containerized harness, claim evaluator, and a 500-task public split, while reserving a 500-task private split to preserve leaderboard integrity.
Show more
View full details
Poster

RTEB: An Overfitting-Resistant Benchmark for Embedding Model Evaluation

Sahil Verma ⋅ Minghan Li ⋅ Andrew Gaut ⋅ Yujie Qian ⋅ Kaidi Cao ⋅ Zhenmei Shi ⋅ Kenneth Enevoldsen ⋅ Roman Solomatin ⋅ Isaac Chung ⋅ Tom Aarsen ⋅ Apoorva Joshi ⋅ Emilia Garcia-Casademont ⋅ Michael Günther ⋅ Niklas Muennighoff ⋅ Yevhen Kostiuk ⋅ Fődi Zoltán ⋅ Tengyu Ma ⋅ Frank Liu
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A107
Embedding models are central to modern information retrieval, retrieval-augmented generation, and downstream NLP systems, yet their evaluation remains unreliable: widely used benchmarks such as BEIR and MTEB rely on fully public datasets, making them vulnerable to overfitting that inflates scores without corresponding gains in real-world generalization. We introduce RTEB, a retrieval-focused embedding benchmark designed to address these limitations. Unlike MTEB's broad multi-task scope, RTEB is purpose-built for retrieval -- the setting most critical to RAG pipelines and agentic systems. RTEB emphasizes underrepresented yet high-impact domains, particularly code and agentic code retrieval, and keeps a large portion of its datasets private to structurally prevent direct test-set overfitting. We evaluate 20 models on 48 datasets (20 public, 28 private) spanning six domains and four languages. Rankings on RTEB diverge substantially from BEIR and MTEB ($\rho = 0.38$), agentic code retrieval emerges as the hardest domain with the best model reaching only 0.48 NDCG@10, and domain-specialized models fail to outperform strong generalists on any domain. Together, these design choices and findings position RTEB as a more reliable, discriminative, and future-proof benchmark for embedding model evaluation.
Show more
View full details
Poster

AI GAMESTORE: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games

Lance Ying ⋅ Ryan Truong ⋅ Prafull Sharma ⋅ Kaiya Zhao ⋅ Nathan Cloos ⋅ Kelsey Allen ⋅ Tom Griffiths ⋅ Katie Collins ⋅ Jose Hernandez-Orallo ⋅ Phillip Isola ⋅ Samuel J Gershman ⋅ Josh Tenenbaum
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A100
Rigorously evaluating machine intelligence against the broad spectrum of human general intelligence has become increasingly important and challenging in this era of rapid technological advance. Conventional AI benchmarks typically assess only narrow capabilities in a limited range of human activity. Most are also static, quickly saturating as developers explicitly or implicitly optimize for them. We propose that a more promising way to evaluate human-like general intelligence in AI systems is through a particularly strong form of general game playing: studying how and how well they play and learn to play all conceivable human games, in comparison to human players with the same level of experience, time, or other resources. We define a “human game” to be a game designed by humans for humans, and argue for the evaluative suitability of this space of all such games people can imagine and enjoy — the “Multiverse of Human Games”. Taking a first step towards this vision, we introduce the AI GAMESTORE, a scalable and open-ended platform that uses LLMs with humans-in-the-loop to synthesize new representative human games, by automatically sourcing and adapting standardized and containerized variants of game environments from popular human digital gaming platforms. As a proof of concept, we generated 100 such games based on the top charts of Apple App Store and Steam, and evaluated seven frontier vision-language models (VLMs) on short episodes of play. The best models achieved less than 10% of the human average score on the majority of the games, and especially struggled with games that challenge world-model learning, memory and planning. We conclude with a set of next steps for building out the AI GAMESTORE as a practical way to measure and drive progress toward human-like general intelligence in machines.
Show more
View full details
Poster

SOL-ExecBench: Speed-of-Light Benchmarking for Real-World GPU Kernels Against Hardware Limits

Edward C Lin ⋅ Sahil Modi ⋅ Siva Kumar Sastry Hari ⋅ Qijing Huang ⋅ Zhifan Ye ⋅ Nestor Y. Qin ⋅ Fengzhe Zhou ⋅ Yuan Zhang ⋅ Jingquan Wang ⋅ Sana Damani ⋅ Dheeraj Peri ⋅ Ouye Xie ⋅ Aditya Kane ⋅ Michael behar ⋅ Rishabh Mehta ⋅ Vartika Singh ⋅ Vikram Sharma Mailthody ⋅ Terry Chen ⋅ Zihao Ye ⋅ Hanfeng Chen ⋅ Tianqi Chen ⋅ Vinod Grover ⋅ Wei Chen ⋅ Wei Liu ⋅ Eric Chung ⋅ Luis Ceze ⋅ Roger Bringmann ⋅ Michael Lightstone ⋅ Christos Kozyrakis ⋅ Humphrey Shi
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A409
As agentic AI systems become increasingly capable of generating and optimizing GPU kernels, progress is constrained by benchmarks that reward speedup over software baselines rather than proximity to hardware-efficient execution. We present SOL-ExecBench, a benchmark of 235 CUDA kernel optimization problems extracted from 124 production and emerging AI models spanning language, diffusion, vision, audio, video, and hybrid architectures, targeting NVIDIA Blackwell GPUs. The benchmark covers forward and backward workloads across BF16, FP8, and NVFP4, including kernels whose best performance is expected to rely on Blackwell-specific capabilities. Unlike prior benchmarks that evaluate kernels primarily relative to software implementations, SOL-ExecBench measures performance against analytically derived Speed-of-Light (SOL) bounds computed by SOLAR, our pipeline for deriving hardware-grounded SOL bounds, yielding a fixed target for hardware-efficient optimization. We report a SOL Score that quantifies how much of the gap between a release-defined scoring baseline and the hardware SOL bound a candidate kernel closes. To support robust evaluation of agentic optimizers, we additionally provide a sandboxed harness with input correctness checks, isolated compilation and execution, CUPTI-based GPU activity timing, and defenses against common reward-hacking strategies. A third-party post-release evaluation by Cursor reported a $1.38\times$ geometric-mean speedup over a single-agent optimized PyTorch baseline and an overall SOL Score of 0.56 across the full benchmark. SOL-ExecBench reframes GPU kernel benchmarking from beating a mutable software baseline to closing the remaining gap to hardware Speed-of-Light.
Show more
View full details
Poster

PhyMetric: Diagnosing Physical Plausibility in Text-to-Video Generation via Scene-Level QA

Siwei Meng ⋅ Xuerou Cai ⋅ Rui Hu ⋅ Ping Liu
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A504
Text-to-video (T2V) generation models can produce visually realistic videos that nevertheless violate the physical consequences implied by the prompt and scene. Existing benchmarks largely rely on holistic plausibility scores or broad quality dimensions, leaving it unclear which physical behavior is being tested and why a model fails. We introduce PhyMetric, a scene-level diagnostic benchmark that recasts physical plausibility evaluation as verifiable, scene-grounded question answering. PhyMetric contains 1115 scenes spanning 50 physical categories across six domains, paired with 8920 balanced binary questions over 12 evaluation dimensions. Each scene is associated with eight targeted yes/no questions and a structured physical prior specifying the governing law, expected temporal sequence, observable signatures, and common failure modes. At evaluation time, a vision-language model (VLM) judge answers each question from question-aware sampled frames conditioned on the prior, and correctness is hierarchically aggregated into scene, dimension, physical category, and model scores. Experiments on four open-source T2V models and three VLM judges show that scene-specific binary QA is a necessary component of the protocol: when each targeted question is replaced by a single generic plausibility query, evaluator accuracy drops to the chance level. Structured physical priors offer a complementary signal that improves human alignment of the resulting scores, with effects that vary across VLM judges of different capacity. These findings establish PhyMetric as a benchmark that turns physical plausibility evaluation from opaque holistic scoring into interpretable, scene-level inspection.
Show more
View full details
Poster

Streaming Interventions: Can Video LLMs Correct Mistakes as They Occur?

Apratim Bhattacharyya ⋅ Shweta Mahajan ⋅ Sanjay Haresh ⋅ Reza Pourreza ⋅ Litian Liu ⋅ Rajeev Yasarla ⋅ Risheek Garrepalli ⋅ Roland Memisevic
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A614
Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) large language models (LLMs) as task guidance assistants. A crucial capability of the real-world success of a prospective task guidance assistant is its ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating *reactive, step-by-step* task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state‑of‑the‑art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non‑interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.
Show more
View full details
Poster

RAVEN-Bench: A Paired EO-IR Video QA Benchmark for Aerial Multimodal Understanding

Yu Hu ⋅ Jianyang Gu ⋅ Yue Cao ⋅ Hao Liu ⋅ Kangnan Wang ⋅ Jozsef Hamari ⋅ Zheng Liu ⋅ Mohsen Zardadi
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A615
Multimodal large language models (MLLMs) have advanced rapidly on image and video understanding, but their ability to reason over paired aerial electro-optical and infrared (EO-IR) videos remains underexplored. Such videos contain small targets, sparse temporal evidence, changing viewpoints, and modality-dependent cues that are poorly captured by RGB-centric benchmarks. We introduce RAVEN-Bench (Reasoning over Aerial Videos with EO-IR aligNment), a paired EO-IR video QA benchmark for low-altitude aerial multimodal understanding. RAVEN-Bench contains 48 temporally aligned EO-IR video pairs and 576 human-verified four-way multiple-choice questions. It supports EO-only, IR-only, and paired EO+IR evaluation, and uses hierarchical capability levels and structured question groups to diagnose cross-modal gain, modality reliance, consistency, and reasoning coherence. We evaluate representative frontier and open-weight MLLMs, showing that aggregate performance can overestimate reliable EO-IR reasoning. RAVEN-Bench provides a focused testbed for assessing whether progress on RGB video benchmarks transfers to aerial cross-spectral understanding.
Show more
View full details
Poster

PerturbReason: A Knowledge-Grounded Benchmark and Framework for Cell-State–Conditioned Mechanistic Reasoning of Perturbation Effects

Dongkwan Kim ⋅ Yiming Gao ⋅ Yining Yang ⋅ Yang Shen
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A625
Evaluating machine learning in scientific domains requires separating correct predictions from correct reasons under realistic distribution shifts. We introduce PerturbReason, a knowledge‑grounded benchmark for cell‑state--conditioned reasoning about perturbation effects. It tests the models to generate mechanistically faithful explanations and assesses their robustness against complex shifts, such as new cells, unseen perturbations, and cross-modal or combinatorial extrapolations. PerturbReason combines single‑cell genetic and chemical perturbation data across multiple cell lines with knowledge graphs, and dynamically conditions pathways on cell‑specific basal states to avoid generic memorization. Evaluations on state-of-the-art models reveal systematic gaps between predictive accuracy and mechanistic reasoning. Specifically, these models exhibit failure modes largely invisible to standard benchmarks, such as deriving correct answers through flawed logic, ignoring cellular context, and generating directionally inconsistent mechanisms. As a reference probe of the benchmark, we present PerturbRM, a large language model trained to align outcome predictions with context-specific regulatory reasoning. PerturbReason thus provides a rigorous diagnostic benchmark for studying and improving faithful, generalizable reasoning in data-rich scientific systems.
Show more
View full details
Poster

A Multimodal Benchmark for Evaluating Cause-of-Death Inference Using Child Health and Mortality Data

Junhe Yang ⋅ Soumyakanti Pan ⋅ Hyun Seung Lim ⋅ YUE CHU ⋅ Yuting Guo ⋅ Nishtha Agarwal ⋅ Varun Babbar ⋅ Gaurav Rajesh Parikh ⋅ Yiqun Chen ⋅ Chris A Rees ⋅ Ziyaad Dangor ⋅ Sanjay G Lala ⋅ Zehang Li ⋅ Samuel J Clark ⋅ Zhenke Wu ⋅ Abhirup Datta ⋅ Li Liu ⋅ Cynthia Rudin ⋅ Samuel Scarpino ⋅ Benjamin M Gyori ⋅ Tyler H. McCormick
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A729
Accurately attributing causes of death is vital for global health, yet fewer than 5% of deaths in resource-constrained regions are medically certified. To assign causes to these unlabeled deaths at scale, practitioners traditionally rely on verbal autopsy, using supervised statistical models to classify based on structured survey data. However, modern mortality surveillance increasingly collects rich, unstructured multimodal data, such as free-text caregiver narratives and postmortem diagnostics, which traditional supervised statistical models struggle to seamlessly integrate. In this paper, we present a comprehensive, multimodal benchmark for cause-of-death classification using data from the Child Health and Mortality Prevention Surveillance (CHAMPS) network, a unique surveillance platform spanning nine countries across South Asia and Sub-Saharan Africa. Using this dataset, we introduce an evaluation framework designed to rigorously assess diagnostic reasoning, moving beyond traditional metrics that fail to capture complex clinical realities. We demonstrate the utility of this benchmark by evaluating zero-shot large language models against supervised baselines across various data modalities. Our results reveal distinct differences in how these modeling approaches synthesize unstructured medical evidence. This benchmark provide a rigorously defined resource for assessing clinical reasoning in next-generation mortality surveillance.
Show more
View full details
Poster

MOVEBENCH: A Benchmark for Global-Scale Wildlife Movement Forecasting

Justin Kay ⋅ Shir Bar ⋅ Ellen O Aikens ⋅ Martin Becker ⋅ Francesca Cagnacci ⋅ Juliet Cohen ⋅ Scott W Forrest ⋅ Jessica Kendall-Bar ⋅ Madeleine Lucas ⋅ Macon Overcast ⋅ Meredith S Palmer ⋅ Will Rogers ⋅ Nicholas J Russo ⋅ Christian Rutz ⋅ Larissa T Beumer ⋅ Michael B Brown ⋅ Ying-Chi Chan ⋅ Sarah C Davidson ⋅ Diego E Soto ⋅ Anne G Hertel ⋅ Roland Kays ⋅ Benjamin Koger ⋅ Guram Mikaberidze ⋅ Thomas Mueller ⋅ Ruth Oliver ⋅ Thorsten Papenbrock ⋅ Robert Patchett ⋅ Jared A Stabach ⋅ Dane Taylor ⋅ Scott W Yanco ⋅ Sara Beery
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A700
Understanding and predicting wildlife movement is critical for ecology and conservation. While trajectory forecasting has advanced for human and vehicle movement, wildlife trajectories present distinct challenges: they are unconstrained in space, highly stochastic, and influenced by environmental conditions. We introduce MOVEBENCH, the first large-scale benchmark for probabilistic wildlife movement forecasting, containing 2.6M GPS locations from 800+ individuals across 110 species in 127 countries, paired with 1.6B environmental raster tiles capturing 160 covariates known or hypothesized to influence movement. We propose a probabilistic evaluation protocol for movement trajectory forecasts, addressing limitations of point-prediction metrics for inherently stochastic phenomena. Through comprehensive empirical evaluation of four method families across multiple temporal and spatial scales, we reveal that: (1) existing predictive methods generalize better to future timepoints than to unseen individuals, (2) deep learning approaches do not consistently outperform simpler baselines, and (3) environmental covariate selection significantly impacts performance. MOVEBENCH enables standardized evaluation of movement forecasting methods and provides a foundation for methodological advances on this ecologically important task.
Show more
View full details
Poster

CultureRed: Benchmarking Culture-Specific AI Safety Based on Global Statutes

Xun Liu ⋅ Mintong Kang ⋅ Seok Min Lim ⋅ Ong C Hui ⋅ Bo Li
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A813
Safety alignment and evaluation of LLMs has become increasingly difficult due to the diverse definitions of harmful behavior across cultures. Existing safety benchmarks are largely culture-agnostic or built on ad-hoc taxonomies, and mainly evaluate refusals for prohibited requests while overlooking legally defined exceptions, making over-refusal difficult to quantify. To address these gaps, we introduce a large-scale culture-aware data synthesis pipeline and CultureRed as the first global culture-specific AI safety benchmark based on 252 legal statutes across 15 countries. Based on legal statutes, our pipeline automatically transforms statutes into actionable policy rules that cover both prohibitions and legally defined exceptions, cluster policy rules into risk topics and fine-grained categories, and generates policy rule-conditioned data that models should either refuse or comply with. The resulting benchmark contains 54,470 policy rule--prompt pairs spanning 6,256 fine-grained risk categories. We further run a human study with government experts to validate the dataset quality. Leveraging CultureRed, we conduct a comprehensive evaluation for 14 advanced generative LLMs and 16 frontier guardrail LLMs, unlocking a range of findings, such as (1) Culture-specific safety performance varies substantially, with EU presenting the most challenging for generative models (average Harm Rate of 39.4%); (2) Generative model series evolution exhibits divergent patterns, where GPT series achieves balanced alignment (5 times Harm Rate reduction while maintaining Help Rate above 90%), but Claude series shows safety-helpfulness tradeoff (5 times Harm Rate reduction but Help Rate drops from 61.9% to 49.9%); (3) Guardrail models systematically under-guard unsafe content, with high false positive rate.
Show more
View full details
Poster

Agents' Last Exam

Yiyou Sun ⋅ Xinyang Han ⋅ Weichen Zhang ⋅ Yuanbo Pang ⋅ Tianyu Wang ⋅ Yuhan Cao ⋅ Yixiao Huang ⋅ Christopher Duroiu ⋅ Haoyun Zhang ⋅ Jeffrey Lin ⋅ Weishu Zhang ⋅ Tianren Zeng ⋅ Ying Yan ⋅ Bo Liu ⋅ Hanson Wen ⋅ Mingyang Xu ⋅ Xiaoyuan Liu ⋅ Zimeng Chen ⋅ Weiyan Shi ⋅ Amanda Dsouza ⋅ Vincent Chen ⋅ Yushan Li ⋅ Wenxi Deng ⋅ Huiqi Wang ⋅ Justin Xu ⋅ Tao Sun ⋅ Zhun Wang ⋅ Chris Liu ⋅ Yafei Cheng ⋅ Rongwang Hu ⋅ Aras Bacho ⋅ Shengcao Cao ⋅ Zengyi Qin ⋅ Yixiong Chen ⋅ Hengduan Fan ⋅ Hao Liu ⋅ Lin Zeng ⋅ Shashank M Bharadwaj ⋅ Litian Gong ⋅ Maojia Song ⋅ Ruheng Wang ⋅ Zongzheng Zhang ⋅ Jianhong Tu ⋅ Zhonghua Wang ⋅ Zheng Zhang ⋅ Zijiao Chen ⋅ Yanqiong Jiang ⋅ Zhendong Li ⋅ Bohan Lyu ⋅ Ma Chang ⋅ Peiran Xu ⋅ Benran Zhang ⋅ Shangding Gu ⋅ Haoyue Hua ⋅ Haoyang Li ⋅ Wanzhe Liao ⋅ Chengzhi Liu ⋅ Junbo Peng ⋅ Zechen Xu ⋅ Bo Chen ⋅ Jiayi Cheng ⋅ Yi Jiang ⋅ Maggie Kuang ⋅ Yuan Li ⋅ Youbang Pan ⋅ Ziyan Rao ⋅ Alexander Schubert ⋅ Ivan Shen ⋅ Vincent Siu ⋅ Xiatao Sun ⋅ Xiaopan Zhang ⋅ Yuchen Zhu ⋅ Ishaan S Chandok ⋅ Lei Ding ⋅ Jingxuan Fan ⋅ Andrew Glover ⋅ Jiaming Hu ⋅ Yiran HU ⋅ Wenbo Huang ⋅ Haoran Jin ⋅ Lukas Kim ⋅ Ming Liu ⋅ Alireza Rafiei ⋅ Kunyang Sun ⋅ Ting Sun ⋅ Yingze Wang ⋅ Yixin Wang ⋅ Hanwen Xing ⋅ Sihan Xu ⋅ Yuzheng Xu ⋅ Zhongxing Xu ⋅ Zhiling Yan ⋅ Boqin Yuan ⋅ Ruiqi Zhang ⋅ Yifan Zhang ⋅ Zibo Zhao ⋅ Haoyue Bai ⋅ Carlo Bosio ⋅ Joe Cavanagh ⋅ Tianxing Chen ⋅ Yipu Chen ⋅ CHENYU ZHU ⋅ Chen Dai ⋅ Stefano De Castro ⋅ Yunfu Deng ⋅ Kaustubh Dhole ⋅ Jiayuan Ding ⋅ Chenchen Du ⋅ Zhehang Du ⋅ Hao Fan ⋅ Run-Ze Fan ⋅ Hengyu Fu ⋅ Shi Gu ⋅ Yifan Gu ⋅ Baihe Huang ⋅ Baixiang Huang ⋅ Ran Jin ⋅ Xin Lan ⋅ Joseph Lee ⋅ Deren Lei ⋅ Chenyu Li ⋅ Daofeng Li ⋅ Jingyan Li ⋅ Yi Li ⋅ Yuangang Li ⋅ Zhixu Li ⋅ Yinsheng Li ⋅ Wenyu Liang ⋅ Longtai Liao ⋅ Kevin Qinghong Lin ⋅ Zeyi (Andy) Liu ⋅ Jiaming Liu ⋅ Kaiyuan Liu ⋅ Xuan Liu ⋅ Pan Lu ⋅ Yicheng Lyu ⋅ Qiuyang Mang ⋅ Kyle Montgomery ⋅ Ruoxi Ning ⋅ Jorin Overwiening ⋅ Xu Pan ⋅ Core Francisco Park ⋅ Justin Purnomo ⋅ Scott Rankin ⋅ Bixuan Ren ⋅ HaoYang Shang ⋅ Hejia Shen ⋅ JIAWEI SHEN ⋅ Shi Qiu ⋅ Tianneng Shi ⋅ Jonah So ⋅ Vladislav Susoy ⋅ Haocheng Wang ⋅ Jialu Wang ⋅ Wei Wang ⋅ Xinyu J Wang ⋅ Zehao Wang ⋅ Ruoxi Wu ⋅ Dehao Wu ⋅ Fangyu Wu ⋅ Mengyuan Wu ⋅ Yu Wu ⋅ Yuchen Wu ⋅ Yuhao Wu ⋅ Qingpo Wuwu ⋅ Weihang Xiao ⋅ Yongyi Xiong ⋅ Fan Xu ⋅ Ruiling Xu ⋅ Mingxuan Yan ⋅ Benjamin Yang ⋅ Jirong Yang ⋅ Xiaoli Yang ⋅ Yushi Yang ⋅ Haoran Ye ⋅ Xiaohu Yu ⋅ Zhengming Yu ⋅ Chenlong Zhang ⋅ Chi Zhang ⋅ Hanning Zhang ⋅ Hanwen Zhang ⋅ Junge Zhang ⋅ Kunpeng Zhang ⋅ Song Zhang ⋅ Wenjin Zhang ⋅ Wenshuo Zhang ⋅ Qijian Zhao ⋅ Yimin Zhao ⋅ Yuhaohua Zheng ⋅ Liwei Zhou ⋅ Tianyue Zhou ⋅ Sichen Zhu ⋅ Yan Zhu ⋅ Yishu Zhu ⋅ Jerry Zuo ⋅ Chonghao Cai ⋅ Helena Casademunt ⋅ Cheng Cheng ⋅ Nawen Deng ⋅ Rao Fu ⋅ Yifan Han ⋅ He Ren ⋅ Zhenyu He ⋅ Qiao Jin ⋅ Langlang Li ⋅ Yuetai Li ⋅ Lu Lu ⋅ Luqing Zhou ⋅ Subhabrata Mukherjee ⋅ Yunqi Ouyang ⋅ Yin Ren ⋅ Dawei Shi ⋅ Haoran Wu ⋅ Zhiyue Wu ⋅ Hannah Yao ⋅ Zhuoran Yi ⋅ Jenny Yu ⋅ Rhea Zhan ⋅ Hang Zhou ⋅ Shiqiao Zhu ⋅ Junfan Zhu ⋅ Zhen Dong ⋅ Ali Emami ⋅ Yanjun Gao ⋅ Haimin Hu ⋅ Erika J Schneider ⋅ Zhenglu Li ⋅ Jiachen Li ⋅ Lihui Liu ⋅ Murphy Niu ⋅ Yi Shao ⋅ Wenqi Shi ⋅ Lichao Sun ⋅ Jianxin Sun ⋅ Chenguang Wang ⋅ Xuhai &quot;Orson&quot; Xu ⋅ Huaxiu Yao ⋅ Aylin Caliskan ⋅ Jing Huang ⋅ Yang Liu ⋅ Mark W Mueller ⋅ Russell Poldrack ⋅ Sanjiv R Das ⋅ Zhiyong Lu ⋅ Radha Poovendran ⋅ Somayeh Sojoudi ⋅ Costas J Spanos ⋅ Molei Tao ⋅ Mikko Tolonen ⋅ Ting Wang ⋅ Alan Yuille ⋅ Patrick Bryant ⋅ Carl Boettiger ⋅ Arvind Rao ⋅ Tapio Schneider ⋅ Georgios N Yannakakis ⋅ Laure Zanna ⋅ Ida Sim ⋅ Tarek Zohdi ⋅ Jack Gallant ⋅ Teresa Head-Gordon ⋅ Dawn Song
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A1010
Recent AI systems achieve strong results on many benchmarks, yet these gains have not translated into economically meaningful deployment across professional domains. We argue that this gap is largely an evaluation problem: existing benchmarks rarely measure sustained performance on real, economically valuable workflows. We introduce Agents’ Last Exam (ALE), a benchmark for evaluating generalist computer-use AI agents on long-horizon professional tasks with verifiable outcomes. ALE is developed in collaboration with 250+ industry experts and organized around a taxonomy grounded in O*NET / SOC 2018. The benchmark spans 55 subdomains across 13 industry clusters and currently contains 1,490 task instances sourced from authentic professional workflows. Tasks require integrated GUI interaction, shell execution, software operation, and long-horizon planning inside real computing environments. To support scalable evaluation, ALE uses deterministic deliverable-based scoring and structured rubric verification rather than open-ended human judgment whenever possible. Experimental results show that current frontier agents remain far from saturation: across mainstream harness and backbone configurations, the average full-pass rate on the hardest tier is only 2.6%. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for measuring the gap between benchmark success and GDP-relevant impact.
Show more
View full details
Poster

A Process-Level Evaluation of LLM Discovery Agents

Haorui Wang ⋅ Yuanqi Du ⋅ Hao Cheng ⋅ Baolin Peng ⋅ Chao Zhang ⋅ Weizhu Chen ⋅ yelong shen
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A1013
Large language models (LLMs) are increasingly used as \emph{discovery agents} that iteratively edit codes against evaluators on tasks from mathematical optimization, algorithmic problem-solving and scientific discovery. Nevertheless, current evaluation reports only a single final score per (model, task) pair under a fixed agent harness: the surrounding scaffolding that governs feedback, memory, reflection, and revision. This final-score view obscures how agents search, where they fail, and which harness mechanisms are actually useful for different classes of tasks. In this paper, we introduce a four-level ladder of agent harnesses and a process-level evaluation framework that measures behavioral profiles and intermediate milestones, alongside the final performance. Using the evaluation workflow, we run a fully crossed evaluation of six contemporary LLMs on $24$ curated discovery tasks across four harness configurations, producing $1{,}728$ trajectories. Our analysis reveals two notable findings. First, harness effectiveness is highly problem-dependent. Second, models can exhibit distinct search behaviors even when their final scores are similar. We release all the trajectories, behavioral profiles, capability rubrics, and harness implementations as reusable evaluation resources for the community.
Show more
View full details
Poster

JobBench: Aligning Agent Work With Human Will

Yuetai Li ⋅ Yichen Feng ⋅ Zhangchen Xu ⋅ Zixian Ma ⋅ Kaiyuan Zheng ⋅ Fengqing Jiang ⋅ Xinghua Sun ⋅ Rulin Shao ⋅ Zichen Chen ⋅ Yue Huang ⋅ Xinyang Han ⋅ Brian Lee ⋅ Kayla Xu ⋅ Shenglai Zeng ⋅ Hang Hua ⋅ Xiangliang Zhang ⋅ Basel Alomair ⋅ Ranjay Krishna ⋅ Luke Zettlemoyer ⋅ Pang Wei Koh ⋅ Bhaskar Ramasubramanian ⋅ Luyao Niu ⋅ Xiang Yue ⋅ Radha Poovendran
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A801
Current benchmarks for occupational AI agents are scoped primarily by economic values, telling a replacement story. We introduce JobBench, which evaluates AI agents on the workflows that experts identify as high-priority for delegation, empowering humans based on their needs instead of replacing them with GDP value. JobBench covers 130 agentic tasks across 35 occupations. Each task is packaged as a workspace of heterogeneous reference files, requiring the agent to reason through the cluttered information streams of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task. We evaluate 36 models; the strongest, Claude Opus 4.7 under Claude Code, reaches only 45.9 \%. We hope JobBench shifts the community's target labour-market effect from replacement to enhancement: building agents that do what humans actually want delegated, not only what is most economically valuable.
Show more
View full details
Poster

ContractBench: Can LLM Agents Preserve Observation Contracts?

Jicheng Wang ⋅ Yifeng He ⋅ Zili Wang ⋅ Hanwen Xing ⋅ Arkaprava De ⋅ Hao Chen
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A1008
Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation contracts: artifacts whose later use is constrained by the external system that produced them. We show that observation contract compliance (preserving the temporal validity and byte-level integrity) is an emergent, regression-prone capability: it is neither guaranteed by general tool-use ability nor consistently improved by larger or newer models. To measure this, we introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two orthogonal failure modes no existing benchmark evaluates: validity failures (using an artifact after expiry) and integrity failures (corrupting an artifact's bytes through the observation-to-action pipeline). Our evaluation is deterministic and programmatic, with a virtual clock controls time and SHA-256 hashes verify byte integrity. We assign the outcomes failure labels drawn from real-world API specifications. We evaluate 38 models and report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%, revealing that current frontier models still fail to comply with observation contracts; (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B: what emerges across the cliff is mid-trajectory restraint, not tool-call competence; (iii) non-monotonic scaling across GPT-5 family: agentic post-training can erode compliance through sycophancy-driven regression; (iv) our failure taxonomy works as an actionable in-context reward signal, yielding +7.1 pp on 42 paired GPT-5.1 failures.
Show more
View full details
Poster

MOSAIC-CONUS: A Multimodal, Multi-Temporally Paired dataset for Earth Sciences

Abhishek Potnis ⋅ Youssef Hussein ⋅ Waqwoya Abebe ⋅ JangHyeon Lee ⋅ Debvrat Varshney ⋅ Jacob Arndt ⋅ Philipe Dias ⋅ Aristeidis Tsaris ⋅ Dan Lu ⋅ Dalton Lunga
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A119
Earth embeddings—vector representations of geographic locations indexed in space and time—are emerging as a unifying interface for geospatial AI. However, their quality depends not only on model design, but on how multimodal Earth observation (EO) data are spatially indexed, temporally aligned, and cross-modally associated during pretraining. We introduce MOSAIC-CONUS ($\textbf{M}$ultimodal $\textbf{O}$bservations with $\textbf{S}$patially $\textbf{A}$ligned Imagery, Urban Points of Interest, $\textbf{I}$n-Situ Measurements and Text $\textbf{C}$aptions), a large-scale EO dataset over the contiguous United States, organized around 250{,}000 stratified point indices that serve as stable spatial keys across seven modalities: active radar, passive optical imagery, lidar-derived elevation, land cover, functional context, hydrometeorological measurements, and textual summaries. Unlike Existing EO datasets, MOSAIC-CONUS introduces four contributions not jointly addressed in prior work: $\textbf{1.}$an open-source, large-scale multimodal EO corpus structured around point-indexed data designed to support Earth embedding learning; $\textbf{2.}$explicit radar-optical pairing tables spanning twelve temporal alignment regimes, formalizing cross-sensor alignment as a controllable variable for analyzing how temporal mismatch across modalities influences learned embeddings quality; $\textbf{3.}$a benchmark suite spanning cross-modal retrieval, annual nightlights regression, and basin-held-out streamflow prediction, positioning MOSAIC-CONUS as a benchmark-ready resource for multimodal AI systems; and $\textbf{4.}$a language-based embedding layer through co-registered textual summaries, enabling Earth embeddings to function as a queryable interface for agentic AI systems. The dataset and pairing protocols are publicly released.
Show more
View full details
Poster

PortPy: A Benchmark for AI and Optimization in Cancer Radiotherapy Planning

Gourav Jhanwar ⋅ Mojtaba Tefagh ⋅ Qijie Huang ⋅ Alireza Golkarieh ⋅ Linda Hong ⋅ Hai Pham ⋅ Vicki T Taasti ⋅ Seppo Tuomaala ⋅ Anthony Magliari ⋅ Michael Folkerts ⋅ Ying Zhou ⋅ Jie Yang ⋅ Saad Nadeem ⋅ Masoud Zarepisheh
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A728
External beam radiotherapy is a cornerstone of cancer treatment, used in more than half of all cancer patients. It delivers high-energy radiation beams to destroy tumors while minimizing radiation dose to surrounding healthy tissues. Treatment planning requires optimizing machine parameters, including the shapes and intensities of radiation beams, based on each patient’s unique anatomy. This gives rise to large-scale, multi-criteria, and often combinatorial optimization problems that must be solved separately for each patient under strict clinical time constraints. Although recent advances in AI/ML show promise for improving many planning and decision-making tasks, progress in this area has been limited by the lack of standardized, publicly available benchmark datasets and baseline algorithms. To address this gap, we introduce PortPy (Planning and Optimization for Radiation Therapy in Python), an open-source benchmark dataset derived from real patient data, together with baseline algorithms and a Python-based framework for data processing, simulation, and standardized evaluation. Several of the provided baseline methods have been clinically validated and deployed at our institution to support automated radiotherapy planning for cancer patients. By providing a common benchmark and open research platform, PortPy aims to accelerate progress in radiotherapy treatment planning and promote collaboration across the AI/ML, mathematical optimization, medical physics, and radiation oncology communities. Toolkit: https://github.com/PortPy-Project/PortPy Dataset: https://huggingface.co/datasets/PortPy-Project/PortPy_Dataset
Show more
View full details
Poster

Benchmarking Fine-Grained Spatio-Temporal Awareness in Embodied Brain Models

Minghao Zhu ⋅ Zhikai Wang ⋅ Ronghao Dang ⋅ Bohan Hou ⋅ Jiangpin Liu ⋅ Kehan Li ⋅ Jiayan Guo ⋅ Sicong Leng ⋅ Yunxuan Mao ⋅ Yuqian Yuan ⋅ Xiao Lin ⋅ Xin Li ⋅ Deli Zhao
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A1011
Large Vision-Language Models (LVLMs) have shown remarkable potential for embodied AI, yet a critical bottleneck persists: the lack of fine-grained spatio-temporal alignment between high-level textual reasoning and low-level visual grounding. Existing benchmarks fail to recognize the critical importance of such fine-grained alignment for embodied tasks. To address this gap, we introduce ECLBench, a comprehensive benchmark designed to systematically assess spatio-temporal embodied understanding across two primary capabilities: Embodied Cognition (Object Cognition and Spatial Cognition) and Embodied Localization (Grounding and Pointing). Together, these four pillars encompass 21 specialized sub-capabilities, supported by 3,616 egocentric video clips (577,998 frames) and 12,000 meticulously curated open-ended questions. Extensive evaluations on ECLBench reveal that general LVLMs excel in semantic understanding but struggle with precise spatio-temporal grounding, while existing embodied agents often sacrifice generalization for specialization. Guided by the benchmark's findings, we develop ECLBrain, a baseline model that discretizes continuous spatial coordinates into integer tokens. This simple yet effective design bridges textual semantics and spatio-temporal vision, demonstrating a promising path toward unified embodied reasoning. ECLBench and ECLBrain together establish a rigorous, multi-dimensional standard for diagnosing current limitations and guiding future research toward spatially-aware and temporally-consistent embodied intelligence.
Show more
View full details
Poster

ReaLM: A Unified Red-Teaming Benchmark for Physical-World VLMs

Yifei Zhao ⋅ Qian Lou ⋅ Mengxin Zheng
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A928
Vision-language models (VLMs) are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems, where perception or reasoning errors can lead to unsafe decisions or actions. Although many red-teaming methods have been developed to probe VLM vulnerabilities, their evaluations remain fragmented across datasets, metrics, and threat models, making direct comparison difficult and obscuring whether observed differences arise from stronger attacks, more vulnerable models, or incompatible evaluation settings. Existing chatbot-centric red-teaming benchmarks mainly standardize jailbreak and content-safety evaluation, but they do not systematically capture physically grounded functional failures or cover red-teaming methods that target physical-world VLMs. This raises the key challenge of comparing diverse attack methods under a unified protocol while targeting the same scenario-specific failures. We introduce REALM, to our knowledge the first unified red-teaming benchmark for physical-world VLMs. REALM integrates 12 red-teaming methods, 3 model-agnostic defenses, and 13 VLMs under a practical black-box threat model with shared datasets and metrics. To align adversarial objectives across attack families, REALM introduces an agentic target-generation pipeline that constructs shared, scenario-specific, and physically grounded attack objectives for each scene, enabling fair comparison of diverse red-teaming methods under aligned adversarial goals. Our evaluation shows that text and typographic injection attacks induce the most failures, multimodal co-optimization yields the strongest visual-perturbation transfer, single-pass attacks approach iterative methods at much lower cost, and model scale alone does not confer adversarial robustness. Code is available at https://anonymous.4open.science/r/REALM-762F.
Show more
View full details
Poster

MLLMs Fail to Refuse when Using Tools Agentically

Rikiya Takehi ⋅ Ryo Hachiuma ⋅ Shaona Ghosh ⋅ Dan Zhao ⋅ Frank Wang ⋅ Yusuke Hirota
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A1012
Agentic multimodal large language models (MLLMs) have recently pushed the frontier of visual reasoning by calling tools such as zooming and tagging. Despite the recent strong success of agentic MLLMs, this work uncovers a critical safety failure in the tool-use paradigm: agentic tool-using MLLMs become less capable of refusing harmful requests. Our experiments confirm that, across three popular safety benchmarks, all the top open- and closed-weight MLLMs we test exhibit significantly lower safety in tool-using settings than in non-tool settings, with a relative refusal failure rate increase of up to 68.7%. Based on analysis of 100,000+ responses, including extended experiments, we also propose two possible reasons for this safety degradation.
Show more
View full details
Poster

Auditing Cross-Lingual Fairness in Language Model Watermarking

Alexander Nemecek ⋅ Osama Zafar ⋅ Debargha Ganguly ⋅ Vikash Singh ⋅ Vipin Chaudhary ⋅ Erman Ayday
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A815
Watermarking schemes for large language model output are evaluated almost exclusively on English text using each scheme's detection threshold and a narrow set of quality measurements. Multilingual deployment exposes evaluation-design choices that are inconsequential on English but determine conclusions cross-lingually. We propose an evaluation framework with four components: detection thresholds calibrated empirically per deployment context, a threshold-independent companion measurement that distinguishes calibration failures from detection failures, three disjoint quality measurement paradigms (distributional, paired-semantic, and reference-perplexity), and a generalized-entropy decomposition of cross-language disparity over a typological family partition. Applied to six watermarking schemes, three open-weight generators, eleven languages spanning four scripts and eight typological families, and both base and instruction-tuned regimes, the framework reveals failure modes that single-language single-paradigm evaluation cannot surface. Across detection and quality, observed disparity is predominantly between-family on the typological partition, indicating that cross-lingual fairness gaps in watermarking are structural to language properties rather than idiosyncratic to particular languages.
Show more
View full details
Poster

FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis

Soham Dan
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A1009
We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, the benchmark contains 2,329 tasks across 24 collections and 5.52M hidden instances in two executable families: statistic synthesis (object → integer) and map synthesis (object → object). Each task provides a mathematical description and at most five public input–output examples; a model must emit a single Python solve function, with no retrieval, tool use, execution feedback, voting, or reranking. Submissions are scored by exact sandboxed execution on held-out combinatorial objects. We evaluate eleven systems under this protocol: four closed-source production models and seven open-weight models served through a common inference provider. Three findings motivate FindStatBench as a stress test for symbolic program induction rather than general software engineering. First, the strongest open- and closed-source systems converge within ~1 pp instance accuracy, while an oracle over all eleven systems improves task accuracy by only ~10 pp over the best single model; five-way sampling from one mid-tier model reaches the same ceiling. Second, examples are not uniformly helpful: on several classical named bijections, zero-example prompts produce perfect implementations while five-example prompts collapse to near-zero hidden accuracy and fail even their public examples, suggesting prompt-induced regression away from canonical algorithms. Third, apparent coverage gaps can reflect output-budget mechanics rather than inability: hidden reasoning can consume the visible response budget before code is emitted, a failure mode observed in both closed-source and open-weight reasoning models. Across all evaluated systems, statistic synthesis remains substantially easier than map synthesis, set partitions and binary trees remain near-zero-accuracy collections, and long prompts induce a sharp accuracy cliff. Cost-normalised, the strongest open-weight systems dominate the closed-source production models on this benchmark. FindStatBench exposes a distinctive capability gap: current models can often write plausible mathematical code, but exact symbolic rule induction over structured objects remains brittle across scale, specialization, and weight provenance.
Show more
View full details
Poster

Physical AI Smart Spaces: A Large-Scale Benchmark for Multi-Camera 3D Perception in Smart Spaces

Yuxing Wang ⋅ Yizhou Wang ⋅ Anqi Li ⋅ Shuo Wang ⋅ Sameer Satish Pusegaonkar ⋅ Haoquan Liang ⋅ Jiajun Li ⋅ Shenxin Jiang ⋅ Jianhe Yuan ⋅ Shangru Li ⋅ Tongwei Dai ⋅ Zihao Chen ⋅ David Anastasiu ⋅ Zheng (Thomas) Tang ⋅ Sujit Biswas ⋅ Xunlei Wu
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A623
Physical AI Smart Spaces is, to the best of our knowledge, the first benchmark to simultaneously provide large-scale, multi-class, and multi-camera 3D perception data for indoor smart spaces. It contains over 280 hours of synchronized 1080p footage captured by nearly 1,800 cameras in warehouses, hospitals, retail venues, and similar settings, together with automatic annotations for multi-camera identities, 2D bounding boxes, 3D bounding boxes, camera calibration, and depth where available. The benchmark spans Isaac Sim synthetic generation, Cosmos Transfer appearance augmentation, and real-world Sim2Real evaluation. For the real-world target, we include two warehouse deployments with time-synchronized streams, automatic VGGT-based calibration, and a 3D labeling interface that projects world-frame 3D boxes into each view for cross-camera verification. We describe the dataset scope, annotation and calibration schema, generation workflow, benchmark protocols, and official evaluation system, which standardizes submission format, and leaderboard reporting. A central contribution is a 3D instantiation of Higher Order Tracking Accuracy (HOTA), extending the usual 2D box-based tracking evaluation to 3D locations and 3D boxes. We further report empirical baselines from the AI City Challenge leaderboards, showing how methods evolve from person-only 3D location tracking to multi-class 3D box tracking under realistic smart-space constraints. The release is available at https://huggingface.co/datasets/nvidia/PhysicalAI-SmartSpaces.
Show more
View full details
Poster

GenScale: A Benchmark for Relative Object Scale in Image Generation and Editing

Lingxiao Li ⋅ Max Whitton ⋅ Ledell Wu ⋅ Boqing Gong
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A505
Modern image generation and editing systems can produce photorealistic, prompt-aligned images, but still often render familiar objects at implausible relative sizes. To measure this failure mode, we introduce GenScale, a benchmark and evaluation protocol for real-world relative object scale in image generation and editing. GenScale contains 900 image-level entries and 1,643 pairwise anchor-target scale relations across common-object generation, human-product generation with metric dimensions, and scale correction from failed generations. We further design a human-calibrated ordinal judge for scalable pairwise scale evaluation. Last but not the least, we introduce ReScale, a model-agnostic post-processing agent for localized scale correction without modifying the source generator. Experiments reveal that state-of-the-art image generators and editors cannot reliably observe relative scale yet, while ReScale consistently improves scale plausibility across generated and edited images. Together, GenScale establishes relative object scale as a distinct, measurable, and actionable capability for image generation systems.
Show more
View full details
Poster

MathlibPR: Pull Request Merge-Outcome Benchmark for Formal Mathematical Libraries

Zixuan Xie ⋅ Xinyu Liu ⋅ Shangtong Zhang
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A800
The ecosystem of Lean and Mathlib has become the de facto standard for large language model (LLM) assisted formal reasoning with remarkable successes in recent years. Those successes, however, only consume Mathlib as an essential dependency but do not directly contribute to it. In the meantime, the growth of Mathlib has recently been bottlenecked by the review process, which requires human reviewers to judge whether proposed pull requests (PRs) follow the Mathlib's conventions and are worth integrating as part of a shared mathematical infrastructure. This leads to our central question: can LLMs help review Mathlib PRs? To this end, we introduce MathlibPR, a benchmark built from real Mathlib4 PR histories. We further propose a staged evaluation protocol and use it to evaluate both LLM models (e.g., DeepSeek, Qwen, Goedel, and Kimina) and LLM agents (e.g., Codex and Claude Code). Surprisingly, both LLM models and LLM agents struggle to distinguish merge-ready PRs from build-passing PRs that were revised or never merged. By turning Mathlib PR histories into a supervised signal, MathlibPR provides a step toward reviewer assistants and reward models that could help evaluate PRs and steer LLMs toward producing merge-ready Mathlib contributions.
Show more
View full details
Poster

Benchmarking Risk Attitudes of LLMs

Bowen Sun ⋅ Rui Min ⋅ Xianyao Li ⋅ Yuxi Wang ⋅ Yang Ye ⋅ Ryan Qi Wang ⋅ Eric Du
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A927
As AI systems are deployed in open-ended, high-stakes domains, a critical behavioral dimension remains unmeasured: whether models differ systematically in how they translate perceived risk into action. We introduce a cross-domain evaluation framework that isolates and quantifies \emph{risk attitude} in large language models (LLMs) by decoupling contextual belief ($B_C$, a model's perceived danger level given situational context) from categorical risk decision ($R_D$, the action chosen in response to that belief). Unlike existing capability benchmarks, which assess factual accuracy but do not probe the belief-to-decision mapping, this framework directly targets the disposition governing whether an agent acts cautiously or aggressively under identical levels of perceived danger. We applied the framework to six frontier LLMs and 100 human participants across spatial navigation, clinical triage, and financial allocation tasks, and identified two qualitatively distinct behavioral profiles. Five models satisfied both reliability criteria (Contextual Belief Consistency and Risk Decision Consistency) across all conditions and tasks. These models showed strong measurement reliability with stable belief formation and belief-to-decision mappings, and exhibited stable, trait-level cross-domain risk attitudes that were preserved across all three tasks (Kendall's $W=1.00$, $p=0.017$). One model, Grok~4, failed both criteria: its belief formation was unstable in Drone Navigation Control task and its decision mapping was inconsistent in Clinical Triage Decision task, yielding task-specific rather than trait-level risk behavior. Across all six LLMs, risk profiles converged toward a restricted region of the broader human risk-attitude distribution, suggesting that alignment training constrains models toward a narrow behavioral consensus. These results demonstrate that the framework reliably measures risk attitude as a stable, model-specific behavioral dimension and can serve as a diagnostic tool for pre-deployment assessment by distinguishing models with coherent risk personalities from those with domain-contingent risk postures.
Show more
View full details
Poster

GPT-Image-Edit-1M: An Auditable Million-Scale Dataset for Instruction-Guided Image Editing

Yuhan Wang ⋅ Siwei Yang ⋅ Bingchen Zhao ⋅ Letian Zhang ⋅ Jiawei Mao ⋅ Juncheng Wu ⋅ Zeyu Zheng ⋅ Huaxiu Yao ⋅ Qing Liu ⋅ Cihang Xie ⋅ Yuyin Zhou
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A506
We present GPT-IMAGE-EDIT-1M, a million-scale instruction-guided image editing dataset in which every sample includes an auditable curation decision and a traceable instruction trail. Open training data for image editing often suffers from instruction-image mismatch, weak edits, and opaque filtering; simply scaling raw data does not address these issues. To improve data quality, we regenerate edited outputs using GPT-Image-1, augment samples with complex multi-step edit instructions, and apply a policy-based quality control pipeline that assigns KEEP/RELABEL/DROP decisions with reason tags. RELABEL samples receive aligned instructions that better match the observed edit, and a second judge independently re-runs the same protocol to measure decision stability. After quality control, average quality improves from 6.90 to 7.67 under a shared rubric. Across two frozen encoder configurations, T5 and QwenVL+T5, models fine-tuned on GPT-IMAGE-EDIT-1M outperform the public FLUX.1-Kontext-dev baseline on key benchmarks, including GEdit-EN (7.22 vs. 6.26), ImgEdit (3.93 vs. 3.52), and KRISBench (66.23 vs. 49.54), while remaining competitive on Complex-Edit and OmniContext SINGLE. Controlled data-swap experiments show that GPT-Image-1 regeneration provides the dominant downstream gain, while quality control yields a smaller but consistent improvement over a size-matched random regenerated subset. Concatenating frozen QwenVL embeddings with T5 substantially improves inferential-complex benchmarks such as RISEBench and KRISBench without degrading general-editing performance. Finally, as an auxiliary diagnostic rather than a primary ranking protocol, inference-time instruction rewriting raises GEdit-EN to 7.75 and KRISBench to 78.55, suggesting that prompt specificity exposes additional instruction-following capacity in the fine-tuned editors.
Show more
View full details
Poster

Automated Hypothesis Discovery for Characterizing Annotation Disagreement

Ankita Gupta ⋅ Alexandra Chouldechova ⋅ Alex Dow ⋅ Miro Dudik ⋅ Jean Garcia-Gathright ⋅ Hanna Wallach ⋅ Nicholas Pangakis
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A122
Annotation disagreement is common in multi-annotated datasets used throughout the AI/ML lifecycle, from model training to evaluation. However, existing training and evaluation practices often collapse annotations into ``ground truth'' labels that treat disagreement as noise. In socially consequential contexts, annotation disagreement can reflect multiple valid perspectives, so it is especially important to understand when and why annotators---be they human or automated---disagree. Existing disagreement analysis methods rely on manually created taxonomies of disagreement sources that are costly to produce and difficult to scale. We introduce CHARDIS, a two-stage automated disagreement analysis method that (1) generates candidate disagreement hypotheses—natural language statements describing textual features that predict disagreement—from a multi-annotated dataset and (2) validates these candidates on a held-out dataset, retaining only statistically significant predictors of disagreement and ranking them by predictive performance. CHARDIS supports scalable, interpretable disagreement analysis across diverse annotation tasks. On semi-synthetic datasets, CHARDIS recovers known disagreement sources. On four real-world datasets, CHARDIS reproduces existing manually created taxonomies of disagreement sources and discovers new disagreement hypotheses. In both settings, CHARDIS achieves better predictive performance than baseline methods. Code to reproduce our experiments is available at: https://anonymous.4open.science/r/disagreement-analysis-pipeline-6319
Show more
View full details
Poster

Beyond Risky Activities: Bridging the Supervision Gap for Situational Risk Reasoning

Zelin Li ⋅ Yifan Liu ⋅ Ruichen Yao ⋅ Yaokun Liu ⋅ Rahul Gupta ⋅ Dong Wang
Dec 12, 8:30 AM - 11:30 AM Hall C1 #A814
Situational risks refer to safety risks that arise from the interaction between a user activity and its surrounding visual scenario, where the same activity may be safe in one situation but unsafe in another. Identifying such risks is particularly important for deploying VLMs as real-time assistants, where models are expected to assist users based on a shared visual context. However, existing safety alignment data is dominated by examples of intrinsically harmful activities, leaving activity--scenario risks underrepresented. We refer to this missing activity--scenario supervision as a **supervision gap** in situational risk reasoning for VLMs. To address this gap, we introduce **Situational Risk Reasoning (SRR)**, the first post-training dataset for improving VLMs' situational risk awareness. SRR is constructed through a hybrid pipeline that combines vision-language reasoning, language-based generation, and text-to-image synthesis. Each example includes a target response in a safety chain-of-thought format, guiding models to reason about activity--scenario interactions rather than simply learning refusal patterns. Experiments show that supervised fine-tuning on SRR substantially improves VLMs' ability to reason about situational risks, while maintaining strong performance on conventional safety benchmarks and general VQA tasks. We also demonstrate the effectiveness of SRR across different VLM backbones and validate the importance of both the synthesized SRR data and the safety chain-of-thought format through ablations. The SRR dataset is available at https://huggingface.co/datasets/Anonymous-SRR/SRR.
Show more
View full details
Poster

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

Kuan Wang ⋅ Chao Zhang
When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success. We find that the fact can remain recoverable from retained user images, including images tagged to entirely different facts, because VLMs use implicit visual cues at inference time. We introduce the Information Provenance Graph (IPG), a taxonomy that classifies memory representations by deletion affordance. The IPG reveals that deletion fails through multiple channels. Our benchmark, MemLeak, measures this across a deletion cascade: direct probing of deletion-capable systems yields <1%, but retained correlated text enables 18.3% recovery, and retained images enable 12.0% recovery (0.0% blind baseline, 0.3% FPR)---with 47% of image leaks not text-recoverable. Content-aware semantic deletion reduces the image residual to 2.0%. The residual appears across multiple VLMs, a production memory system, and real Unsplash-licensed photographs. Dual-annotator human validation ($\kappa$ = 0.88) confirms judge reliability.
Show more
View full details
Oral

AgentAbstain: Do LLM Agents Know When Not to Act?

Xun Liu ⋅ Yi Evie Zhang ⋅ Vira Kasprova ⋅ Parisa Rabbani ⋅ Pardissadat Zahraei ⋅ Tianyu Zhang ⋅ Ali Ebrahimpour-Boroojeny ⋅ Varun Chandrasekaran
Dec 10, 7:30 AM - 7:45 AM Hall C3
Agent systems based on Large Language Models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain. This gap poses real risks: under ambiguity, conflicting constraints, or tool failures, agents may execute unintended and irreversible actions. To close this gap, we present the first systematic evaluation framework for agentic abstention: the ability to recognize when not to act. At its core, AgentAbstain is a paired-task benchmark that defines 8 abstention categories spanning pre-execution reasoning and runtime discovery, with 263 paired tasks across 42 executable sandbox environments, each pairing a should-act task with a should-abstain variant produced by a controlled perturbation to the instruction, tool, or environment state. Scaling such paired evaluations poses two practical challenges: manually authoring diverse tasks is expensive, and static benchmarks risk data contamination as models evolve. To address both, we propose AbstainGen, a fully automated pipeline that synthesizes sandbox environments and generates paired tasks end-to-end, validated by deterministic replay and semantic LLM judges. Its scalable design enables on-demand regeneration of fresh task instances, and human validation rates 96% of generated tasks as well-designed. Evaluating 17 frontier LLMs across 4 agent harnesses, the best model (Gemini 3.1 Pro) achieves only 59.5% paired accuracy (correct on both the act and abstain sides of each paired task). More importantly, abstention capability is largely independent of general task-solving capability, indicating that scaling task-solving alone will not close this gap. We identify failure modes such as Post-hoc Abstention, in which agents execute irreversible actions before recognizing abstention triggers. For instance, an agent may cancel a flight reservation before noticing contradictory rebooking instructions, leaving the user stranded. These findings underscore the need for rigorous abstention evaluation to develop more trustworthy LLM agents.
Show more
View full details
Oral

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

Bohan Lyu ⋅ Yucheng Yang ⋅ Siqiao Huang ⋅ Jiaru Zhang ⋅ Qixin Xu ⋅ Xinghan Li ⋅ Xinyang Han ⋅ Yicheng Zhang ⋅ Huaqing Zhang ⋅ Runhan Huang ⋅ Kaicheng Yang ⋅ Zitao Chen ⋅ Wentao Guo ⋅ Junlin Yang ⋅ Xinyue Ai ⋅ Wenhao Chai ⋅ Yadi Cao ⋅ Ziran Yang ⋅ Kun Wang ⋅ Dapeng Jiang ⋅ Huan-ang Gao ⋅ Shange Tang ⋅ Chengshuai Shi ⋅ Simon Du ⋅ Max Simchowitz ⋅ Jiantao Jiao ⋅ Dawn Song ⋅ Chi Jin
Dec 12, 7:30 AM - 7:45 AM C111-112
Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. As large language models demonstrate advanced capabilities in reasoning, coding, and engineering tasks, it is increasingly important to understand whether they can discover such methods rather than only apply existing ones. We introduce MLS-Bench, a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods. MLS-Bench contains 140 tasks across 12 domains, each requiring an agent to improve one targeted component of an ML system or algorithm and demonstrate that the improvement generalizes across controlled settings and scales. We find that current agents remain far from reliably surpassing human-designed methods, and that engineering-style tuning is easier for them than genuine method invention. We further study the effects of test-time scaling, adaptive compute allocation, and context provision on agents' discovery performance, together with case studies of their behavior. Our analyses suggest that the bottleneck is not only in proposing new methods, but also in the scientific insight needed to plan, validate, and scale claims about them. More search, compute, or context alone does not remove this bottleneck. We build and maintain a community platform for cumulative and comparable iteration, and release the data and code at https://mls-bench.com.
Show more
View full details
Oral

SALART-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images

Xiaoxiao Sun ⋅ Ruotian Zhang ⋅ Junzhe Huang ⋅ James Burgess ⋅ Serena Yeung-Levy
Dec 11, 7:45 AM - 8:00 AM C111-112
Vision-language models (VLMs) are increasingly used to detect whether AI-generated images contain visible artifacts, yet their ability to analyze such artifacts remains poorly understood. A correct image-level decision can still hide important failures: a model may correctly flag an artifact while relying on the wrong visual cue, selecting the wrong region, or describing a defect that the image does not support. To evaluate these behaviors directly, we introduce SalArt-VQA, a diagnostic benchmark for fine-grained salient artifact understanding in AI-generated images. SalArt-VQA contains 950 images and 3,681 human-authored multiple-choice questions spanning artifact images, matched real reference images, and paired generated reference images. Four aligned question types evaluate presence detection, semantic localization, spatial grounding, and evidence-grounded defect identification, while the reference splits test calibration and abstention when the annotated defect is absent. Across 20 VLMs, SalArt-VQA reveals failures that image-level detection accuracy hides: the strongest model reaches 99.37% detection recall on artifact images but answers all four artifact-side questions correctly on only 53.26% of images. Comparing artifact images with defect-absent references reveals a sensitivity–calibration tradeoff: sensitive models often make unsupported artifact claims, while conservative models avoid false alarms largely by missing real artifacts. These results show that high artifact detection accuracy alone does not imply grounded artifact understanding. SalArt-VQA exposes these hidden failure modes and provides a fine-grained evaluation of whether VLM artifact claims are supported by local visual evidence.
Show more
View full details
Oral

HearSayBench: Can LLMs Navigate from Abstract Human Rights to Lived Lives?

Sobhan Lotfi ⋅ Ava Iranmanesh ⋅ Ali Iranmanesh ⋅ Liwei Jiang
Dec 12, 7:45 AM - 8:00 AM C111-112
"Hearing is never like being.'' —Persian Proverb Large language models have become the default advisors for life-critical human problems. While they democratize access to personal counseling, they suffer from a silent foundation bias: the internet is a record of people with the freedom to act. Current benchmarks assume users have this same agency, ignoring the reality of those in war zones or navigating statelessness. These long-tail experiences are not only missing from training data, but authentic evaluation data to measure them is equally scarce. We introduce HearSayBench, a human-verified dataset of 400 scenarios from respected archives like the United Nations, covering 80 regions across three specific barriers: social, personal, and environmental. Our work uses Capabilities Approach to test if a model can distinguish between what a person is legally promised and what they are actually free to do in their specific environment. While these models may have "heard'' about global inequality during training, we find that this knowledge is merely hearsay. Across 11 frontier and open-weight models, we identify a systemic %37 performance drop between situational comprehension and structural reasoning. When faced with the most vulnerable users, models consistently offer a "Checklist of Impossible Things'': polite, fluent advice that is physically impossible or legally suicidal to follow. Ultimately, we show that the true digital divide is no longer about access to technology, but about whether an AI can recognize the reality of your life.
Show more
View full details
Oral

MOVEBENCH: A Benchmark for Global-Scale Wildlife Movement Forecasting

Justin Kay ⋅ Shir Bar ⋅ Ellen O Aikens ⋅ Martin Becker ⋅ Francesca Cagnacci ⋅ Juliet Cohen ⋅ Scott W Forrest ⋅ Jessica Kendall-Bar ⋅ Madeleine Lucas ⋅ Macon Overcast ⋅ Meredith S Palmer ⋅ Will Rogers ⋅ Nicholas J Russo ⋅ Christian Rutz ⋅ Larissa T Beumer ⋅ Michael B Brown ⋅ Ying-Chi Chan ⋅ Sarah C Davidson ⋅ Diego E Soto ⋅ Anne G Hertel ⋅ Roland Kays ⋅ Benjamin Koger ⋅ Guram Mikaberidze ⋅ Thomas Mueller ⋅ Ruth Oliver ⋅ Thorsten Papenbrock ⋅ Robert Patchett ⋅ Jared A Stabach ⋅ Dane Taylor ⋅ Scott W Yanco ⋅ Sara Beery
Dec 12, 8:00 AM - 8:15 AM C111-112
Understanding and predicting wildlife movement is critical for ecology and conservation. While trajectory forecasting has advanced for human and vehicle movement, wildlife trajectories present distinct challenges: they are unconstrained in space, highly stochastic, and influenced by environmental conditions. We introduce MOVEBENCH, the first large-scale benchmark for probabilistic wildlife movement forecasting, containing 2.6M GPS locations from 800+ individuals across 110 species in 127 countries, paired with 1.6B environmental raster tiles capturing 160 covariates known or hypothesized to influence movement. We propose a probabilistic evaluation protocol for movement trajectory forecasts, addressing limitations of point-prediction metrics for inherently stochastic phenomena. Through comprehensive empirical evaluation of four method families across multiple temporal and spatial scales, we reveal that: (1) existing predictive methods generalize better to future timepoints than to unseen individuals, (2) deep learning approaches do not consistently outperform simpler baselines, and (3) environmental covariate selection significantly impacts performance. MOVEBENCH enables standardized evaluation of movement forecasting methods and provides a foundation for methodological advances on this ecologically important task.
Show more
View full details
Oral

DD-Ranking: Rethinking the Evaluation of Dataset Distillation

Zekai Li ⋅ Xinhao Zhong ⋅ Samir Khaki ⋅ Zhiyuan Liang ⋅ Yuhao Zhou ⋅ Mingjia Shi ⋅ Ziqiao Wang ⋅ Xuanlei Zhao ⋅ Wangbo Zhao ⋅ Ziheng Qin ⋅ Mengxuan Wu ⋅ Pengfei Zhou ⋅ Haonan Wang ⋅ David Junhao Zhang ⋅ Jia-Wei Liu ⋅ Shaobo Wang ⋅ Dai Liu ⋅ Linfeng Zhang ⋅ Guang Li ⋅ Kun Wang ⋅ Zheng Zhu ⋅ Zhiheng Ma ⋅ Joey Tianyi Zhou ⋅ Jiancheng Lv ⋅ Yaochu Jin ⋅ Peihao Wang ⋅ Kaipeng Zhang ⋅ Yiran Huang ⋅ Zhiwei Deng ⋅ Xindi Wu ⋅ George Cazenavette ⋅ Yuzhang Shang ⋅ Justin CUI ⋅ Jindong Gu ⋅ Qian Zheng ⋅ Hao Ye ⋅ Shuo Wang ⋅ Xiaobo Wang ⋅ Yan Yan ⋅ Angela Yao ⋅ Mike Zheng Shou ⋅ Tianlong Chen ⋅ Hakan Bilen ⋅ Baharan Mirzasoleiman ⋅ Manolis Kellis ⋅ Konstantinos N Plataniotis ⋅ Zhangyang &quot;Atlas&quot; Wang ⋅ Bo Zhao ⋅ Yang You ⋅ Kai Wang
Dec 12, 8:15 AM - 8:30 AM C111-112
Dataset distillation has provided a reliable solution for data compression, where models trained on the resulting smaller synthetic datasets achieve performance comparable to those trained on the original datasets. To further improve the performance of synthetic datasets, various training pipelines and optimization objectives have been proposed, greatly advancing the field of dataset distillation. Recent decoupled dataset distillation methods introduce soft labels and stronger data augmentation during the post-evaluation phase and scale dataset distillation up to larger datasets. However, this raises a question: Is accuracy still a reliable metric to fairly evaluate dataset distillation methods? Our empirical findings suggest that the performance improvements of these methods often stem from additional techniques rather than the inherent quality of the images themselves, with even randomly sampled images achieving superior results. Such misaligned evaluation settings severely hinder the development of DD. Therefore, we propose DD-Ranking, a unified evaluation framework, along with new general evaluation metrics to uncover the true performance improvements achieved by different methods. By refocusing on the actual information enhancement of distilled dataset, DD-Ranking provides a more comprehensive and fair evaluation standard for future research advancements.
Show more
View full details