Sydney Evaluations & Datasets
EchoXFlow: A Beamspace Echocardiography Dataset for Cardiac Motion, Flow, and Function
Elias Stenhede ⋅ Joanna Sulkowska ⋅ Eivind B Orstad ⋅ Henrik Schirmer ⋅ Arian Ranjbar
We introduce EchoXFlow, a clinical echocardiography dataset for learning from ultrasound in its native acquisition geometry rather than from scan-converted Cartesian videos. Existing public datasets offer limited opportunities to study cross-modal relationships between cardiac anatomy, myocardial motion, and blood flow, as Doppler is typically absent or fused as RGB overlays, and acquisitions are released after lossy vendor display processing. EchoXFlow comprises 37125 recordings from 666 routine-care examinations, preserving the timing, geometry, and modality relationships needed for physically grounded echo learning. Each recording is retained as separable modality-specific streams: temporally resolved 1D, 2D, and 3D data alongside multiple Doppler modalities, paired with a synchronized ECG. Clinical annotations span guideline-based measurements to dense 2D myocardial contours and 3D left-ventricular endocardial meshes. With its associated open-source tooling, EchoXFlow enables cross-modal, acquisition-aware learning tasks that cannot be formulated from conventional scan-converted videos alone, and serves as a testbed for 4D vision and physically grounded multi-modal learning more broadly.
Show more
DynaCell: an Evaluation Framework for Dynamic 3D Virtual Staining of Live Cells
Alexandr Kalinin ⋅ Dihan Zheng ⋅ Taylla Milena Theodoro ⋅ Ivan Ivanov ⋅ Eduardo Hirata Miyasaki ⋅ See-Chi Lee ⋅ Aofei Liu ⋅ Sricharan Reddy Varra ⋅ Talon Chandler ⋅ Soorya Pradeep ⋅ Chad Liu ⋅ Manuel D Leonetti ⋅ Carolina Arias ⋅ Bo Huang ⋅ Shalin Mehta
Imaging of dynamic cellular responses to perturbations requires live-cell measurements, posing a challenge for fluorescence-based assays that can visualize molecular phenotypes by specific labeling but often affect cell health. Virtual staining helps by predicting fluorescence-like signals from label-free microscopy. Although many models have been developed for this task, their performance on volumetric time-lapse data remains poorly characterized. We address this gap by introducing DynaCell, an evaluation framework for 3D time-lapse virtual staining of live cells, comprising a new paired label-free and fluorescence 3D time-lapse imaging dataset, a suite of baseline models, and a three-tier metric panel measuring pixel-level fidelity, organelle segmentation performance, and single-cell phenotypic similarity. Using DynaCell, we study model behavior across 2 cell types and microscopes, 4 organelles, and 3 perturbation states. We find that regression baselines better predict the spatial localization of target organelles and are more robust to label-free input distribution shifts, while the generative baseline better captures population-level phenotype distribution across cells and organelles. These differences expose trade-offs between preservation of structural and phenotypic fidelity that are obscured by single-metric evaluation, and suggest how virtual staining can support downstream tasks such as localization, segmentation, tracking, and phenotypic profiling. DynaCell provides reusable data, code, checkpoints, and documentation for assessing virtual stains for live-cell biological measurements.
Show more
Political Neutrality as Balanced Approval: A Large-Scale Human Evaluation of AI Responses
Jonathan Stray ⋅ David Yang ⋅ Steven Luo ⋅ Miu N Takagi ⋅ Serina Chang
As AI systems increasingly shape political views, defining and evaluating AI political neutrality is an urgent problem. Here, we propose a new definition of AI political neutrality and design a large-scale user study to test it, releasing a new dataset PARETO with 6,696 participants and 187,488 evaluations of AI responses. Our definition follows a simple principle grounded in political theory: when asked about a controversial issue, an AI model should generate responses that maximize approval across groups with opposing viewpoints, while balancing approval between groups. This maximum equal approval definition allows empirical testing of whether an AI response is “neutral” and generalizes to any political context without pre-supposing a single left-right axis of division. We construct a benchmark of controversial US issues, with prompts sourced from politically charged questions on Reddit and responses from frontier AI models, and recruit human participants to rate AI responses. Across all 20 issues, we find that it is possible for AI responses to achieve high rates of approval on both sides, even as those sides disagree strongly with each other on the substance of the issues. We also find that default responses lean liberal for GPT, Gemini, Claude, and Llama, but not Grok, and that user prompts with political charges are harder to respond to than neutral prompts. This work introduces a rigorous definition and benchmark of AI political neutrality, and a dataset to measure progress toward it.
Show more
AesGI-Bench: Benchmarking and Evaluating the Aesthetic Quality of AI-Generated Images via Large Multimodal Models
Wang Jiarui ⋅ Xubo Su ⋅ Huiyu Duan ⋅ WeizeSun ⋅ Ziheng Jia ⋅ Wang Juntong ⋅ Yu Zhao ⋅ Guangtao Zhai ⋅ Xiongkuo Min
The rapid advancement of generative models has led to a new era of AI-generated images (AIGIs). Early generation models often suffered from basic defects, such as geometric distortions and poor text-image alignment. Modern generation models have largely overcome these issues, narrowing the perceptual gap between AI-generated and real-world images, thereby shifting evaluation from basic correctness to high-level aesthetic excellence. In this paper, we introduce **AesGI-Bench**, a comprehensive and multidimensional benchmark for fine-grained aesthetic evaluation of AIGIs. Our benchmark covers 18 representative generation models, 19,485 images, and 117K pairwise comparisons from perspectives of visual aesthetic, technical quality, and style alignment. Based on AesGI-Bench, we propose **AesGI-Assessor**, a lightweight and human preference-aligned aesthetic evaluation model for AIGIs. Built upon Qwen3.5, AesGI-Assessor achieves a **pairwise-to-score** learning paradigm, using tie-aware pairwise loss for training and three lightweight score heads for multi-dimensional aesthetic score prediction. Although trained from pairwise human preferences, it learns to predict precise continuous scores for individual images, enabling both fine-grained image-level evaluation and model-level ranking. Furthermore, we explore the potential of large multimodal models (LMMs) as automatic aesthetic evaluators. Experiments show that LMM-based evaluators better align with human aesthetic preferences than conventional metrics. AesGI-Assessor achieves the highest agreement with human judgments, validating its effectiveness as a lightweight, preference-aligned evaluator for fine-grained AIGI aesthetic assessment. The database and codes are publicly available.
Show more
From Pixels to Concepts: Do Segmentation Models Understand What They Segment?
Shuang Liang ⋅ Zeqing Wang ⋅ Yuxian LI ⋅ Xihui Liu ⋅ Han Wang
Segmentation is a fundamental vision task underlying numerous downstream applications. Recent promptable segmentation models, such as Segment Anything Model 3 (SAM3), extend segmentation from category-agnostic mask prediction to concept-guided localization conditioned on high-level textual prompts. However, existing benchmarks primarily evaluate mask accuracy or object presence, leaving unclear whether these models faithfully ground the queried concept or instead rely on visually salient but semantically misleading cues. We introduce CAFE: \textbf{C}ounterfactual \textbf{A}ttribute \textbf{F}actuality \textbf{E}valuation, a novel benchmark for evaluating concept-faithful segmentation in promptable segmentation models. Our \textbf{CAFE} is built on attribute-level counterfactual manipulation: the target region and ground-truth mask are preserved, while attributes such as surface appearance, context, or material composition are modified to introduce misleading semantic cues. The benchmark contains 2,146 paired test samples, each consisting of a target image, a ground-truth mask, a positive prompt, and a misleading negative prompt. These samples cover three counterfactual categories: Superficial Mimicry (\textbf{SM}), Context Conflict (\textbf{CC}), and Ontological Conflict (\textbf{OC}). We evaluate various model types and sizes on our CAFE. Experiments reveal a systematic gap between localization quality and concept discrimination: models often generate accurate masks even for misleading prompts, suggesting that strong mask prediction does not necessarily imply faithful semantic grounding. Our CAFE provides a controlled benchmark for diagnosing whether promptable segmentation models perform concept-faithful grounding rather than shortcut-driven mask retrieval.
Show more
TRACE: Tourism Recommendation with Accountable Citation Evidence
Zixu Zhao ⋅ SIJIN WANG ⋅ Yu Hou ⋅ YUANYUAN XU ⋅ Yufan Sheng ⋅ Xike Xie ⋅ Wenjie Zhang ⋅ Won-Yong Shin ⋅ Xin Cao
Tourism is a high-stakes setting for conversational recommender systems (CRS): a plausible-sounding suggestion can waste real money and trip time once a traveler acts on it. Existing CRS benchmarks primarily evaluate systems with a single Recall@$k$ score over entity mentions, and tourism-specific resources add spatial or knowledge-graph context, yet none of them couple multi-turn recommendation with verbatim review-span evidence and rejection recovery. This leaves an evaluation gap for tourism recommendation that is simultaneously *trustworthy*, *verifiable*, and *adaptive*: recommend the right point of interest (POI) for multi-aspect preferences (such as cuisine, price, atmosphere, walking distance), justify each suggestion with verifiable evidence from prior visitors so the traveler can act without trial and error, and recover when the first recommendation is rejected mid-dialogue. We introduce **TRACE**, where each item is *a multi-turn tourism recommendation dialogue with review-span citations and explicit rejection turns*: 10,000 dialogues over 2,400 Yelp POIs and 34,208 reviews across eight U.S. cities, paired with 14 retrieval, planning, and LLM baselines, along with 25 metrics organized under *Accuracy*, *Grounding*, and *Recovery*. Across these baselines, TRACE reveals the **Three-Competency Gap**: LLM Zero-Shot leads in closed-set Recall@1 and rejection recovery but cites less densely than retrievers; non-LLM retrievers achieve surface-verbatim grounding but with low accuracy; Multi-Review Synthesis fails on Recovery. The Grounding Score agrees with human citation precision (Spearman $\rho = +0.80$, $p < 10^{-20}$), and paired $t$-tests reproduce the per-baseline ranking ($p < 0.01$ on the dominant contrasts). TRACE reframes accountable tourism recommendation as a joint target (right POI, verifiable evidence, adaptive repair) rather than a single-axis leaderboard.
Show more
ClawBenchPro: Benchmarking How Well Agent Harnesses Work
Yufei Liu ⋅ Yirong Zeng ⋅ Yuxiang He ⋅ Yutai Hou ⋅ Yuxian Wang ⋅ Qunyao Du ⋅ WangXu ⋅ Xiao Ding ⋅ Wu Ning ⋅ Hao Cong ⋅ Dandan Tu ⋅ Qixun Zhang ⋅ Ting Liu
The evolution of Large Language Models (LLMs) from conversational assistants to autonomous agents is increasingly driven by agent harnesses (e.g., OpenClaw), which provide the operational substrate for executing complex, multi-step workflows in real-world environments. However, existing claw-supported benchmarks often suffer from narrow domain coverage and reliance on single-harness evaluations, leading to evaluation distortion and limited diagnostic diversity. To bridge this gap, we present ClawBenchPro, a comprehensive benchmark comprising 1.0k samples across 43 distinct domains. By integrating skill augmentation, environmental heterogeneity, and multi-session dependencies, we replicate complexity workflow at scale. Extensive experiments across 14 frontier LLMs and four representative harnesses, show model's average scores clustering at 60-74\%. We identify a critical domain disparity: models excel in cyber-native tasks but struggle with real-world operational workflows. Moreover, a pervasive reliability bottleneck emerges, where incremental progress rarely translates to zero-defect execution. Comparative results show that while OpenClaw incurs higher computational costs, NanoClaw offers superior cost-efficiency (despite a 3.2–6.5\% accuracy trade-off), and Hermes-Agent emerges as the most stable framework for claw-style interactions. ClawBenchPro establishes a rigorous, reproducible standard for diagnosing agent limitations and advancing next-generation harness architectures.
Show more
PhysTC: A Physics-Enhanced Dataset and Architecture for High-Precision Tropical Cyclone Forecasting
Zhaoran Feng ⋅ Xuanhong Chen ⋅ Zengbing Chen ⋅ Shengjun Wu ⋅ Bin He ⋅ Kairui Feng
A critical observability gap arises in multiscale chaotic systems when coarse observations smooth out high-frequency extremes. For tropical cyclones, this makes standard reanalysis products insufficient for high-precision intensity prediction, as spectral truncation can mask peak winds by over $30\%$. In this work, we introduce Physics-enhanced Tropical Cyclone forecasting (PhysTC), a physics-enhanced benchmark framework for forecasting tropical cyclone track and intensity from globally consistent coarse observations. We curate a harmonized multi-basin dataset spanning 1950--2023, incorporating scale-robust physical predictors to mitigate resolution-induced bias. Building on this dataset, we propose the Physics-enhanced Tropical Cyclone Network (PhysTCN), a model that integrates synoptic spatial structure, storm-following temporal history, and physical consistency constraints to infer intensity-relevant dynamics beyond direct grid-space readout. PhysTCN significantly outperforms state-of-the-art baselines, with particularly strong gains in recovering extreme wind speeds that are underestimated in reanalysis data. This work provides a scalable framework for physically guided forecasting in data-sparse, cross-scale dynamical systems.
Show more
MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
Fangda Ye ⋅ Yuxin Hu ⋅ Pengxiang Zhu ⋅ Yibo Li ⋅ Ziqi Jin ⋅ Yao Xiao ⋅ Yibo Wang ⋅ Lei Wang ⋅ Zhen Zhang ⋅ Lu Wang ⋅ Yue Deng ⋅ Bin Wang ⋅ Yifan Zhang ⋅ Liangcai Su ⋅ Xinyu Wang ⋅ He ZHAO ⋅ CHEN WEI ⋅ Qiang Ren ⋅ Bryan Hooi ⋅ Bo An ⋅ Shuicheng Yan ⋅ Lidong Bing
Recent progress in deep research systems has been impressive, yet their evaluation remains limited not only by benchmark coverage but also by methodology. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce **MiroEval**, an evaluation framework for deep research systems, accompanied by a 100-task benchmark (70 text-only, 30 multimodal) constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality evaluation via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 11 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with declines concentrated in synthesis and process rather than factuality. Robustness experiments, a human ranking study, and three-annotator verification confirm the reliability of both the evaluation framework and the benchmark. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.
Show more
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
Jinhao Jing ⋅ Zheng Ma ⋅ Jinwei Liang ⋅ Qiannian Zhao ⋅ Shuang Chen ⋅ Jing Yang ⋅ Prayag Tiwari ⋅ Jingjing Bai ⋅ Benyou Wang ⋅ Por L Yee ⋅ Zhan Su ⋅ Lewei Lu
Large Multimodal Models (LMMs) often struggle with geometric reasoning due to visual hallucinations and a lack of mathematically precise Chain-of-Thought (CoT) data. To address this, we propose the GeoSym Engine, an automated and scalable neuro-symbolic framework. By leveraging a type-conditional grammar and an analytic SymGT Solver, it derives exact symbolic ground truths and seamlessly integrates with a robust rendering pipeline to produce high-precision geometric diagrams. Using this engine, we construct GeoSym127K, a difficulty-stratified dataset featuring 51K high-resolution images, 127K questions with symbolic ground truths, and 55K answer-verified CoT QA pairs. We also introduce GeoSym-Bench, an expert-curated suite of 511 complex samples for rigorous evaluation. Through extensive supervised fine-tuning (SFT), we demonstrate that GeoSym drives concentrated improvements specifically on diagram-dependent and multi-step geometry tasks. Our Qwen3-VL-8B model gains an absolute +22.21% on the MathVerse Vision-Only subset and reaches 61.52% (+6.19% improvement) on WeMath, mitigating long-horizon logic fragmentation and outperforming advanced closed-source models like Doubao-1.8. Furthermore, applying Reinforcement Learning with Verifiable Rewards (RLVR) via GRPO reveals that initializing from structural SFT checkpoints substantially elevates the performance ceiling over zero-shot RL. Driven by deterministic exact-match signals, this showcases the robust scaling potential of our verifiable reasoning synthesis. Datasets and code are available at https://huggingface.co/datasets/Tomie0506/GeoSym127K and https://github.com/Tomie56/GeoSym127K
Show more
Static Recovery Is Not Dynamic Stability: Dynamics-Aware Benchmarking of Protein Motif Scaffolding
Emile de Bruyn ⋅ Moritz Schäffler ⋅ Jannik Schneider ⋅ Sebastian R Schmidt ⋅ Chanwoong Hwang ⋅ Aleena Siji ⋅ Utkarsh Upadhyay ⋅ Alexander Schug ⋅ Stefan Bauer ⋅ Artur Yakimovich ⋅ Stefan Kesselheim ⋅ Alina Bazarova
Protein motif scaffolding benchmarks aim to measure whether generative models can build scaffolds around functionally important residue sets. Their conclusions depend on two coupled choices: which motifs are tested and which criteria define success. Existing benchmarks typically use small hand-curated motif sets and static refolding-based criteria such as motif recovery, self-consistency, and structural novelty. However, protein function often depends on conformational ensembles and transitions, raising the question of whether static refolding success translates to dynamic fidelity of motifs and scaffolds. Here, we address both benchmark scope and success criteria. We construct the largest systematically derived benchmark of structurally conserved functional motifs from PROSITE-linked experimental structures, yielding 220 cases from 174 PROSITE motif-pattern entries of varying conformations. We evaluate six recent scaffold-generation models and find that performance separates into target coverage and unique-solution yield: RFdiffusion and Proteina solve more targets, whereas ESM3 produces more unique successful scaffolds on the targets they solve. We then perform all-atom molecular dynamics simulations for a set of statically solved designs, comparing motif drift, scaffold drift, motif-contact preservation, and motif-dihedral free-energy landscapes against reference simulations. Of 55 statically solved model–problem pairs, 31 satisfy our MD-supported criterion and only 8 pass all checks. Refolding success is therefore useful evidence for local motif recovery, but not sufficient evidence for dynamical fidelity of the full scaffold, exposing misalignment between static benchmark success and downstream dynamical behavior. The mismatch is motif-dependent: compact, self-contained motifs are often MD-supported, whereas interface-like or context-dependent motifs fail more often. We show that motif-scaffolding models have varying performance on three complementary axes: target coverage, unique-solution yield, and MD-supported dynamical fidelity.
Show more
ConnectomeBench2: A Unified Benchmark for Automated Connectomic Proofreading
Jeff Brown ⋅ Tim Farkas ⋅ Gleb Razgar ⋅ Edward Boyden
Proofreading—correcting segmentation errors in 3D brain reconstructions—is the rate-limiting step in synapse-resolution connectomics. We release ConnectomeBench2, a unified multi-species dataset of over 302,401 expert-labeled proofreading decisions with >2,000,000 associated images spanning four major open connectomes (mouse, human, zebrafish, fly), spanning both split and merge error correction. Trained on this dataset, a single Vision Transformer with shared encoders for mesh geometry and electron microscopy reaches human-level accuracy across species for split error correction and merge error identification; with performance scaling with data size and improving with species diversity. Beyond accuracy, we show that the model is well-calibrated within distribution, that measures of distribution distance predict where calibration and accuracy will degrade on unseen data, and that connectomics-specific pretraining and active learning-based sample selection show potential to substantially reduce the labeling effort needed to extend to new species and brain regions. The benchmark provides the infrastructure to train and evaluate increasingly capable vision models for connectomic proofreading.
Show more
ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
Udari Madhushani Sehwag ⋅ Zhengyang Shan ⋅ Heming Liu ⋅ Dileepa Lakshan ⋅ Joseph Brandifino ⋅ Max Fenkell
Clarification-seeking behavior is widely regarded as a desirable property of LLM agents, enabling them to resolve ambiguity before acting on underspecified tasks. However, the security implications of this interaction pattern remain unexplored. We investigate whether the transition from standard execution to a clarification-seeking state increases an agent's susceptibility to prompt injection attacks. We introduce **ASPI** (**A**mbiguous-**S**tate **P**rompt **I**njection), a benchmark of 728 task-attack scenarios that isolates clarification as a distinct agent state and measures how this state transition affects vulnerability under controlled conditions. Each benchmark instance is evaluated under matched execution and clarification settings: in the execution setting, the agent acts on a fully specified instruction and encounters adversarial content only through tool-returned data; in the clarification setting, the agent must first request and incorporate additional user input before acting. We evaluate ten frontier LLMs and find that clarification-seeking consistently and substantially amplifies vulnerability. For instance, attack success rises from 1.8\% to 34.0\% for o3 and from 2.2\% to 35.7\% for Gemini-3-Flash. A decomposition analysis reveals that this gap reflects both a state-dependent shift in how models process incoming content and a channel-specific effect arising from the agent-solicited clarification interface. These findings demonstrate that standard execution-time security evaluation systematically underestimates the attack surface of interactive agents, and that robustness under fully specified tasks does not translate to robustness under ambiguity. For reproducibility our data and source code is available at https://anonymous.4open.science/r/aspi-5074.
Show more
STRIDE: Automated Evaluation of Text-to-Trajectory Alignment across Diverse Contexts
Wanchun Ni ⋅ Tao Qi ⋅ Leonel Aguilar ⋅ Jiugeng Sun ⋅ Marlene Wagner ⋅ Verena Zimmermann ⋅ Mennatallah El-Assady
Language-conditioned trajectory generation is here, but its evaluation has not kept pace. Existing pedestrian trajectory metrics compare trajectories with real-world human data. This does not scale to text-to-trajectory generation across diverse contexts, as collecting human trajectories for every scenario is costly and infeasible. Moreover, pedestrian behavior is heterogeneous and context-dependent, with no single metric as the correct answer, and current evaluation frameworks are not transferable to this domain. These challenges make scalable, reliable evaluation difficult. We introduce STRIDE, the first framework for evaluating context alignment between scenario descriptions and pedestrian trajectories. STRIDE addresses these challenges through three design choices. First, we derive our VRDST evaluation protocol from sociological theories to define a complete evaluation space. Second, it decomposes high-level context into scenario-adaptive behavioral questions. Third, every question is resolved against a deterministic measurement tool library that yields reproducible answers. Together, STRIDE enables complete, verifiable, automated, and scalable evaluation across diverse contexts without requiring human trajectory data. We instantiate STRIDE in the crowd domain as STRIDE-Bench, comprising 1K scenarios, 6K behavioral questions, and 11K measurements with calibrated expected answers across 30 real-world maps. Comprehensive human validations show that STRIDE-Bench is consistent with human behavior and judgment, achieving 80% human agreement. We further evaluate several text-to-trajectory models, finding limited context-alignment capability and persistent challenges in fine-grained context conditioning. We believe that the STRIDE framework provides a first step toward principled evaluation of context-aligned pedestrian trajectory generation. Code: https://anonymous.4open.science/r/STRIDE-Bench-7006 Dataset: https://huggingface.co/datasets/anonymous1ads34/STRIDE-Bench
Show more
Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests
Patrick Rim ⋅ Tom Long ⋅ Ekta Prashnani ⋅ Ruth Rosenholtz ⋅ Ben Boudaoud ⋅ Peter Xenopoulos ⋅ Alex Wong ⋅ Joohwan Kim ⋅ Jae-Hyun Jung
Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual, such as recognizing object boundaries, or cognitive, such as reasoning about occlusion to infer hidden geometry. We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats (free-response text, multiple-choice images, and image editing) alongside a human baseline. The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy, supporting the value of the proposed decomposition as both a diagnostic tool and a training signal.
Show more
MM-SCALE: Evaluating Evidence-Grounded Moral Judgment in Vision-Language Models
Eunkyu Park ⋅ Wesley Deng ⋅ Cheyon Jin ⋅ Matheus Kunzler Maldaner ⋅ Jordan Wheeler ⋅ Jason I Hong ⋅ Hong Shen ⋅ Adam Perer ⋅ Kenneth Holstein ⋅ Motahhare Eslami ⋅ Gunhee Kim
Vision-Language Models increasingly make socially consequential judgments from image-text inputs, yet current evaluations often stop at verdict-level accuracy: whether the final label is human-aligned, not whether it follows from the correct evidence. We introduce MM-SCALE (Multimodal Moral SCALE), a benchmark for evidence-grounded moral judgment built around a design choice: each image is paired with multiple action scenarios, so models must compare how different actions interact with the same visual context rather than score images in isolation. MM-SCALE contains 8,444 image contexts and 21,977 action scenarios, each annotated with a 5-point moral acceptability rating and a modality-grounding label indicating whether the human judgment relied on text, image, or both. Three tasks evaluate whether models (i) assign calibrated scalar moral scores, (ii) preserve human orderings within a shared visual context, and (iii) ground rationales in the evidence source humans found critical to judgment. Across models, aggregate metrics overstate alignment: NDCG@5 remains high across models, while within-image pairwise accuracy remains only 0.53--0.62 even on scenario pairs whose human mean ratings differ by at least one point. CoT has inconsistent effects on calibration and does not reliably improve within-image ordering. Evidence-grounding evaluations further show that models often describe visual context without making it critical for the verdict. These results reveal an evidence-grounding failure in current VLMs and position MM-SCALE as an evaluation testbed for moral judgment under shared image contexts. This dataset includes potentially harmful and sensitive visual content. Images are intended solely for evaluation.
Show more
VLAN: Vision-Language Accessible Navigation
Jie Hu ⋅ Yu Zheng ⋅ Wenzhi Wu ⋅ Weixing Feng ⋅ Cunlei Cheng ⋅ Jiaming Zhang
Accessible mobility is user-conditioned rather than scene-intrinsic, as walkable paths for one embodiment are inaccessible to another. Yet, existing navigation benchmarks evaluate traversability under a fixed and limited embodiment. We present a dataset called **Vision-Language Accessible Navigation (VLAN)**. The dataset contains 1000 scenes, 20K navigation trajectories, and 160K keyframes, spanning diverse lighting, environmental conditions, scene semantics, and path geometries. For each trajectory, VLAN provides embodiment-specific actions and accessibility features for legged and wheeled agents, including humanoid and quadruped robots, as well as people with visual or mobility impairments. Furthermore, everyday obstacles and accessibility labels are included and categorized as Passable, Bypass, HighRisk, and Blocked. Unlike goal-only evaluations, our benchmark measures whether model outputs are both physically safe and aligned with user-specific constraints through goal-reaching, collision-aware, and accessibility-aligned metrics. We benchmark modern vision-language models and find that current systems often fail to consistently adapt obstacle relevance, path selection, and safety criteria across user profiles. These results suggest that accessibility-aware navigation requires dedicated datasets, tasks, and metrics beyond conventional embodied navigation evaluation. All data and code are open-source at https://huggingface.co/datasets/anonymousxxd/VLAN.
Show more
Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action
Ben Slater ⋅ Lucy G Cheke ⋅ John Burden ⋅ Winnie Street
Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in increasingly agentic and autonomous forms demands new evaluations. In this paper we evaluate an agent's ability to induce specific belief states in other agents by taking actions rather than using conversational persuasion, a capability we call Non-Conversational Planning ToM (NCP-ToM). NCP-ToM is likely to be essential for many agent use-cases, including within user-assistant interactions and pedagogical contexts, but may also present manipulation or misinformation risks. Using a novel framework, NCP-ExploreToM, we subvert the conventional task structure by providing models with a set of belief state goals and requiring them to move objects or direct characters into rooms to achieve their goals. We evaluated six frontier models, including GPT-5, Gemini 2.5 Pro and the Claude 4 series, and a cohort of human participants, across 600 task instances. GPT-5 was successful on approximately 80\% of tasks in the agentic setting, and was the only model to outperform human participants on our task, but was still less robust than humans across contexts. We additionally found that all models, like humans, performed better on tasks inducing true belief states than false belief states, which is a positive signal for alignment efforts. These findings highlight emerging social-reasoning capabilities in LLMs for non-conversational task completion and underscore the necessity of agentic evaluations for understanding the safety and alignment of autonomous social agents.
Show more
Signature-Kernel Evaluation Metrics for Robust Probabilistic and Tail-Event Forecasting
Benjamin R Redhead ⋅ Thomas L Lee ⋅ Peng Gu ⋅ Víctor Elvira ⋅ Amos Storkey
Standard sample-based metrics like Continuous Ranked Probability Score (CRPS) and Quantile Loss (QL) are frequently used in the evaluation of probabilistic time-series forecasting. However, these metrics fail to capture multivariate correlations or assess the ability to capture distribution tails. These global metrics are insensitive to errors on high-utility events in distributional tails due to over-representation of the distribution's body. To accurately measure distributional fidelity on tail-regions and regions of high utility while still having a proper scoring rule, we introduce a family of censored signature kernel metrics. Our proposed metrics concentrate evaluation of forecasts to a focus region, representing high-utility or distributional tails, by collapsing the body of the distribution to a single pivot. Our benchmarks on time-series foundation models (TSFMs) reveal that while a clear ranking can be formed for distribution capture, there is no clear winner for tasks like systemic load prediction. This indicates that models with a strong ability to capture an overall distribution do not produce forecasts with high downstream utility. To encourage the use of the proposed metrics we open-source the efficient signature kernel (ESK) library. This library facilitates batch computation, with custom triton kernels, achieving a speed-up of up to 3.58$\times$ compared to implementations via the popular SigKernel library. Code for our metrics are available on GitHub.
Show more
MyoChallenge 2025: A New Benchmark for Human Athletic Intelligence
Cheryl Wang ⋅ Chun Kwang Tan ⋅ Balint Hodossy ⋅ Shirui Lyu ⋅ Jun Guo ⋅ Wentao Zhao ⋅ Huaping Liu ⋅ Chengkun Li ⋅ Merkourios Simos ⋅ Bianca Ziliotto ⋅ Alexander Mathis ⋅ Siyuan Liu ⋅ Jiahao Chen ⋅ Shanlin Zhong ⋅ Bo Jiang ⋅ Ci Song ⋅ Yaoye Zhu ⋅ Chenhui Zuo ⋅ Yanan Sui ⋅ Irfan Refai ⋅ Massimo Sartori ⋅ Guillaume Durandau ⋅ Vikash Kumar ⋅ Vittorio Caggiano
Athletic performance represents the pinnacle of human motor intelligence, demanding rapid choices, precise control, agility, and coordinated physical execution. Replicating this seamless combination of capabilities remains elusive in current artificial intelligence and robotic systems. Concurrently, understanding the biological mastery of these movements is hindered because complex muscle coordination is rarely measured in vivo due to the limitations of physical equipment. To bridge this fundamental gap in understanding, $\textit{MyoChallenge}$ at NeurIPS 2025 established a pioneering benchmark for motor control intelligence in sports, leveraging high-fidelity musculoskeletal models within physics simulation combined with machine learning-driven algorithms. The competition introduces two distinct tracks emphasizing either upper or lower limbs control: a table tennis rally task utilizing a biomechanic upper limb composed of an arm with a hand (MyoArm) and a trunk (MyoTorso); and a soccer penalty kick using a biomechanic model of legs (MyoLeg) and a trunk (MyoTorso). Marking the fourth iteration of the MyoChallenge series, this event attracted almost 70 teams and over 560 submissions globally, uniting a diverse community ranging from physicians and neuroscientists to machine learning experts. The competition facilitated the development of several state-of-the-art control algorithms for a musculoskeletal system capable of sports agility, leveraging techniques such as physics-based motion planners, on-policy behaviour cloning, hierarchical planning, and muscle synergies that significantly surpassed our provided baseline performance. By integrating standardized tasks and physiologically realistic models into the open-source framework of MyoSuite, MyoChallenge'25 serves as a reproducible and reusable testbed to accelerate interdisciplinary research across machine learning, biomechanics, sports science, and neuroscience. Project page: https://www.myosuite.org//myochallenge/myochallenge-2025
Show more
CME–SpectrumBench: Can LLMs Analyze Condensed Matter Spectral Data?
Jin Gene Wong ⋅ Anjney Midha ⋅ Joseph Tennyson ⋅ Wei-Lin Chiang ⋅ Ion Stoica ⋅ Zhi-Xun Shen
Large Language Models (LLMs) are presently well assessed by expert–level mathematics and physics benchmarks which focus on theoretical, symbolic reasoning. However, we lack experimental benchmarks that evaluate data analysis in frontier research problems, a capability that future domain–specific foundation models must have. To this end, we present CME–SpectrumBench, an expert–designed benchmark consisting of 190 questions that measures how reliably frontier LLMs analyze research–level spectral scientific data in condensed matter experiments (CME). Using simulated CME spectral data, we prompt models to identify structures from 2D intensity arrays in the face of noise and limited instrumental resolution, a test of dense context reasoning. Models are scored by how accurately their numerical answers approach the ground truth. Under direct query, scores generally pale in comparison to handwritten code due to clear failure modes: models can find simple features but struggle to trace patterns or compute derived quantities, and performance degrades with increased data resolution and noise/convolution levels. LLM performance becomes comparable to handwritten code by allowing tool use (access to a Python interpreter) with/without ReAct prompting, in effect mapping questions onto agentic code generation problems. Our findings show that frontier models can serve as research assistants by interfacing with data through code, but still face difficulties when processing information directly from large datasets — a key barrier future foundation models must overcome.
Show more
MARBLE: an Agent Benchmark for Spatial Reasoning and Visual Abstraction
Yulun Jiang ⋅ Naël Ouerghemi ⋅ Yekun Chai ⋅ Maria Brbic ⋅ Michael Moor
A central challenge for multimodal language models (MLLMs) in real-world tasks is to abstract visual observations into structured spatial states and reason over how those states change under actions over long-horizons. Existing multimodal benchmarks often focus on static question answering, where answers can be directly extracted from images, or assume symbolic states are available and bypass the visual-to-spatial abstraction process. We present MARBLE, a diagnostic benchmark for spatially grounded multimodal reasoning and planning across three agent environments: Cube, Maze and Blox. MARBLE spans 2D and 3D spatial puzzles under geometric constraints, dynamic layouts, and spatial transformations, with difficulty systematically controlled along two axes: observation format and planning horizon. Extensive evaluations show that frontier MLLMs struggle on MARBLE, with the strongest model GPT-5.4 achieving only a 7.3\% average success rate across the three environments in the hardest setting. Pairwise ablations on the open-weight models identify visual-to-spatial abstraction as the major bottleneck: when models must infer spatial states from images instead of receiving human-annotated states, success drops from 50.7\% to 3.7\% for Gemma-4-31B and from 87.9\% to 6.5\% for Kimi-K2.6. The results indicate that models often fail to construct precise spatial states from visual observations before planning, a task that is straightfoward for human. Overall, MARBLE reveals the gap of reliable visual-to-spatial for frontier MLLMs and provides a controlled testbed for measuring progress in spatially grounded multimodal reasoning and planning.
Show more
OpenView: Empowering MLLMs with Out-of-view VQA
Qixiang Chen ⋅ Cheng Zhang ⋅ Chi-Wing Fu ⋅ Jingwen Ye ⋅ Jianfei Cai
Recent multimodal large language models (MLLMs) show great potential in natural image understanding. Yet, they perform well, mainly on reasoning in-view contents within the image frame. This paper presents the first study on out-of-view (OOV) understanding, i.e., the ability to reason about objects, activities, and scenes beyond the visible frame of a perspective view. Our technical contributions are threefold. First, we design OpenView, a four-stage pipeline that leverages panoramic imagery for large-scale and diverse multi-choice VQA synthesis, where full-scene coverage enables flexible view framing while preserving global scene awareness. Second, we curate OpenView-Dataset, a high-quality synthetic dataset from diverse real-world panoramas to empower MLLMs upon supervised fine-tuning. Third, we build OpenView-Bench, a benchmark that jointly measures choice and rationale accuracy for interpretable and diagnosable evaluation. Experimental results show that despite having a large gap from human performance in OOV VQA answer selection, multiple MLLMs, upon empowered by OpenView, can consistently boost their performance, uplifted from 48.3% to 62.3% on average. Code, benchmark, and data will be released.
Show more
SENTINEL: A Multi-Level Formal Framework for Safety Evaluation of Foundation Model-based Embodied Agents
Simon Zhan ⋅ Philip Wang ⋅ Yao Liu ⋅ Yiyan Peng ⋅ Yiqi Lyu ⋅ Zinan Wang ⋅ Qineng Wang ⋅ Zhian Ruan ⋅ Xiangyu Shi ⋅ Xinyu Cao ⋅ Frank Yang ⋅ Zhenyang Ni ⋅ Kangrui Wang ⋅ Ruohan Zhang ⋅ Huajie Shao ⋅ Manling Li ⋅ Qi Zhu
We present **SENTINEL**, a framework for formally evaluting the physical safety of foundaiton model (FM)-based embodied agents. SENTINEL is the first to provide multi-level safety evaluation across semantic interpretation, plan generation, and physical execution within a unified formal framework. Unlike prior methods that rely on heuristic rules or subjective FM judgments, SENTINEL grounds practical safety requirements in formal temporal logic (TL) semantics that can precisely specify state invariants, temporal dependencies, and timing constraints. It employs a multi-level verification pipeline where (i) at the semantic level, intuitive natural language safety requirements are formalized into TL formulas and the agent's understanding of these requirements is probed for alignment with the TL formulas; (ii) at the plan level, high-level action plans and subgoals generated by the agent are verified against the TL formulas to detect unsafe plans before execution; and (iii) at the trajectory level, multiple execution trajectories are merged into a computation tree and efficiently verified against physically-detailed TL specifications for a final safety check. We apply SENTINEL in VirtualHome and Ai2Thor, and formally evaluate multiple FM-based embodied agents against diverse safety requirements. Our experiments show that by grounding physical safety in temporal logic and applying verification methods across multiple levels, SENTINEL provides a rigorous foundation for systematically evaluating the safety of FM-based embodied agents in simulation-based physical environments, and can effectively expose potential safety violations in interpreting, planning, and executing the tasks.
Show more
PULSE: A Synchronized Five-Modality Dataset for Sensorimotor Coordination in Long-Horizon Daily Activities
Wolin Liang ⋅ Yang Gao ⋅ Peiyu Yan ⋅ YueXiang Hu ⋅ Yingjing Xiao ⋅ Xiongfeng Ying ⋅ YingNian Guo ⋅ Cheng Zhang ⋅ Zhanpeng Jin
Existing human-activity datasets typically cover only one or two sensor channels and often focus on short, isolated actions, leaving long-duration compositional tasks largely unsupported as benchmarks for temporal cross-modal sensorimotor learning. We introduce PULSE ($\mathbf{P}$hysiological $\mathbf{U}$nified $\mathbf{L}$ong-duration $\mathbf{S}$ynchronized $\mathbf{E}$mbodiment), a synchronized five-modality dataset that enables benchmarking temporal cross-modal sensorimotor learning: how models recognize ongoing activities, anticipate future contact, reconstruct unobserved signals, and remain robust when physiological, motion, gaze, inertial, and tactile channels are partially observed. Collected from 40 volunteers across 8 ecologically valid daily-activity scenarios together with a dedicated motion-primitive collection, PULSE records full-body optical motion capture with finger-level hand articulation, surface EMG, binocular eye tracking, wearable IMU, and a fingertip pressure array recording quantitative grip force, all sampled at 100 Hz. Each scenario is a long-horizon compositional task containing many smaller sub-tasks, yielding over 7,700 annotated action segments. Each segment is tagged with a motor primitive, the hand involved, the manipulated object, and a natural-language description with four paraphrased variants. On top of the dataset, we define multiple benchmark tasks for temporal cross-modal sensorimotor learning: scene and fine-grained action recognition, grasp onset anticipation, missing-modality robustness, tactile-driven sub-second grasp-state prediction, cross-modal pressure reconstruction, etc. Together these tasks evaluate how information transfers across physiology, motion, gaze, and touch, rather than reducing the dataset to an activity-label catalog. We evaluate three backbone architectures, nine fusion strategies, seven published baselines, and task-specific models including SyncFuse and DailyActFormer. Detailed task definitions and results are in the Appendices. Data, annotations, and baseline code will be publicly released.
Show more
AVENUE: Audio-Video Editing Understanding and Evaluation
Hayeon Kim ⋅ Yoojin Jang ⋅ Jaejun Yoo
Audio-video (AV) editing aims to modify audio and video content according to a target prompt. Unlike single-modality editing, AV editing requires models to infer a modality-selective edit scope from the prompt alone: determining not only what should change, but also which modality should be preserved. Faithfully evaluating such models therefore requires both (i) benchmarks that span diverse edit types and modality categories, and (ii) evaluation that is itself modality-aware and sample-specific. However, existing AV editing benchmarks provide limited coverage of edit types and modality combinations, while current evaluation systems are often modality-blind and sample-agnostic, making it difficult to assess whether models faithfully preserve the unintended modality. To address these gaps, we introduce AVENUE, Audio-Video EditiNg Understanding and Evaluation, comprising two contributions: (1) a benchmark of 1,291 source clips and 7,957 editing instructions across audio-targeted, video-targeted, and AV-coupled edit types, curated and human-verified from VGGSound; and (2) a sample-specific, modality-aware evaluation framework that specifies, for each sample, both the intended change and the content that must remain intact. We evaluate representative AV editing models spanning three editing paradigms — joint, sequential, and separate — providing the first systematic analysis of modality-selectivity across paradigms. Our findings reveal a fundamental open challenge: when editing one modality, existing models frequently induce unintended changes in the other, regardless of paradigm. AVENUE provides a benchmark and modality-aware evaluation framework to drive progress toward more controllable AV editing models.
Show more
Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models
Julian Skirzynski ⋅ Harry Cheon ⋅ Shreyas Kadekodi ⋅ Meredith Stewart ⋅ Berk Ustun
Concept bottleneck models predict outcomes from high-level concepts detected in inputs. Although concepts provide a simple way to reap benefits from interpretability, very few datasets include concept labels. This limits researchers' ability to determine which problems are suitable for these models, isolate the factors that drive their performance or lead to failures, or uncover which algorithms perform well. In this paper, we develop synthetic benchmarks for concept-bottleneck models, focusing on their two main use cases: decision support, in which models assist humans in making better decisions, and automation, in which models handle routine tasks without supervision. Our benchmarks can generate labeled datasets while controlling for properties that affect performance, including data modality, concept choice, annotation quality, and completeness. We demonstrate how the benchmarks can be used to evaluate representative classes of concept bottleneck models. Our demonstrations show how the benchmarks can diagnose failure modes and guide follow-up testing.
Show more
SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark
Taeyoung Yu ⋅ Seonbin Jo ⋅ Jiwon Kim ⋅ Junyoung Byun
Origin-destination (OD) flow forecasting supports decisions about where and how many people move through a city. Most existing public OD benchmarks provide only a partial view of citywide mobility, covering a single transport service or at most two modes. With partial mode coverage, observed demand changes are difficult to separate into shifts across transport modes and changes in total travel. We introduce SeoulMMOD, a large-scale public benchmark for citywide multimodal mobility in Seoul. SeoulMMOD provides three years of hourly OD flows estimating total mobility for 6 urban travel modes across 25 districts and 426 sub-districts, together with travel-time, travel-distance, calendar, rainfall, and point-of-interest (POI) signals. Using SeoulMMOD, we benchmark 14 forecasting baselines under a common protocol and study spatial scaling, joint training versus training one model per mode, and cross-year generalization. Results show that several spatio-temporal models that work at district level do not scale to sub-district OD forecasting, and that naive joint multi-mode training does not consistently improve accuracy. Together, the dataset and benchmark establish a reproducible testbed for city-scale multi-mode OD forecasting across multiple years. Project page: https://anonymous.4open.science/r/SeoulMMOD
Show more
SafeDrug: A Benchmark Dataset for Safety-Critical Pharmacological Reasoning in LLMs
Tengfei Ma ⋅ Yushan Yang ⋅ HOU Jiahao ⋅ Yujie Chen ⋅ Guanghui Ye ⋅ Xuanbai Ren ⋅ Yiping Liu ⋅ Bosheng Song ⋅ xiangxiang Zeng
Large language models (LLMs) have shown strong performance on biomedical tasks, yet evaluating their reasoning in safety-critical pharmacological contexts requires large-scale, structured datasets. We present SafeDrug, a benchmark dataset for systematic evaluation of pharmacological reasoning across multiple task dimensions. SafeDrug integrates heterogeneous sources, including adverse drug event records, drug–drug interaction knowledge, literature-derived evidence, and population-specific information such as child, adult, and older adult groups, into a unified reasoning-oriented QA format. The benchmark comprises two complementary subsets. \textbf{SafeDrug-large} provides broad coverage at scale, while \textbf{SafeDrug-small} is a high-quality human-annotated subset for precise evaluation. It supports tasks such as prediction, risk assessment, mechanism explanation, evidence-grounded QA, and drug replacement across both single-drug and multi-drug settings. Our evaluation shows that LLMs capture outcome-level associations but struggle with mechanistic reasoning, evidence alignment, and population-aware predictions. SafeDrug provides a foundation for reproducible benchmarking and advances research on evidence-grounded and population-aware pharmacological reasoning in LLMs.
Show more
AbsoluteDegradation: A Physics-Inspired Synthetic Film-Degradation Pipeline and Archival Film Restoration Benchmark
Mikołaj Jastrzębski ⋅ Dawid Glinkowski ⋅ Dawid Zieliński ⋅ Daniel Borkowski ⋅ Wojciech Kozlowski ⋅ Kamil Adamczewski
Restoring archival film remains a fundamentally challenging problem due to the absence of paired training data and the lack of standardized evaluation benchmarks. Pristine versions of deteriorated footage are physically unrecoverable, forcing supervised methods to rely on synthetic data that often fail to capture the complex, temporally coherent nature of real film degradation. At the same time, existing real-world datasets are limited in scale, quality, and accessibility, hindering reliable evaluation and fair comparison across methods. We address both limitations with AbsoluteDegradation, a physics-inspired, modular pipeline for synthesizing realistic film degradations, and a new large-scale archival benchmark. The proposed pipeline models the analog-to-digital process as a structured composition of artifact families, incorporating signal-dependent grain, parametric scratches, and temporally coherent camera motion, enabling controlled generation of diverse degradation regimes. In parallel, we introduce a curated dataset of 81,576 high-resolution frames sourced from real archival footage, designed for consistent evaluation under authentic conditions. Together, these contributions provide a unified framework for training and benchmarking restoration models. Extensive experiments across multiple architectures show that models trained with AbsoluteDegradation generalize better to real-world footage, while the proposed benchmark reveals systematic failure modes of current methods. We hope this work establishes a foundation for reproducible and domain-authentic evaluation in archival film restoration.
Show more
DualWorldBench: Can Agents Plan Deliveries across Symbolic and Grounded Worlds?
Yang Chen ⋅ Bin Wen ⋅ Danyang Peng ⋅ Hong-Jie You ⋅ Ziqiao Shang ⋅ Lan-Zhe Guo
Recent multimodal agents have shown promising progress in embodied planning and web interaction. However, existing evaluations often rely on simplified tasks with limited combinatorial structure, leaving it unclear whether current agents can robustly integrate perception and reasoning for multi-goal planning under temporal, physical, and multimodal constraints. We introduce DualWorldBench, a unified benchmark for evaluating agent planning under controlled symbolic--temporal and grounded--topological complexity. Built around a shared pickup--delivery interface, DualWorldBench consists of three complementary components: TextWorld, which isolates symbolic planning under varying task compositions and release dynamics; SimWorld, which evaluates grounded path planning under controlled topological variation with text, image, or multimodal inputs; and DualWorld-VQA, a diagnostic suite spanning perception, scene understanding, and planning. Experiments show that current state-of-the-art agents remain brittle under increasing combinatorial complexity. Multimodal input does not consistently improve planning and can even degrade performance, revealing difficulties in aligning symbolic task specifications with grounded topology. Diagnostics further show that models handle perception or understanding in isolation, but fail when decision-making requires integrating both. These findings establish DualWorldBench as a challenging testbed for agents with tighter perception--reasoning--planning integration. All datasets and code are publicly available at our project website https://dualworld.netlify.app/.
Show more
SPDEBench: An Extensive Benchmark for Learning Stochastic PDEs
Yuantu Zhu ⋅ Zheyan Li ⋅ Dai Shi ⋅ Luke Thompson ⋅ Oliver Nash ⋅ Jose M Rangel ⋅ Siran Li ⋅ Bingguang Chen ⋅ Rongchan Zhu ⋅ Qi Meng ⋅ Hao Ni
Stochastic Partial Differential Equations (SPDEs) driven by random noise play a central role in modeling physical processes with rough spatio-temporal dynamics, such as turbulence flows, superconductors, and quantum dynamics. Although machine learning (ML)-based surrogate models have shown promise for efficiently approximating such dynamics, progress remains limited by the lack of a unified benchmark with controlled data generation and comprehensive evaluation. This gap is particularly significant for singular SPDEs, for which benchmark datasets are largely unavailable and reliable simulation requires numerically delicate schemes based on renormalization. Moreover, subtle differences in data-generation procedures, such as noise approximation, basis choice, and the inclusion of renormalization, can significantly affect the resulting datasets and, consequently, model evaluation. We introduce SPDEBench, the first unified benchmark for ML-based SPDE learning. SPDEBench provides ready-to-use datasets for physically and mathematically significant SPDEs on 1-3D domains with periodic or Dirichlet boundary condition. Both regular and singular SPDEs are taken into consideration. SPDEBench also incorporates representative ML baselines in operator learning, together with 7 evaluation metrics, including Sobolev and distributional metrics beyond the standard $L^2$-error. Supported by SPDEBench, we conduct systematic evaluations of model accuracy, robustness, and out-of-distribution generalization under controlled data variations. Our numerical results show that SPDE-aware architectures generally achieve stronger performance than generic operator-learning baselines. These findings establish SPDEBench as a reproducible and extensible resource, paving pathway for principled benchmarking and architecture design for stochastic spatio-temporal dynamics.
Show more
Get a GRIP, this will be a long TRIP: A Quantifiable Long-Range Framework for Verifying Over-squashing
Ferran Hernandez Caralt ⋅ Simon Heilig ⋅ Adrián Bazaga ⋅ Asja Fischer ⋅ Moshe Eliasof ⋅ Pietro Lió
Empirical claims about the connection between over-squashing and long-range interactions in GNNs, can only be trusted if the benchmarks used to validate them genuinely require long-range interactions. The de-facto standard, the Long Range Graph Benchmark, has been repeatedly shown to be saturated by tuned short-range models, with existing synthetic alternatives being tied to specific topologies. As such, there is a lack of principled certificate of long-rangedness on *arbitrary* graphs. This state reflects the absence of a precise characterization of long-ranged benchmarks. We address this fundamental gap by introducing four verifiable *axioms*: *Predictability*, *Tightness*, *Strictly $k$-Range*, and *Topology-Invariance*, that any task claiming to test $k$-hop interactions must satisfy. We formally prove that violating any one of them admits failure modes that undermine conclusions drawn from the task. Based on these axioms, we introduce TRIP (*Truly Ranged Interactions Problem*) and its generalisation GRIP (*Generally Ranged Interactions Problem*), constructive procedures that turn *any* graph into a provably long-ranged task by drawing features from *stable* distributions. Moreover, by construction, GRIP admits a closed-form, per-range Maximum-Likelihood oracle that yields the first *a priori per-range* lower bound on test error available on any benchmark. Using our framework, we: (i) audit 4 common long-range benchmarks and identify their failures modes with respect to our axioms; (ii) on TRIP-instantiated topologies, we find a popular notion of curvature is uncorrelated with GNN performance, supporting topological-vs-computational bottleneck distinction; and (iii) we show that a novel benchmark's over-squashing measures factors beyond pure long-rangedness. Code to use the framework and reproduce experiments is released anonymously https://github.com/ferranhernandezc/graph-grip.
Show more
RUBRIC-MME: Real-User Behavior-grounded Rubric for Multimodal Interaction Capability Evaluation
Jiajie Teng ⋅ Jianping Jiang ⋅ Huiyu Duan ⋅ Jingdong Chen ⋅ Yi Yuan ⋅ Guangming Yao ⋅ Sijing Wu ⋅ Yuqin Cao ⋅ Yixuan Gao ⋅ Xiongkuo Min ⋅ Guangtao Zhai
Multimodal large language models (MLLMs) increasingly serve as always-on assistants in continuous, multi-turn, goal-driven user interactions, yet existing benchmarks evaluate them through pre-authored capability probes, single-turn correctness, or aggregate leaderboard scores. As frontier models saturate such benchmarks, a widening gap emerges between leaderboard performance and real interaction quality, and current evaluation infrastructure can neither characterize nor diagnose this drift. We introduce **RUBUIC-MME**, a multimodal multi-turn interaction benchmark designed to close this gap along three axes: an *authenticity* axis, a *competence* axis, and a *diagnosability* axis. To close the authenticity axis, we propose *behavior-grounded benchmark construction*, a privacy safe pipeline that extracts distributional regularities from real deployment logs, including scenarios, persona-goal pairings, intent typologies, multi-turn structures, and uses them as priors to guide public video filtering and a *scenario–persona–intent–dialogue* generation chain, yielding a fully synthesizable benchmark for two interaction modes, streaming video and multi-image sequences. To close the competence axis, we define a multi-level structured rubric that evaluates models simultaneously at the turn level, including grounding, relevance, and helpfulness, and the session level, including intent-shift recovery, cross-turn consistency, and goal completion, sliced by capability, scenario, and modality. To close the diagnosability axis, RUBRIC-MME further embeds an automated analysis pipeline that clusters failure modes into per-model diagnostic reports with actionable improvement directions, forming an **evaluation–analysis–guidance** loop. Evaluating representative frontier and open-source MLLMs, we find that session-level competence diverges sharply from turn-level accuracy, and that there are also significant differences in performance across different scenarios and capabilities. We position RUBRIC-MME as a step toward benchmarks that function as evaluation infrastructure rather than ranking artifacts.
Show more
INVITA-WheatFieldState: A Real-World Benchmark for Crop-State Estimation in Wheat Field Trials
Heming Du ⋅ Ruihan Lu ⋅ Javier Fernandez ⋅ Zijian Wang ⋅ Yan Zhao ⋅ Scott C Chapman ⋅ Zi Huang ⋅ Xin Yu
Field trials use plot-level crop-state measurements to interpret how crops grow under real field conditions, but repeated measurements of canopy greenness, canopy amount, canopy closure, and growth stage require field visits, sensing campaigns, and post-processing. Modern wheat trials also collect weather records, trial metadata, field-camera images, UAV and satellite observations, proximal sensors, canopy products, and phenological surveys. Turning these records into a machine-learning benchmark is nontrivial because observations are sparse, asynchronous, unevenly collected, missing for structured reasons, and tied to measurement pipelines. We introduce INVITA-WheatFieldState, a benchmark derived from the INVITA wheat field-trial archive for estimating the crop state of a plot on a target date from observations available for that plot and date. Each plot-date example specifies a plot, date, and crop-state target, and is paired with available observations after temporal and source-provenance filtering. The derived dataset contains over 220k examples across NDVI, LAI, FCover, and Zadoks growth stage, covering canopy greenness, canopy amount, canopy closure, and phenological timing. LAI and FCover are released with provenance labels that distinguish product and proxy targets rather than pooled manual ground truth. The benchmark suite provides a plot-disjoint split matched to the plot-level experimental unit, validators, prediction schemas, regression representations, and prediction-level diagnostics. Results show that source, timing, metadata, and observation-availability structure are strong signals. Observation-set representations improve some all-example targets, while sensor-sequence, field-camera, and fusion representations remain target-dependent and coverage-bound. INVITA-WheatFieldState offers a concrete test case for ML methods that must learn from incomplete, asynchronous, and provenance-sensitive measurements used to study crop development in real fields.
Show more
ER-Reason: A Benchmark Dataset for LLM Clinical Reasoning in the Emergency Room
Nikita Mehandru ⋅ Niloufar Golchini ⋅ Namrata Garg ⋅ Kathy T LeSaint ⋅ Christopher J Nash ⋅ Anu Ramachandran ⋅ Travis Zack ⋅ Liam Greg McCoy ⋅ Adam Rodman ⋅ David Bamman ⋅ Melanie Molina ⋅ Ahmed Alaa
Existing benchmarks for evaluating the clinical reasoning capabilities of large language models (LLMs) often lack a clear definition of "clinical reasoning" as construct, fail to capture the full breadth of interdependent tasks within a clinical workflow, and rely on stylized vignettes rather than real-world clinical documentation. As a result, recent studies have found significant discrepancies between LLM performance on stylized benchmarks derived from medical licensing exams and their performance in real-world prospective studies. To address these limitations, we introduce ER-Reason, a benchmark designed to evaluate LLM reasoning as clinical evidence accumulates across decision-making tasks spanning the full workflow of emergency medicine. ER-Reason comprises 25,174 de-identified clinical notes from 3,437 patients, supporting evaluation across all stages of the emergency department workflow: triage intake, treatment selection, disposition~planning, and final diagnosis. Crucially, evaluation in ER-Reason extends beyond diagnostic accuracy to include stepwise Script Concordance Test (SCT)-style questions grounded in real patient cases, which assess whether LLMs update their diagnostic beliefs in the correct direction and magnitude as clinical evidence accumulates, scored against 2,555 emergency physician annotations. We evaluate reasoning and non-reasoning LLMs on ER-Reason, and show that our tasks provide a more nuanced view of how LLM reasoning fails on real patient cases than existing benchmarks allow.
Show more
Colosseum: Auditing Collusion in Cooperative Multi-Agent Systems
Mason Nakamura ⋅ Abhinav Kumar ⋅ Saswat Das ⋅ Sahar Abdelnabi ⋅ Saaduddin Mahmud ⋅ Ferdinando Fioretto ⋅ Shlomo Zilberstein ⋅ Eugene Bagdasarian
Multi-agent systems, where LLM agents communicate through free-form language, enable sophisticated coordination for solving complex cooperative tasks. This surfaces a unique safety problem when a group of agents forms a coalition and colludes to pursue secondary goals and degrade the joint objective. In this paper, we present Colosseum, a framework for auditing LLM agents' collusive behavior in multi-agent settings. We ground how agents cooperate through a formal multi-agent decision-making framework and measure action-based collusive behavior in actions via regret relative to the cooperative optimum and compare it with communication-based collusive behavior. Colosseum enables audits of LLM agents for collusion under benign settings, different coalition objectives, persuasion tactics, and network topologies. We then introduce a new behavioral probe by creating secret communication channels between agents, showing that most out-of-the-box models exhibit a propensity to collude under this probe, which we term emergent collusion. Furthermore, we discover ``collusion on paper'' when agents plan to collude in text but often pick non-collusive actions. Colosseum provides a new way to audit collusion in cooperative multi-agent systems while presenting observations about how collusion emerges, what affects collusion efficacy, and which strategies may mitigate it.
Show more
SC$^3$: A Multi-Solvent Solubility Challenge and Benchmark
Vansh Ramani ⋅ Tarak Karmakar ⋅ Sayan Ranu ⋅ Dhairya Kuchhal ⋅ Har Ashish Arora ⋅ Sergei V Tatarin ⋅ Lev Krasnov
Solubility prediction is a standard benchmark in computational chemistry, yet multi-solvent models which reportedly approach the experimental-noise ceiling (i.e. the aleatoric limit) are not yet reliable enough to be deployed. We argue that this gap is partly artefactual: published benchmarks differ in curation policies, evaluate on count-weighted RMSE that hides failure on tail-heavy solvent distributions, and treat the widely cited 0.6–0.8 log S inter-laboratory figure as the aleatoric ceiling even though it reflects worst-case, not expected, disagreement. We introduce SC3, a multi-solvent solubility benchmark built on BIGSOLDB v2.1 with three contributions: (i) a reproducible curation pipeline yielding 101,535 measurements over 1,327 solutes and 206 solvents, with a recalibrated aleatoric floor of 0.106 log S—roughly 6× tighter than the conventional figure; (ii) nested Gold/Silver/Bronze consensus tiers with per-point σ\sigma σ, three leakage-checked splits, and a multi-solvent metric suite (PS-RMSE, Z-RMSE); and (iii) a 31-model benchmark across six families, whose best Bronze PS-RMSE sits at ≈5×εaleatoric\approx 5 \times \varepsilon_{\text{aleatoric}} ≈5×εaleatoric, and we observe this is a gap unclosed by any deep alternative tested. We perform three follow-on analyses: data scaling, transfer from quantum-chemistry solvation energies, and feature-level attribution, which demonstrates that calibrated per-point uncertainty is a reusable infrastructure for diagnosis beyond point prediction.
Show more
Castle-in-the-Air: Probing the Foundational Visual Deficits of MLLMs via Bottom-Up Cognitive Factors
Jen-Tse Huang ⋅ Dasen Dai ⋅ Jen-Yuan Huang ⋅ Youliang Yuan ⋅ Xiaoyuan Liu ⋅ Wenxuan Wang ⋅ Wenxiang Jiao ⋅ Pinjia He ⋅ Zhaopeng Tu ⋅ Haodong Duan
Humans develop perception through a bottom-up hierarchy: from basic primitives and Gestalt principles to high-level semantics. Current Multimodal Large Language Models (MLLMs) are trained directly on complex downstream tasks. Does it guarantee their performance on these foundational visual capabilities? To systematically investigate this gap, we introduce VisFactor, a benchmark that digitizes 20 vision-centric subtests from FRCT, a well-established cognitive psychology assessment spanning four domains of human visual cognition. Furthermore, we design algorithms to automatically construct and validate unlimited test cases with controllable difficulty. Using VisFactor, we evaluate 39 frontier MLLMs, including both proprietary (e.g., GPT, Gemini) and open-source (e.g., LLaMA, Qwen) models. The best model achieves a score of only 55.9%. Analysis reveals good internal consistency (Cronbach's alpha = 0.94) and construct validity (compared to existing vision benchmarks). Models consistently fail on tasks such as mental rotation, spatial relation inference, and figure–ground discrimination, regardless of model size or prompting strategy. These findings suggest that performance improvements on existing general benchmarks might represent castles in the air instead of a genuine mastery of human-like visual cognition.
Show more
MinerU2.5-Pro: Pushing the Limit of Document Parsing via a Calibrated Evaluation-Data Flywheel
Bin Wang ⋅ Tianyao He ⋅ Linke Ouyang ⋅ Fan Wu ⋅ Zhiyuan Zhao ⋅ Tao Chu ⋅ Yuan Qu ⋅ Zhenjiang Jin ⋅ Weijun Zeng ⋅ Ziyang Miao ⋅ Bangrui Xu ⋅ Junbo Niu ⋅ Wei Li ⋅ Mengzhang Cai ⋅ Qiu Jiantao ⋅ Qintong Zhang ⋅ Dongsheng Ma ⋅ Yuefeng Sun ⋅ Hejun Dong ⋅ Wenzheng Zhang ⋅ Jutao Xiao ⋅ shi J yong ⋅ Pengyu Liao ⋅ Xiaomeng Zhao ⋅ Huaping Zhong ⋅ Zhengren Wang ⋅ Liqun Wei ⋅ Jing Yu ⋅ Jie Yang ⋅ Shasha Wang ⋅ Qianqian Wu ⋅ Xuanhe Zhou ⋅ Weijia Li ⋅ Zhenxiang Li ⋅ Zhongying Tu ⋅ Jiang Wu ⋅ Lijun Wu ⋅ Chao Xu ⋅ Kai Chen ⋅ Wentao Zhang ⋅ Yu Qiao ⋅ Bowen Zhou ⋅ Dahua Lin ⋅ Conghui He
Document parsing is now critical infrastructure for LLM systems, yet standard benchmarks suggest near saturation: on OmniDocBench v1.5, leading methods cluster around a 95-point ceiling. We argue that this reflects a stalled evaluation–training flywheel, not solved parsing: v1.5 under-samples hard, long-tail documents and uses fixed-granularity matching that can penalize semantically correct outputs. We re-engage the flywheel with two co-designed protocols. OmniDocBench v1.6 repairs evaluation with MGAM (Multi-Granularity Adaptive Matching) and a strictly held-out 296-page Hard Subset, with Base / Hard / Full reporting. The corrected benchmark exposes two data gaps—hard cases are under-represented and unreliably annotated—which the MinerU2.5-Pro Data Engine addresses through DDAS (Difficulty-Driven Adaptive Sampling) and Judge-and-Refine, a render–verify–refine annotation pipeline, yielding 65.5M auto-labeled and 192K human-annotated samples. Conversely, the same discovery and annotation mechanisms provide a scalable path for future Hard-subset refreshes. Under a fixed 1.2B MinerU2.5 backbone, these interventions raise OmniDocBench v1.6 Full performance from 92.98 to 95.69, with the clearest separation on the corrected Hard split. The result reframes apparent saturation as a repairable benchmark–data loop: document parsing progress must be easier to measure correctly, not just easier to claim.
Show more
Verifier Choice is a Benchmark Design Variable: Auditing Structural Counting Evaluation in Text-to-Image Models
Shurun Li ⋅ Michael H Wang ⋅ Haibo Zeng ⋅ Long Wang
Automatic evaluation is now central to text-to-image benchmarking, but the verifier that converts generated images into scores is often treated as an implementation detail. We argue that this is unsafe: an automatic benchmark score is a property of the image generator, prompt distribution, verifier, verifier-use protocol, and scoring rule, not of the generator alone. We study this issue in structural counting, a controlled setting where human-visible counts can be audited and where both instance-level object counts and parent-bound substructure counts arise naturally. We construct a verifier-centric audit suite over seven counting objects: cell phones, chairs, and traffic lights for object counting, and clover leaflets, dresser drawer fronts, power-strip sockets, and turbine blades for substructure counting. Across 6,240 generated images from Qwen-Image-2512 and FLUX.2-dev, we compare open MLLM verifiers under count-only, reject-aware, zero-shot and few-shot protocols, plus TIFA-like target-query variants and targeted frontier closed-model sanity checks on two hard objects. We also compare MLLMs with a detector baseline. On a balanced 2,280-image evaluation split, Qwen3.5-27B with few-shot reject/count achieves the best image-level agreement among open verifiers ($\kappa=0.754$) with a small aggregate score gap ($+1.54\text{pp}$), but changing the protocol, model size, or generator distribution can shift automatic scores by tens of points. Targeted GPT-5.5 and Claude Opus 4.7 checks do not remove the failure modes: GPT-5.5 still has only 59.92\% clover valid-count fidelity, while Claude Opus 4.7 has lower same-slice agreement ($\kappa=0.432$). We propose reporting a factorized verifier reliability profile---aggregate score gap, image-level agreement, valid-image count fidelity, rejection behavior, output compliance, and directional bias---rather than a single automatic benchmark score.
Show more
Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios
Linus Folkerts ⋅ Will Payne ⋅ Simon Inman ⋅ Philippos M Giavridis ⋅ Joe Skinner ⋅ Sam Deverett ⋅ James Aung ⋅ Ekin Zorer ⋅ Michael Schmatz ⋅ Mahmoud Ghanem ⋅ John Wilkinson ⋅ Alan Steer ⋅ The Vy Hong ⋅ Harry Coppock ⋅ Jessica Wang
We evaluate the autonomous offensive cyber capabilities of frontier AI agents on two purpose-built cyber ranges---a 32-step corporate network attack and a 7-step industrial control system attack targeting a simulated cooling tower---complemented by a private 82-task CTF suite spanning four difficulty tiers to measure narrow cyber skills. Estimated human-expert solve times are roughly 20 hours for the corporate range, 22 hours for the cooling tower, and from 1 minute to 40 hours across the CTF suite. Across twelve models released over a twenty-month period---from GPT-4o in August 2024 to early predeployment checkpoints of Mythos and GPT-5.5---we observe two trends. First, performance increases approximately log-linearly with inference-time compute, with no plateau observed for the strongest models up to 100M tokens. Second, newer frontier models generally complete more steps at fixed token budgets. GPT-5.5 and the Mythos Preview checkpoint each completed all 32 steps in at least one run, the first end-to-end completions of the corporate range; GPT-5.5 averaged 22.3 of 32 steps across ten 100M-token runs. On the control system range, the latest models reached 4 of 7 steps, primarily by directly probing the operational-technology protocol rather than following the intended chain of IT compromise and authenticated control. These results indicate rapid improvements in autonomous cyber capability and suggest that complex but undefended multi-step ranges are no longer reliably distinguishing the strongest frontier models. Future evaluations should emphasise hardened environments, active defences, defensive-control effectiveness, and operational security---priorities that grow more pressing as the threat of AI-driven cyber attacks becomes increasingly realistic.
Show more
Don't Pause! Every prediction matters in a streaming video
Dibyadip Chatterjee ⋅ Zhanzhong Pang ⋅ Fadime Sener ⋅ Yale Song ⋅ Angela Yao
Streaming video models should respond the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming outputs untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming outputs by their temporal precision and balanced coverage across the entire video. Our benchmark reveals that during streaming inference: (i) MLLMs detect events reliably in a stream but spam predictions unprompted; (ii) post-training MLLMs for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate Ctrl-SPOT, a training-free streaming controller for MLLMs, that retains their event perception while controlling their streaming outputs for balanced coverage. This establishes a strong baseline on SPOT-Bench.
Show more
CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
Taein Lim ⋅ Seongyong Ju ⋅ Munhyeok Kim ⋅ Hyunjun Kim ⋅ Hoki Kim
Large language models (LLMs) are increasingly deployed as autonomous agents in offensive cybersecurity. In this paper, we reveal an interesting phenomenon: different agents exhibit distinct attack patterns. Specifically, each agent exhibits an attack-selection bias, disproportionately concentrating its efforts on a narrow subset of attack families regardless of prompt variations. To systematically quantify this behavior, we introduce CyBiasBench, a comprehensive 630-session benchmark that evaluates five agents on three targets and four prompt conditions with ten attack families. We identify explicit bias across agents, with different dominant attack families and varying entropy levels in their attack-family allocation distributions. Such bias is better characterized as a trait of the agents, rather than a factor associated with the attack success rate. Furthermore, our experiments reveal a bias momentum effect, where agents resist explicit steering toward attack families that conflict with their bias. This forced distribution shift does not yield measurable improvements in attack performance. To ensure reproducibility and facilitate future research, we release an interactive result dashboard at \url{https://cy-bias-bench.vercel.app/} and a reproducibility artifact with aggregated session-level statistics and full evaluation scripts at \url{https://anonymous.4open.science/r/CyBiasBench-2026/}.
Show more
MAEB: Massive Audio Embedding Benchmark
Adnan E Assadi ⋅ Isaac Chung ⋅ Chenghao Xiao ⋅ Roman Solomatin ⋅ Animesh Jha ⋅ Rahul Chand ⋅ Silky Singh ⋅ Kaitlyn Wang ⋅ Ali Sartaz Khan ⋅ Marc M Nasser ⋅ Sufen Fong ⋅ Pengfei He ⋅ Alan Xiao ⋅ Ayush Sunil Munot ⋅ Aditya Shrivastava ⋅ Artem Gazizov ⋅ Niklas Muennighoff ⋅ Kenneth Enevoldsen
We introduce the Massive Audio Embedding Benchmark (MAEB), a large-scale benchmark covering 30 tasks across speech, music, environmental sounds, and cross-modal audio-text reasoning in 100+ languages. We evaluate 56 models and find that no single model dominates across all tasks: contrastive audio-text models excel at environmental sound classification (e.g., ESC50) but score near random on multilingual speech tasks (e.g., SIB-FLEURS), while speech-pretrained models show the opposite pattern. Clustering remains challenging for all models, with even the best-performing model achieving only modest results. We observe that models excelling on acoustic understanding often perform poorly on linguistic tasks, and vice versa. We also release MAEB+, the full 98-task collection from which MAEB is curated; to our knowledge, it is the largest publicly released collection of audio embedding evaluation tasks. MAEB preserves task diversity while reducing evaluation cost and integrates into the MTEB ecosystem for unified evaluation across text, image, and audio modalities. We release MAEB and all 98 tasks along with code and a leaderboard at https://anonymous.4open.science/r/mteb-2FEB/.
Show more
SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation
Ivan Molodetskikh ⋅ Kirill Malyshev ⋅ Mark Mirgaleev ⋅ Nikita Zagainov ⋅ Evgeney Bogatyrev ⋅ Dmitriy Vatolin
Modern image super-resolution methods generate detailed, visually appealing results, but they often introduce visual artifacts: unnatural patterns and texture distortions that degrade perceived quality. These defects vary widely in perceptual impact—some are barely noticeable, while others are highly disturbing—yet existing detection methods treat them equally. We propose artifact prominence as an evaluative target, defined as the fraction of viewers who judge a highlighted region to contain a noticeable artifact. We design a crowdsourced annotation protocol and construct SR-Prominence, a dataset suite containing 3,935 artifact masks from DeSRA, Open Images, Urban100, and a realistic no-ground-truth Urban100-HR setting, annotated with prominence. Re-annotating DeSRA reveals that 48.2% of its in-lab binary artifacts are not noticed by a majority of viewers. Across the suite, we audit SR artifact detectors, image-quality metrics, and SR methods. We find that classical full-reference metrics, especially SSIM and DISTS, provide surprisingly strong localized prominence signals, whereas no-reference IQA methods and specialized artifact detectors often fail to generalize across datasets and reference settings. SR-Prominence is released with an objective scoring protocol that allows new metrics to be benchmarked on our suite without further crowdsourcing. Together, the data and protocols enable SR artifact evaluation to move from binary defect presence toward perceptual impact.
Show more
GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation
Zihe Wei ⋅ Sheng Xiang ⋅ Ying Zhang ⋅ changjun jiang
Graph-structured data underpins applications from citation analysis and social-network modeling to molecular design and knowledge-graph construction, and Large Language Models (LLMs) are increasingly used as prompt-driven graph synthesizers. Classical graph-generation reviews catalog deep generative models and their evaluation primitives, but predate the LLM era and provide no foundation for evaluating instruction-following graph synthesis. Recent LLM-era benchmarks evaluate models along graph-type or task-domain axes; such organizations, however, average over structural complexity and cannot localize \emph{where} in the complexity spectrum an LLM breaks down. To close this diagnostic gap, we introduce GraphInstruct, a progressive-complexity benchmark that stratifies LLM graph generation into six complexity levels and five evaluation dimensions, paired with 800 hand-authored instructions, 1,582 algorithmically synthesized reference solutions, and a 12-LLM capability evaluation across 45 (model, strategy) configurations. We find that discriminative power peaks at multi-constraint composition rather than reasoning depth, that no single prompting strategy dominates across levels or model families, and that domain-semantic constraints remain iteration-invariant under all tested methods---pointing to retrieval rather than additional compute as the next research frontier. Atop the benchmark, a verification-guided iterative framework with constraint-aware adaptive prompting consistently surpasses the prompt-engineering ceiling on tested target models, demonstrating that the benchmark's fine-grained signals drive method development. Data, code, and reproducibility artifacts are included in the supplementary materials and will be released publicly upon acceptance.
Show more
SurvCancel: A Longitudinal Dataset and Benchmark for Dynamic Order Cancellation Prediction in On-Demand Ride-Sharing Systems
Huayang Liu ⋅ Mingyu Zheng ⋅ Wei Zhang ⋅ Yijun Bian ⋅ Kai Wang
Passenger-order cancellation is a critical source of uncertainty in on-demand ride-sharing systems, where requests, assignments, and vehicle routes evolve continuously in real time. Due to shared capacity and coupled routes, a single cancellation may propagate through downstream matching and routing decisions, degrading system efficiency and service reliability. However, cancellation prediction in on-demand ride-sharing remains difficult to study systematically due to the lack of large-scale longitudinal datasets, standardized evaluation protocols, and fine-grained real-time benchmarks. In this work, we introduce SurvCancel, a real-world longitudinal dataset and benchmark for dynamic passenger-order cancellation prediction in on-demand ride-sharing systems. Constructed from operational logs collected across four service regions in the real world, SurvCancel contains 173K released orders and 7.9M time-dependent snapshots. Each order is represented as a longitudinal sequence of system states, capturing both order-level attributes and the evolving supply-demand environment throughout its lifecycle. We formulate cancellation prediction as a stage-conditioned dynamic survival task: at each landmark time, models estimate short-term cancellation risk from the historical state sequence observed so far. We evaluate representative survival and dynamic-risk models under both in-distribution and distribution-shift protocols. Our results show stage-dependent model behavior in the in-distribution setting and reveal distinct robustness patterns under regional and temporal distribution shifts.
Show more
RTCBench: Evaluating Tool Use of Large Language Models Beyond Oracle Access
Qi Wei ⋅ Bo An
Large language models (LLMs) have demonstrated promising capabilities in tool calling. However, existing benchmarks predominantly evaluate models in an idealized setting where ground-truth tool definitions are provided in context. This assumption diverges significantly from real-world deployment scenarios, where the model must first identify user intent and retrieve relevant tools from a large, heterogeneous tool repository before invoking them. In this work, we introduce a Retrieval-augmented Tool-calling Benchmark (RTCBench). In RTCBench, we collect 1,840 tasks from existing benchmarks and augment them with two tool environments: 1) None: no available tools provided in the prompt and 2) Distractor: strong distractor tools, while the target tools are excluded. By removing oracle tool access, RTCBench compels LLMs to actively retrieve relevant tools from a large-scale repository via a unified search tool before making tool calls, thereby faithfully simulating the end-to-end complexity of real-world tool-use pipelines. Through comprehensive evaluation of diverse LLMs ranging from 4B-parameter models to proprietary frontier models, we identify two core deficiencies: unreliable tool discovery, where models fail to initiate effective retrieval actions, and post-retrieval grounding failures, where models cannot reliably select and invoke the correct tools from semantically similar retrieved candidates. Furthermore, to mitigate these issues, we release 13,278 retrieval-augmented tool-calling expert trajectories with high-quality chain-of-thought reasoning for LLM post-training. Experiments show that a 4B-parameter model post-trained with GRPO improves its tool-calling accuracy on RTCBench from approximately 47.9\% to over 85.8\%, approaching the performance of the state-of-the-art model Gemini-3.1-Pro.
Show more
How Complete Should a Reference Be? A Benchmark Audit for Fluorescence Spot Detection
Jiyeong Kong ⋅ Pan-Gyun Jeong ⋅ Kyung-Tae Kang
Fluorescence spot-detection benchmarks in biology and medicine often rely on reference annotations that are incomplete, ambiguous, or inconsistently defined. In such settings, detector performance is not determined solely by model predictions; it can also depend on which faint or ambiguous signals are counted as reference positives. We formulate this issue as a reference-audit problem: instead of treating the evaluation reference as fixed ground truth, we test whether benchmark outcomes remain stable under plausible changes in reference definition and annotation completeness. Using the public FISH_spots dataset, a benchmark for fluorescence in situ hybridization (FISH) spot detection, we construct Raw, Lenient, and Strict audit references and evaluate diverse classical and learning-based detectors under a unified point-matching protocol. We further apply annotation-retention stress tests that simulate random and structured missing-label mechanisms. Across these audits, changing only the evaluation reference can alter measured F1 scores and model rankings, including the selected top-1 detector, even when aggregate rank correlation remains high. These findings suggest that model selection in fluorescence spot-detection benchmarks should be tested across plausible reference definitions, rather than based on a single reference annotation set.
Show more
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
Jiho Jin ⋅ Junho Myung ⋅ Juhyun Oh ⋅ Junyeong Park ⋅ Rifki A Putri ⋅ Sunipa Dev ⋅ Vinodkumar Prabhakaran ⋅ Alice Oh
As large language models (LLMs) are increasingly deployed to users around the world, they are integrated into everyday tasks across diverse cultural contexts, from drafting personal communications to brainstorming creative ideas. These tasks are inherently cultural: they require contextual appropriateness, symbolic resonance, and tacit cultural expectations that native speakers draw on instinctively, meaning that a response can be factually plausible yet unmistakably wrong to a local reader. Existing cultural benchmarks have treated culture as a flat set of facts via fact verification or norm entailment methods, and have adopted LLM-as-a-Judge without examining whether they can capture such thick cultural errors. To address this gap, we present JuICE (Benchmark for LLM-Judge in Identifying Cultural Errors), a multilingual dataset of 7,470 span-level annotations of cultural and linguistic errors, collected from native speakers in long-form LLM responses. It covers 1,050 query-response pairs from four countries (United States, South Korea, Indonesia, and Bangladesh), in both English and their countries' main languages. Using JuICE, we find that even the strongest LLM-judge achieves only an F1 of 0.52 against human-annotated errors, and that this gap is systematic. LLM-judges reliably detect surface-level linguistic and factual mistakes but consistently miss thick cultural errors that local residents readily identify. Our findings suggest that robust cultural evaluation must move beyond surface-level detection toward frameworks that account for the depth and situatedness of cultural meaning.
Show more
MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
Honglin Lin ⋅ Zheng Liu ⋅ Yun Zhu ⋅ Chonghan Qin ⋅ Juekai Lin ⋅ Xiaoran Shang ⋅ Conghui He ⋅ Bin CUI ⋅ Wentao Zhang ⋅ Lijun Wu
Recent advances in Vision Language Models (VLMs) have driven significant progress in visual reasoning. However, open-source multimodal models still lag behind proprietary systems, largely due to the lack of high-quality reasoning data. Existing datasets offer limited coverage of challenging domains such as STEM diagrams and visual puzzles, and lack consistent, long-form Chain-of-Thought (CoT) annotations essential for eliciting strong reasoning capabilities. To bridge this gap, we introduce MMFineReason, a large-scale multimodal reasoning dataset comprising 1.8M samples and 5.1B solution tokens, featuring high-quality reasoning annotations distilled from Qwen3-VL-235B-A22B-Thinking. The dataset is established via a systematic three-stage pipeline: (1) large-scale data collection and standardization, (2) CoT rationale generation, and (3) comprehensive selection based on reasoning quality and difficulty awareness. The resulting dataset spans STEM, visual puzzles, games, and complex diagrams, with each sample annotated with detailed, visually grounded reasoning traces. We fine-tune Qwen3-VL-Instruct on MMFineReason to develop MMFineReason-2B/4B/8B versions. Our models establish new state-of-the-art (SOTA) results for their size class. Notably, MMFineReason-4B surpasses Qwen3-VL-8B-Thinking, and MMFineReason-8B even outperforms Qwen3-VL-30B-A3B-Thinking while approaching Qwen3-VL-32B-Thinking, demonstrating remarkable parameter efficiency. Crucially, we uncover a "less is more" phenomenon via our difficulty-aware filtering strategy: a subset of just 7\% (123K samples) achieves performance comparable to the full dataset. Notably, we reveal a synergistic effect where reasoning-oriented data simultaneously boosts general capabilities.
Show more
SudoBench: A Contextual Authorization Benchmark for LLM Agents
Vincent Siu ⋅ Tianneng Shi ⋅ Shangding Gu ⋅ Zhun Wang ⋅ Dawn Song ⋅ Chenguang Wang
Secure LLM agents should take actions for authorized users in authorized environments. However, current agents' (e.g., OpenClaw) runtimes expose neither user nor environment authorization and existing agent security benchmarks similarly evaluate models without relevant authorization context. We propose SudoBench, a benchmark of 135 paired scenarios in a synthetic consulting universe with a static access control list, spanning user authorization (confused deputy), environment authorization (source-label indirect prompt injection), and both (scope-check indirect prompt injection). For each pair of scenarios, the authorization context is flipped such that an identical prompt should be appropriately complied with or refused. We score each pair as a joint pass requiring the model to produce the correct outcome on both paired scenarios. Across ten frontier closed- and open-weight LLMs, joint pass rate without authorization context sits below 16\% on every model. Adding user and environment authorization raises joint pass to as high as 67--73\%. Even with this authorization context, frontier models remain far from reliably handling contextual authorization, framing authorization context as an important research agenda for agent security.
Show more
FraudBench: A Legal Evaluation of AI Deception on Realistic Tasks
Kevin Wei ⋅ Sumaya N Adan ⋅ Stephan Llerena ⋅ Mark L Gitau ⋅ Jakob Merane ⋅ Anita Srinivasan ⋅ Colette Le Brannan ⋅ Keethan R. Kleiner ⋅ Alessandro Tacconelli ⋅ Michael T Tiu ⋅ Case Thomason ⋅ Kyle Emili ⋅ Natasza Gadowska ⋅ Alex Mark ⋅ Eric Ren ⋅ Umang Bhatt ⋅ Jatinder Singh ⋅ Jacy Anthis ⋅ Doni Bloomfield ⋅ Noam Kolt
We introduce FraudBench, an evaluation of the propensity of large language models (LLMs) to recommend, assist, or directly commit legal fraud, as measured by a novel dataset of 159 scenarios based on real-world U.S. court cases and validated by legal experts. As LLM use rises, models' ability to comply with the law becomes more important to ensure safety and public benefit. However, evaluating models' propensity to engage in fraud is challenging due to the need for specialized expertise, the contextual nature of legal standards, and the lack of high-quality evaluation data. \textsc{FraudBench} fills this gap with an evaluation framework for measuring fraud propensity, a dataset of 477 tasks (3 per scenario), and large-scale empirical results on 20 LLMs and over 300,000 samples. We also present a pre-registered human baseline ($n_{\texttt{annotations}} = 2957$, $n_{\texttt{humans}} = 996$). We find that (1) LLM fraud propensities are highly individualized, ranging from near 0\% to over 40\%; (2) LLMs with reasoning enabled behave less fraudulently ($\beta = -0.17$, log-odds scale); (3) LLM fraud propensity increases with task autonomy (6.4\% overall fraud rate when advising users, 26.9\% when acting directly); (4) LLMs are more likely to conceal than actively misrepresent information ($\beta = 0.38$); (5) humans and LLMs responded similarly to contextual cues, and differences in human-LLM fraud rates were not statistically significant; (6) system prompts can have substantial effects on fraud propensity across LLMs.
Show more
CyberDualEval: Measuring Dual-Use Cyber Risks in Frontier Language Models
Magnus Saebo ⋅ Francesco Piccoli ⋅ Maxwell Watson ⋅ Pranjali Thakur ⋅ Yangruibo Ding ⋅ ZHUO ZHANG
Frontier language-model agents are rapidly becoming capable cybersecurity assistants, creating a dual-use cyber risk: the same capabilities that help defenders find, understand, and validate vulnerabilities can also help attackers weaponize them. Existing cyber benchmarks largely measure either raw offensive capability or generic refusal behavior, but they do not evaluate whether models can preserve benign security utility while refusing assistance that materially advances misuse. We introduce CyberDualEval, a benchmark for measuring this boundary on real vulnerabilities. Each task is decomposed into three phases of increasing operationalization: vulnerability analysis, proof-of-concept generation, and exploit execution. To capture the contested middle ground, we annotate each proof-of-concept task with a Task Weaponization Score that measures how much work remains to turn a minimally sufficient demonstration into unauthorized cyber misuse. CyberDualEval evaluates frontier language models with varied agentic scaffolds and reports refusal, defensive analysis accuracy, and optional exploit validation as distinct signals. Our evaluation reveals two complementary failure modes. Many models over-comply, assisting with high-weaponizability PoCs or exploit requests that should be refused. Conversely, recent safety-tuned models often over-refuse, declining low-risk or defensive requests and thereby reducing benign cyber utility. These results show that cyber safety cannot be assessed by capability or refusal rate alone: safe deployment requires calibrated refusal that tracks weaponizability while preserving useful defensive assistance.
Show more
RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation
Chengzhi Shen ⋅ Weixiang Shen ⋅ Tobias Susetzky ⋅ Chen Chen ⋅ Jun Li ⋅ Yuyuan Liu ⋅ zhang xuepeng ⋅ Zhenyu Gong ⋅ Daniel Rueckert ⋅ Jiazhen Pan
Intensive care units (ICU) generate dense, evolving streams of clinical information, where physicians must repeatedly reassess patient states under time pressure, underscoring a clear need for reliable AI decision support. Existing ICU benchmarks typically treat historical clinician actions as ground truth. However, these actions are made under incomplete information and limited temporal context of the underlying patient state, and may therefore be suboptimal, making it difficult to assess the true reasoning capabilities of AI systems. We introduce RealICU, a hindsight-annotated benchmark for evaluating large language models (LLMs) under realistic ICU conditions, where labels are created after senior physicians review the full patient trajectory. We formulate four physician-motivated tasks: assess \emph{Patient Status}, \emph{Acute Problems}, \emph{Recommended Actions}, and \emph{Red Flag} actions that risk unsafe outcomes. We partition each trajectory with 30-min windows and release two datasets: RealICU-Gold with 930-window annotations from 94 MIMIC-IV patients, and RealICU-Scale with 11,862 windows extended by \emph{Oracle}, a physician-validated LLM hindsight labeler. Existing LLMs including memory-augmented ones performed poorly on RealICU, exposing two failure modes: a recall–safety tradeoff for clinical recommendations, and an anchoring bias to early interpretations of the patient. We further introduce ICU-Evo to study structured-memory agents that improves long-horizon reasoning but does not eliminate safety failures. Together, RealICU provides a clinically grounded testbed for measuring and improving AI sequential decision-support in high-stakes care.
Show more
CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
Lukas Thede ⋅ Stefan Winzeck ⋅ Zeynep Akata ⋅ Jonathan Richard Schwarz
Large language model (LLM) post-training enhances latent skills, unlocks value alignment, improves performance, and enables domain adaptation. Unfortunately, post-training is known to induce forgetting, especially in the ubiquitous use-case of leveraging third-party pre-trained models, which is typically understood as a loss of parametric or factual knowledge. We argue that this accuracy-centric view is insufficient for modern foundation models and instead define forgetting as systematic model drift that degrades behavior and user experience. In this context, we introduce __CapTrack__, a capability-centric framework for analyzing forgetting in LLMs that combines a behavioral taxonomy with an evaluation suite centered on capability-specific metrics. Using CapTrack, we conduct a large-scale empirical study across post-training algorithms, domains, and model families, including models up to 80B parameters. We find that forgetting extends beyond parametric knowledge, with pronounced drift in robustness and default behaviors. Instruction fine-tuning induces the strongest relative drift, while preference optimization is more conservative and can partially recover lost capabilities. Differences across model families persist, and no universal mitigation emerges.
Show more
What Survival Benchmarks Don’t Tell You: Impact of Model Selection and Dataset Regimes
Sandra Dening ⋅ Emre Kavak ⋅ Christian Wachinger ⋅ George H Chen
Despite the proliferation of survival analysis methods, current benchmarks often operate under narrow conditions: they heavily rely on the concordance index (C-index) for evaluation and model selection, and frequently exclude deep learning (DL) or underrepresent challenging dataset regimes. To address this, we conduct a large-scale, neutral benchmark comparing 10 classical, tree-based, and DL methods across 49 diverse datasets. Crucially, we treat the model selection criterion -- optimizing for C-index, Integrated Brier Score (IBS), or Mean Absolute Error (MAE-PO) -- as an independent experimental variable. Using Bayesian mixed-effects modeling, we demonstrate that evaluation design heavily dictates conclusions. For complex DL and tree-based architectures, shifting hyperparameter selection from the C-index to IBS unlocks massive gains in probabilistic accuracy and calibration at a negligible cost to discriminative ranking. Furthermore, method success is fundamentally governed by dataset characteristics. While DL models scale exceptionally well with larger sample sizes, they suffer notable performance drops in high-dimensional feature spaces. Additionally, we expose a structural vulnerability where heavy censoring artificially inflates absolute calibration metrics, creating overly conservative tests. Ultimately, our findings challenge the utility of global single-number performance summaries and advocate for regime-conditioned reporting. We further demonstrate that aligning the model selection strategy with the target application is not merely an implementation detail, but a critical requirement for realizing the true probabilistic and temporal accuracy of complex survival architectures. We make our benchmark publicly available upon publication.
Show more
AptaBench: A Benchmark for Aptamer-Small Molecule Binding
Mariia Eremeyeva ⋅ Nikita Serov
Aptamers are programmable DNA and RNA receptors with broad potential in diagnostics, therapeutics, biosensing, and molecular monitoring. Although they offer chemically synthesizable alternatives to protein-based binders in several applications, computational aptamer discovery remains much less mature, especially for small-molecule targets. This gap reflects three persistent evaluation failures: fragmented and inconsistent affinity annotations, unreliable negatives derived from untested rather than experimentally validated pairs, and random splits that leak related aptamers or recurring ligand identities across train and test sets. As a result, current benchmarks make it difficult to distinguish transferable aptamer-ligand recognition from dataset-specific shortcuts. We present AptaBench, the first curated, standardized, and leakage-aware benchmark for aptamer-small-molecule binding prediction with experimentally reported active and inactive measurements and paired classification and affinity-regression tasks. AptaBench integrates eight curated sources into 6,289 interaction pairs covering 1,610 DNA/RNA aptamers and 942 ligands. We standardize sequences, resolve ligands to canonical molecular representations, harmonize duplicate and conflicting records, and convert dissociation constants to p$K_d$. More than 30% of entries contain quantitative affinity values, while inactive labels are taken only from explicitly reported non-binding or low-affinity measurements rather than synthetic cross-pairing. AptaBench provides fixed in-distribution, molecule-disjoint, and aptamer-disjoint protocols reflecting practical discovery scenarios: prediction for unseen targets and evaluation of new candidate sequences. Across descriptor-based models, pretrained sequence and molecular encoders, and sequence-ligand fusion architectures, in-distribution evaluation substantially overestimates generalization. The best classifier reaches ROC-AUC 0.95 in-distribution, but drops to 0.87 for unseen molecules and 0.86 for unseen aptamers. Affinity prediction is more sensitive, with $R^2$ decreasing from 0.65 to approximately 0.30 under disjoint protocols. These results show that reliable progress in computational aptamer discovery requires broader chemical coverage, experimentally grounded inactive labels, and standardized quantitative annotations. By releasing curated data, fixed leakage-aware splits, reproducible baselines, and preprocessing code, AptaBench is poised to expose the generalization limits of existing models, challenge future sequence-molecule recognition methods, and provide a rigorous foundation for practice-oriented aptamer evaluation.
Show more
FireMPC: A Multi-Source Pan-Canadian Wildfire Benchmark Revealing Spatiotemporal Generalization Gaps
Zhengsen Xu ⋅ Sibo Cheng ⋅ Lanying Wang ⋅ Aryan Sharma ⋅ Linlin Xu
Under Climate Change, wildfire activity is intensifying worldwide, with Canada experiencing increasingly severe events that now affect every province and territory. However, existing machine-learning (ML) datasets remain constrained by geographic bias, narrow driver sets, opaque sample selection, and limited exploration of model failure modes. We introduce \textbf{FireMPC} dataset, the first pan-Canadian multisource wildfire risk benchmark, covering approximately one billion hectares, or $9{,}840 \times 4{,}639$ km$^2$, across the entire Canada at $1$ km daily resolution from $2000$ to $2025$, and integrating $55$ drivers across fuel, topography, meteorology, and human activity. FireMPC is also the first large-scale machine-learning benchmark to embed the six knowledge-driven indicators of the Canadian Forest Fire Danger Rating System (CFFDRS), bridging knowledge-based and data-driven modeling. Building on this dataset, we propose \textbf{FWI-guided hard negative mining (FWI-HNM)} strategy. We score each non-fire candidate using a calibrated CFFDRS composite, and then incorporate these hard negatives during model training. FWI-HNM mitigates the trivial-negative failure mode of random sampling, yielding F1/PR-AUC gains of $1\%$-$2\%$ and ECE reductions of up to $82\%$ across thirteen benchmark ML models. Stress tests under horizon-extension and spatial-generalization protocols expose substantial generalization gaps. Mean F1 falls from $70.2\%$ at the next-day horizon ($\Delta t = 1$) to $54.2\%$ at the 30-day horizon ($\Delta t = 30$). All six ecological, land-cover, and regional spatial-shift scenarios yield a $25\%$-$29\%$ relative F1 drop. These results motivate the development of large-scale, driver-diverse, spatially comprehensive datasets. The dataset, code, and sampling artifacts are available at \url{https://anonymous.4open.science/r/FireMPC}.
Show more
CT-Lesion: A Multi-Region CT Dataset for Co-existing Lesion Segmentation and Detection
Ruihan Lu ⋅ Leiqiang Pan ⋅ Shijie Wang ⋅ Dingyi Zhang ⋅ Jianjun Shen ⋅ Xin Yu
Lesion analysis in CT imaging, the task of identifying and delineating abnormal regions across diverse anatomical sites, plays a fundamental role in clinical diagnosis. Existing CT lesion datasets typically focus on a single disease type within a specific region. This design leads to two main limitations. First, co-existing lesions outside the predefined categories are left unannotated, so models may treat real lesions as background during training. Second, lesion categories within any single region follow a long-tailed distribution, in which rare but clinically important categories are underrepresented. However, collecting enough samples for every rare category within a single region is often impractical. We observe that certain lesion categories are rare in one region but occur more frequently in another, motivating the use of cross-region complementarity for lesion detection. In this work, we curate a multi-region CT lesion dataset, namely CT-Lesion, which covers brain, chest, and abdomen CT images. Our CT-Lesion is sourced from a private cohort of 500 clinical CT scans. Seven radiologists spent approximately 1,600 working hours delineating lesions of any type, rather than restricting annotations to a predefined set of categories. As a result, CT-Lesion provides clinically confirmed voxel-level segmentation masks and 28 fine-grained lesion labels. Moreover, we further group the lesions into 9 coarse categories to alleviate data scarcity issues. To demonstrate the effectiveness and utility of CT-Lesion, we benchmark the representative methods on CT-Lesion for lesion segmentation and detection. Furthermore, pretraining on CT-Lesion improves model performance on external benchmarks, including LiTS, KiTS, and LIDC-IDRI, demonstrating its potential as a transferable resource for CT lesion analysis. Our dataset is available at Kaggle.
Show more
CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations
Zixian Su ⋅ Hongkai Zhang ⋅ Fan Gao ⋅ Encheng Su ⋅ Taiping Qu ⋅ Jingwei Guo ⋅ Zhang Nan ⋅ Hui Wang ⋅ Zhen Zhou ⋅ Kairui Bo ⋅ Yan Chen ⋅ Yue Ren ⋅ Shuai Li ⋅ Lei Xu ⋅ Henggui Zhang
Multimodal Large Language Models (MLLMs) have shown strong performance on medical public benchmarks, but existing evaluations remain limited as clinically grounded assessments because they often rely on isolated inputs lacking study-level structure and simplified, recognition-style tasks that are far from real diagnostic workflows. We introduce \textbf{CardioLens}, a leakage-resistant evaluation testbed using multi-sequence Cardiovascular Magnetic Resonance (CMR), constructed from private hospital archives via a rigorous report-to-QA construction and verification pipeline. CardioLens is built from 473,896 slices and 13,494 verified QA pairs across 4D Cine, LGE, perfusion, and T2-weighted imaging, and evaluates three stages of CMR interpretation: image understanding, report generation, and disease diagnosis. Across 24 state-of-the-art MLLMs, CardioLens reveals a substantial clinical reality gap. Models perform poorly overall, with performance degrading along the real CMR workflow. Confusion analysis further reveals a category-collapse failure mode: models often default to frequent abnormal categories rather than reliably telling distinct clinical findings. To rule out MLLM-compatible input construction as the failure cause, we compare random, clinically-motivated, and data-driven selection protocols under different slice budgets; performance changes only marginally, typically by about 1\%. Explicit reasoning prompts also fail to rescue performance, often driving models toward conservative predictions rather than improving visual evidence use. These results show that current MLLMs remain far from reliable CMR interpretation, where clinical decisions require integrating distributed evidence across sequences, views, and temporal phases. By exposing where and how current MLLMs fall short, CardioLens provides a clinically grounded testbed for developing the next generation of MLLMs toward real-world clinical depolyment.
Show more
MedEvoEval: Evaluating Continual Evolution of Doctor Agents through Simulated Clinical Episodes
Hui Zhang
Doctor agents are moving beyond single-turn answer generation toward evolving clinical decision systems that acquire evidence within episodes and reuse experience across episodes. Fixed-input medical QA and many interactive benchmarks provide limited support for evaluating how episode-level decisions interact with cross-episode experience. We introduce MedEvoEval, an executable longitudinal evaluation framework based on action-gated simulated outpatient episodes. Each case is converted into role-specific patient, examination, and manager views; evidence is revealed through valid actions, and structured traces link observations, actions, final outputs, scores, and optional experience write-back. Experiments show that these traces expose process costs hidden by final-answer scoring, characterize how MDT-style consultation reallocates resources, and support analyses of memory maturation, held-out transfer, update-stage response, and backward retention. On 200 held-out cases, the mature-memory condition with sequential write-back reduces interaction turns by 0.915 and improves composite utility by 0.0483 after Holm correction. MedEvoEval provides a concrete basis for evaluating experience-driven improvement, transfer, and retention. The accompanying artifact provides 700 processed episode records with provenance, role-specific schemas, and execution, scoring, and analysis code.
Show more
MedHorizon: Towards Long-context Medical Video Understanding in the Wild
Bodong Du ⋅ Bowen Liu ⋅ Yang YU ⋅ Xinpeng Ding ⋅ Zhiheng Wu ⋅ Shuning Wang ⋅ Shuo Nie ⋅ Naiming Liu ⋅ Qifeng Chen ⋅ Yangqiu Song ⋅ Xiaomeng Li
Medical multimodal large language models (MLLMs) have advanced image understanding and short-video analysis, but real clinical review often requires full-procedure video understanding. Unlike general long videos, medical procedures contain highly redundant anatomical views, while decisive evidence is temporally sparse, spatially subtle, and context dependent. Existing benchmarks often assume this evidence has already been localized through images, short clips, or pre-segmented videos, leaving the retrieval-before-reasoning problem under-tested. We introduce MedHorizon, an in-the-wild benchmark for long-context medical video understanding. MedHorizon preserves 759 hours of full-length clinical procedures and provides 1,253 evidence-grounded multiple-choice questions that jointly evaluate sparse evidence understanding and multi-hop clinical reasoning. Its evidence is extremely sparse, with only 0.166% evidence frames on average, requiring models to search noisy procedural streams before interpreting and aggregating findings. We evaluate representative general-domain, medical-domain, and long-video MLLMs. The best model reaches only 41.1% accuracy, showing that current systems remain far from robust full-procedure understanding. Further analysis yields four key findings: performance does not scale reliably with more frames, evidence retrieval and clinical interpretation remain primary bottlenecks; these bottlenecks are rooted in weak procedural reasoning and attention drift under redundancy, and generic sampling methods only partially balances local detail with global coverage. MedHorizon provides a rigorous testbed for MLLMs that retrieve sparse evidence and reason over complete clinical workflows.
Show more
EndoSCOP-V: A Multi-Turn Video Understanding Evaluation Framework for Multimodal Models in Endoscopy Reporting and Clinical Reasoning
Abhishek Tyagi ⋅ TANMAY JAIN ⋅ Vilas Biradar ⋅ Krithi K Koduri ⋅ Misba h Masoodi ⋅ Swetha Rapolu ⋅ HARDIK RUGHWANI ⋅ Rachana Kotha ⋅ Anjan K kaipa ⋅ Anudeep Katrevula ⋅ Rakesh Garlapati ⋅ ANIRUDDHA P SINGH ⋅ Rajendra Patel ⋅ Abhay Tyagi ⋅ SUNDEEP LAKHTAKIA ⋅ MOHAN RAMCHANDANI ⋅ RAKESH KALAPALA ⋅ NAGESHWAR REDDY
Real-world endoscopic diagnosis unfolds across native video, depending on continuous traversal of gastrointestinal organs where a single positive finding triggers a multi-minute reasoning chain over location, morphology, severity, and intervention. Existing endoscopy multimodal benchmarks evaluate only static images or single-disease videos that fail to capture the temporal and clinical reasoning demands of real-world endoscopy reporting. We introduce EndoSCOP-V, a video-grounded multi-turn evaluation framework for long-horizon clinical reasoning in endoscopy. The benchmark contains 1,307 real-world endoscopy and colonoscopy videos (63.2 h; 97% 1080p; mean ~174 s) — the time-scale at which endoscopists actually reason about a finding. The corpus is paired with 20,275 question–answer turns deterministically instantiated from clinician-filled structured endoscopy-reporting EHRs, avoiding synthetic or LLM-rephrased per-case annotations. Questions are derived from 1,033 clinician-reviewed templates spanning a clinically aligned 9-task × 30-sub-task reporting taxonomy across 54 World Endoscopy Organization (WEO) MST 3.0 diseases at ~18 templates per disease, substantially deeper per-disease examination than prior endoscopy benchmarks. To mirror real-world case mix, 22% of procedures carry co-existing pathologies, up to six per case. To model the hierarchical and conditional nature of clinical reporting, EndoSCOP-V introduces a Markovian context-pruned oracle-forcing protocol that evaluates multi-turn dialogues turn-by-turn. By injecting the gold answer before each subsequent turn, the protocol isolates per-turn reasoning from cascading dialogue errors and enables fine-grained analysis of clinical reasoning capabilities. Clinician-marked key-frame anchors further separate temporal-localization from frame-level comprehension failures. A panel of 15 board-certified endoscopists anchors a 78% strict-accuracy baseline on a 1,500-question audit. Across 14 open-weight LMMs, the best 27B-class model reaches 59.8% strict accuracy, falling to 53.4% once format-trivial disease-rejection items are excluded. On a between-subjects spot-check, the best of 3 frontier proprietary models trails the clinician panel by 23 pp on strict accuracy. Sub-task analysis reveals consistent gaps in quantitative measurement, multi-variable grading, and intervention reasoning on long-horizon video, providing a clinically grounded benchmark for evaluating multimodal reasoning in endoscopy.
Show more
StakeBench: Evaluating Language Understanding Grounded in Market Commitment
Yunhua Pei ⋅ Jingyu Hu ⋅ Yiwei Shi ⋅ Hongnan Ma ⋅ Weiru Liu ⋅ John Cartlidge
Existing financial NLP benchmarks often rely on labels supplied by outside observers, measuring how language is perceived rather than what speakers have committed to in the market. We introduce \textbf{StakeBench}, an evaluation framework for language understanding grounded in market commitment. StakeBench links 560,876 comments from 2,261 resolved markets to verified position, action, and market-odds records across Polymarket and Manifold. Supervision is derived from observable market behavior. Position sides, post-comment trading actions, and market-odds trajectories replace human annotation. Four diagnostic tasks test whether models detect market commitment, identify the revealed side, anticipate future action, and perform collective odds projection. Three commitment-aware metrics measure alignment with revealed preferences rather than perceived sentiment. Validity audits and explicit interpretation boundaries help distinguish observable commitment signals from latent belief and causal market-odds impact. Across 15 LLMs and 18 topics and platform settings, models partially recover position-side signals, with Directed Accuracy from 0.506 to 0.599, but show structural failures on later tasks. Ten of the fifteen models collapse to one or two action labels in future action anticipation, and no model consistently improves on the naive odds-direction baseline in collective odds projection. Model scale is not correlated with performance, finance-domain tuning does not improve revealed-side identification, and platform incentives strongly shape higher-order results. StakeBench is packaged with evaluation code and dataset under CC-BY~4.0.
Show more
The Narrative Gap: Can LLMs help us Navigate Diverse Narratives Across Languages?
Nikhil Sharma ⋅ Kelly Marchisio ⋅ Kenton Murray ⋅ Ziang Xiao
Large language models increasingly mediate how individuals seek information about complex, often contested topics. However, no existing benchmark evaluates whether models can navigate the diverse narratives that develop around the same event across languages and regions. We argue that to support diverse information seeking, the system requires four core information navigation capabilities: differentiating narratives, presenting diverse views, resisting confirmatory queries, and achieving linguistic information parity. We introduce \textsc{NarrativeBench}, a benchmark designed to evaluate these capabilities jointly in real-world settings. \textsc{NarrativeBench} comprises 588 news articles in 17 languages from 26 regions spanning 12 geopolitical conflicts. To evaluate the models we curated 213 grounded queries from Reddit resulting in 1917 queries across 9 languages. The unit of evaluation is the model's ability to navigate diversity, not the truth of any single account. Evaluating 14 models, we find current multilingual LLMs (1) are unable to navigate diverse multilingual narratives, (2) lack the ability present diverse narratives across multilingual contexts, (3) have asymmetry in performance across different languages, (4) reduce diversity when faced with confirmatory queries which may exacerbate linguistic filter bubbles, echo chambers and reduce common ground across regions. \textsc{NarrativeBench} provides a capability-grounded foundation for building information systems that can promote equitable information access, reduce polarization, and enhance democratic discourse.
Show more
VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing
Hayeon Bang ⋅ Hounsu Kim ⋅ Wonil Kim ⋅ Juhan Nam
Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench separates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7%. To our knowledge, VocalCoachBench provides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback.
Show more
WordEval: Evaluating Word-Native Operation Fidelity in Document Editing
Jinyoung Park ⋅ Yupan Huang ⋅ Shimin Zhang ⋅ Jiayu Ding ⋅ Yilin Jia ⋅ Yuzhong Zhao ⋅ Tengchao Lv ⋅ Wenshan Wu ⋅ Shaohan Huang ⋅ Nan Yang ⋅ Xiangyang Zhou ⋅ Dongdong Zhang ⋅ Li Dong ⋅ Lei Cui ⋅ Sun Mao ⋅ Xun Wang ⋅ Tao Ge ⋅ Huitian Jiao ⋅ Si-Qing Chen ⋅ Changick Kim ⋅ Furu Wei
Microsoft Word is a central environment for office automation, yet evaluating whether an agent has correctly edited a Word document remains difficult. Unlike plain text generation, Word editing changes a structured document object, where formatting, lists, fields, layout, and object properties may all be part of the requested effect. Existing evaluation protocols based on whole-document text similarity, rendered visual similarity, or LLM-as-a-judge scores cannot reliably determine whether the requested Word-native effect is realized as editable document state or whether relevant surrounding state is preserved. We introduce WordEval, a benchmark for evaluating Word-native document editing from natural-language instructions. Each case consists of an input .docx file and an editing instruction; the evaluator compares the agent’s output with a reference document using deterministic checks over Word-readable object properties. WordEval is constructed by design from planned editing units, rather than sampled from arbitrary documents: each unit specifies an input precondition, target object, operation, parameters, reference-generation procedure, and oracle specification. The current release contains 642 cases, including 546 atomic cases over 231 Word commands and 96 compositional cases instantiated from 27 composition templates. Its analysis layer covers 140 unique object-property pairs across 65 Word COM objects and 15 document subsystems. The compositional cases are linked to corresponding atomic cases for their component operations, allowing analyses of whether failures arise from individual operations or from combining multiple operations in one document. By combining controlled document construction, reference-based evaluation, and command-grounded deterministic oracles, WordEval provides a fine-grained testbed for measuring Word-native document automation.
Show more
SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows
Jian Zhu ⋅ Yuzheng Zhang ⋅ Zeyao Ma ⋅ Bohan Zhang ⋅ Armin Schoepf ⋅ Daniel Woloch ⋅ Peter Wang ⋅ Guangyu Robert Yang ⋅ Samuel Jacob ⋅ Siddharth Nagisetty ⋅ Abhiram Chundru ⋅ Jean Lin ⋅ Spencer Mateega ⋅ Jing Zhang
Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings. We introduce SpreadsheetBench 2, a workflow-level benchmark for spreadsheet agents that covers three task categories: generation, debugging, and visualization. The benchmark is constructed from authentic business data, including financial reports and corporate filings, and is annotated and validated by domain experts. The benchmark contains 321 tasks; each instance averages 11.8 worksheets and requires 593.5 cell modifications, reflecting large multi-sheet workbooks with cross-sheet dependencies. We evaluate eight frontier large language models under a unified multi-turn agent scaffold, and additionally include several LLM-based spreadsheet products as complementary baselines. Results show that current systems remain far from reliable on real-world workflows: the best model achieves 34.89% overall task accuracy, and debugging accuracy is as low as 12.00%. Trajectory analysis and a failure taxonomy further indicate that insufficient spreadsheet inspection and incorrect target-cell selection are the dominant bottlenecks. Together, these findings position SpreadsheetBench 2 as a challenging testbed for advancing reliable spreadsheet automation.
Show more
TSNBench: Benchmarking LLM Proficiency in Time-Sensitive Networking
Rubi Debnath ⋅ Daniel Bujosa ⋅ Luxi Zhao ⋅ Silviu S. Craciunas ⋅ Paul Pop ⋅ Sebastian Steinhorst
We present TSNBench, the first benchmark for evaluating large language model (LLM) proficiency in Time-Sensitive Networking (TSN), a suite of IEEE 802.1 standards for deterministic communication with bounded latency in safety-critical domains such as autonomous vehicles, aviation, defense, and industrial automation. While LLMs have been extensively evaluated on general knowledge tasks, their capabilities in safety-critical networking domains remain largely unexplored. TSNBench comprises 939 expert-validated multiple-choice questions (MCQs) covering diverse TSN mechanisms, along with 100 open-ended Worst-Case Delay (WCD) computation tasks for Credit-Based Shaper (CBS) and Cyclic Queuing and Forwarding (CQF) across varying network topologies and traffic conditions. MCQ answers are validated by domain experts, and open-ended ground truth WCD values are computed using a verified Network Calculus (NC) solver for CBS and closed-form mathematical upper bounds for CQF. We evaluate 16 LLMs and find that although models achieve 67 to 95\% accuracy on MCQs, they fail substantially on open-ended WCD computation. For CBS, only GPT-5 achieves a Mean Absolute Percentage Error (MAPE) of 36.2\%, meaning its predicted WCD deviates by 36.2\% of the actual TSN flow delay on average, while most models exceed 80\%. For CQF, the best model achieves 41.8\% MAPE, with most models clustering between 80\% and 100\%. Such errors are large relative to TSN latency budgets and can lead to violations of real-time constraints and unsafe configurations. TSNBench demonstrates that MCQ benchmarks may overestimate LLM capabilities in safety-critical networking domains.
Show more
Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces
Jiawei Chen ⋅ Ruoxi Xu ⋅ Boxi Cao ⋅ Ruotong Pan ⋅ Zhang yunfei ⋅ Yifei Hu ⋅ Yong Du ⋅ Tingting Gao ⋅ Han Li ⋅ Yaojie Lu ⋅ Yingfei Sun ⋅ Xianpei Han ⋅ Le Sun ⋅ Xiangyu Wu ⋅ Hongyu Lin
The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain constrained to isolated scenarios, narrow action spaces, or synthetic data, failing to capture the holistic nature of authentic human behavior. To bridge this gap, we introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data, integrating long-horizon, cross-scenario, and heterogeneous behavioral patterns into a unified framework. Based on this benchmark, we first provide empirical evidence that previous datasets with isolated scenarios suffer from tunnel vision, whereas real-world decision-making relies on long-term, cross-scenario causal chains. Extensive evaluations of state-of-the-art LLMs reveal that current models struggle to accurately simulate these complex behaviors, with performance plateauing even as context windows expand. Crucially, a systematic comparison between simulated and authentic behaviors uncovers a fundamental structural bias: LLMs tend to converge toward a positive average person, exhibiting hyper-activity, persona homogenization, and a utopian bias. This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
Show more
NarrativeBench: Benchmarking Multi-trait Automated Scoring and Feedback Efficacy of Large Language Models for Chinese Narrative Essays
Jin Wu
Automated scoring and feedback generation for Chinese narrative essays (CNE) represent a critical task in educational measurement. While Large Language Models (LLMs) have demonstrated transformative progress, their potential in CNE evaluation scenarios has yet to be systematically under-explored. Existing essay benchmarks face some challenges: (1) \textbf{Standard Alignment}: The varied complexity of scoring traits and rubrics across different benchmarks makes it difficult to establish a consistent alignment mechanism for heterogeneous evaluation scenarios; (2) \textbf{Evaluation Scope}: Current benchmarks only focus on discriminative scoring accuracy for LLMs, yet they lack a systematic framework for assessing the quality of generative feedback. (3) \textbf{Prompt Coverage}: There is a lack of CNE datasets that cover both closed-ended and open-ended prompts. These issues have, to some extent, constrained the trust of educators in and their adoption of LLM-based methods for essay assessment. To bridge these gaps, we introduce \textbf{NarrativeBench}, a multi-trait and feedback-oriented evaluation benchmark for CNE. NarrativeBench covers both closed-ended and open-ended writing tasks and consists of 4,500 student essays from grades 3 to 5 across 14 different writing prompts. Finally, we benchmark 13 popular LLMs, revealing that their performance exhibits significant gaps compared to human evaluations and pretrained language models. Through NarrativeBench, we comprehensively assess LLM performance on Chinese narrative essay tasks, thus facilitating LLM progress in Chinese essay data analysis domains.
Show more
CoilStellaration: A Dataset and Benchmark for Engineering-Aware Stellarator Coilset Generation
Jan-Hendrik Ewers ⋅ Santiago Cadena ⋅ Nicolò Foppiani ⋅ Andrea Merlo
Stellarators are fusion energy devices that confine hot plasma using a carefully shaped magnetic field, and they are one of the most promising candidates in the race towards commercial fusion. Among the many systems that make up a stellarator, the electromagnetic coils generate the confining magnetic field. Their design is the key factor in deciding whether a stellarator can be practically engineered and cost‑effectively realized. Coilset design is an intrinsically complex problem: it does not admit either a closed or a unique solution, making it suitable for optimization. Coilset optimization is therefore at the heart of stellarator design: a coilset must accurately reproduce a target magnetic field while satisfying stringent engineering constraints on coil spacing, curvature, and length. Although recent datasets have accelerated progress on stellarator equilibria optimization, data-driven coilset design is limited by the lack of standardized datasets that pair target plasma configurations with engineering-aware coilsets. Here, we release a dataset (https://huggingface.co/datasets/proxima-fusion/coilstellaration) of 177,000 optimized stellarator coilsets targeting plasma configurations sampled from the publicly available ConStellaration dataset. For each configuration, we optimized coilsets to reproduce the magnetic field while satisfying a diverse set of engineering constraints. Alongside the dataset, we introduce a benchmark for conditional coilset generation: given a target plasma boundary and a set of engineering requirements, a model should produce a coilset that satisfies the engineering constraints while reproducing the target field. We provide reference code, evaluation scripts and baselines models trained on a subset of the provided data (https://github.com/proximafusion/coilstellaration). Beyond enabling better optimization initializations and one-shot coil design, our dataset provides a testbed for data-driven constrained optimization and for studying which plasma geometry features predict simpler, more engineerable coils. By releasing data, and code for benchmarks and baselines, we aim to lower the barrier for machine learning and optimization researchers to engage with stellarator coilset design and accelerate progress toward engineerable and cost-effective stellarators power plants.
Show more
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
Haoxiang Wang ⋅ Da Yu ⋅ Huishuai Zhang
Evaluating large language models (LLMs) today rests on *fixed benchmarks* that apply the same set of items to any model, producing ceiling and floor effects that mask capability gaps. We argue that the most informative evaluation signal lies at the *boundary*, where the per-prompt pass probability is near 0.5, and propose **Dynamic Boundary Evaluation (DBE)**, which actively locates each model's boundary and places it on a globally comparable difficulty scale. DBE delivers three artifacts: (i) a calibrated item bank covering safety, capability, and truthfulness, with per-item difficulty labels validated across 9 reference LLMs; (ii) **Skill-Guided Boundary Search (SGBS)**, a search algorithm that finds boundary items for a given target LLM using only API-level query access; and (iii) an evaluation protocol that places a new LLM on a unified ability scale and grows the evaluation set adaptively when the target falls outside the bank's coverage. We instantiate DBE on four categories spanning safety (harmful refusal, over-refusal), capability (constrained instruction following), and truthfulness (multi-turn sycophancy resistance). The resulting evaluation covers a broader model spectrum without saturation while remaining compatible with existing datasets.
Show more
HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation
Jaehwi Song ⋅ Suchae Jeong ⋅ Byeongguk Jeon ⋅ Sungdong Kim ⋅ Hyungmok Son ⋅ Kimin Lee
Large-scale demonstration datasets have been central to recent progress in general-purpose robot policies. However, existing datasets are collected in human-absent settings, and policies trained on such data may perform tasks competently in isolation but fail to exhibit human-aware behaviors. To address this gap, we introduce HABIT, a large-scale robot demonstration dataset for human-present environments. We organize tasks into three roles capturing distinct modes of human-robot interaction: Collaborator, where human and robot jointly accomplish a task; Coworker, where they pursue separate tasks in a shared space; and Supervisor, where the human directs the robot. The dataset comprises over 7K episodes and over 100 hours across 48 tasks. Our experiments show that training on human-present data elicits human-aware behaviors that robot-only data fails to produce: spatiotemporal synchronization in Collaborator tasks, yielding in Coworker tasks, and gesture grounding in Supervisor tasks. Moreover, training on HABIT enables rapid adaptation to new human-robot interaction tasks. By introducing human presence as a new axis of dataset diversity, HABIT extends robot policies to environments shared with humans.
Show more
NavOCR: A Dataset Generator for Navigation-Relevant Text Detection in Mobile Robots
Chaehyeuk Lee ⋅ Sooyong Shin ⋅ Sehyeon Yoon ⋅ Youngsun Je ⋅ Jinmyoung Lee ⋅ Seula Lee ⋅ Tu Vo ⋅ Sheir A. Zaheer ⋅ Chan Youn Park
Textual cues provide important navigation-relevant semantic information like room names, store signs, and floor labels. However, conventional optical character recognition (OCR) systems and datasets are designed to detect all visible text, including advertisements and price tags, regardless of their relevance to navigation tasks. This characteristic of OCR can produce noisy semantic cues, making it difficult to identify which text is useful for robot navigation. A straightforward solution is to curate existing OCR datasets by retaining only navigation-relevant text and removing irrelevant annotations, but this process requires substantial manual effort. To address this problem, we introduce NavOCR, a dataset generator for navigation-relevant text detection that does not require manual curation. By leveraging conventional OCR, open map resources, and filtering with a vision-language embedding model, NavOCR identifies which text instances serve as place identifiers for robot navigation and constructs datasets from images available on the web. NavOCR aims to complete the entire process within a single day, from data collection to model training, enabling rapid adaptation to diverse environments and languages. Compact models trained on datasets generated by NavOCR achieve higher detection accuracy and substantially better computational efficiency than vision language models and open vocabulary detectors. We further show that detecting navigation-relevant text improves the quality of semantic cues used for localization and mapping in robot navigation applications.
Show more
HierSVA: A Synthesis Pipeline, Dataset, and Benchmark for LLM-Driven Hierarchical Hardware Formal Verification
Maohua Nie ⋅ Jiang Zhu ⋅ Jingqun Zhang ⋅ Zhichen Zeng ⋅ Jiayi Wang ⋅ Sibo Zhang ⋅ Jialin Wang ⋅ C-J R Shi
We present HierSVA, an integrated suite that combines a pipeline, dataset, and benchmark for LLM-driven hierarchical hardware formal verification. HierSVA-SP pairs an RTL preprocessing toolchain with an LLM-in-the-loop formal verification flow to produce reference SystemVerilog Assertions (SVA) on hierarchical RTL. Applying it to BaseJump STL yields HierSVA-DS, a dataset of 342 modules, with hierarchy metadata and depths 0--9, accompanied by a deep subset of 28 module-bug pairs with natural-language specifications and bug variants. HierSVA-B decomposes assertion quality into six metric axes: syntax correctness, assertion proof success rate, vacuity, faithfulness, mutation coverage, and formal core coverage. Applying HierSVA-B to twelve recent LLMs reveals three findings. First, the module-level compile rate is 67.1\%; among generated assertions in evaluable runs, 82.1\% prove non-vacuously, but the corresponding assertion sets detect only 70.2\% of eligible injected faults and cover 36.2\% of the formal core. Second, on 211 evaluable model--module entries in the deep subset, assertion sets flag buggy RTL with 0.87 recall, but 40\% of predicted-buggy outcomes are false positives on correct RTL, limiting precision to 0.60. Third, agentic mode improves S1-style provability and strength metrics, but gains plateau and oscillate, and we do not evaluate agentic C3 faithfulness.
Show more
VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora
Yuting Xu ⋅ Jiayi Tian ⋅ Jian Liang ⋅ Xin Xiong ⋅ Hang Zhang ⋅ Mu Xu ⋅ Xiao-Yu Zhang
Existing benchmarks have laid the foundation for travel planning agents by establishing API-centric paradigms. However, as the capabilities of Autonomous Agents continue to advance, their evaluation must evolve beyond simple tool execution toward handling the inherent complexities of the open web. Current benchmarks bypass core cognitive hurdles: they fail to account for information noise, ignore multi-source factual contradictions, and overlook the necessity of grounding visual perception into logical planning. We introduce VeriTrip, a verifiable benchmark designed to meet the increasing demands for agent robustness and reliability. VeriTrip shifts the evaluation focus to evidence-grounded reasoning over unstructured multimodal web corpora. It establishes a Multimodal Retrieval Base (MRB) derived from real-world sources, forcing agents to autonomously orchestrate queries across heterogeneous data. A synchronized Verifiable Knowledge Base (VKB) enables a cell-wise verification protocol that precisely quantifies factual reliability, distinguishing systematic reasoning failures from parametric hallucinations. Our evaluations across leading MLLMs reveal a critical \textit{retrieval-reasoning trade-off}: the cognitive load of autonomous retrieval significantly erodes instruction retention. VeriTrip provides the rigorous foundation necessary for the next generation of planning agents capable of operating in unconstrained, multimodal environments.
Show more
SETA: Scaling Environments for Terminal Agents
Qijia Shen ⋅ Zhiqi Huang ⋅ Vamsidhar Kamanuru ⋅ Aznaur Aliev ⋅ Jay Rainton ⋅ Ahmed Awelkair ⋅ Boyuan Ma ⋅ Qizheng Zhang ⋅ Jiwei Fu ⋅ Yuzhen Mao ⋅ Wendong Fan ⋅ Ping Nie ⋅ Philip Torr ⋅ Bernard Ghanem ⋅ Changran Hu ⋅ Jonathan Li ⋅ Urmish Thakker ⋅ Guohao Li
Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interfaces (GUIs). Among these, the terminal command line provides a text-based, general-purpose interface, covering tasks from system operations to data science and machine learning. However, scaling terminal-agent training remains challenging, as it requires diverse and coherent task instructions, executable environments, and reliable verification, while lacking naturally grounded supervision data. In this work, we propose SETA, a scalable framework for generating verifiable terminal environments for reinforcement learning (RL). The framework consists of two pipelines sharing a unified verification mechanism: SETA-Synth converts diverse sources into standardized RL environments, and SETA-Evol further expands from existing environments with adaptive control of difficulty and diversity. Together, we construct and release SETA-Env, the largest open-source verifiable terminal RL dataset to date, containing over $4{,}500$ environments. We evaluate our dataset by training a Qwen3-8B model with GRPO on SETA-Env, achieving $12$\% pass rate on Terminal-Bench 2.0, the best reported result for an RL-trained model at the 8B scale. The results demonstrate that SETA-Env provides high-quality training environments for terminal agents and serves as a valuable resource for advancing research on terminal-based agent learning.
Show more
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
Alexis Limozin ⋅ Eduard F Ďurech ⋅ Torsten Hoefler ⋅ Imanol Schlag ⋅ Valentina Pyatkin
Recent mixed-policy optimization methods for LLM reasoning that interleave or blend supervised and reinforcement learning signals report improvements over the standard SFT-then-RL pipeline. We show that numerous recently published research papers rely on a faulty baseline caused by two distinct bugs: a CPU-offloaded optimizer bug in DeepSpeed that silently drops intermediate micro-batches during gradient accumulation (affecting multiple downstream frameworks including TRL, OpenRLHF and Llama-Factory), and a loss aggregation bug in OpenRLHF that incorrectly weights per-mini-batch losses. Together they suppress SFT performance, with the optimizer bug accounting for most of the gap and the loss aggregation bug contributing a smaller additional effect. Once corrected, the standard SFT-then-RL pipeline surpasses every published mixed-policy method we evaluate by +3.8 points on math benchmarks with Qwen2.5-Math-7B and by +22.2 points with Llama-3.1-8B. Even a truncated variant with just 50 RL steps outperforms mixed-policy methods on math benchmarks while using fewer FLOPs.
Show more
Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift
Achref Doula ⋅ Max Mühlhäuser
Conformal prediction (CP) for language models is typically evaluated under a \emph{matched} protocol, in which calibration and deployment use the same post-training inference configuration, such as quantization mode and inference temperature. In deployment, these choices often differ from those used at calibration to meet memory or throughput constraints or serving policies. We argue that this exposes a limitation in current CP evaluation protocols for language models. We formalize the underlying phenomenon as \emph{inference-time model shift}: a change in the conformal score distribution induced by inference configuration rather than by the input--label distribution, and theoretically characterize how it breaks split-conformal exchangeability even when the task distribution and trained model are unchanged. We then evaluate its consequences across $8$ language models, $4$ finite-label benchmarks, and $12$ inference configurations. We find that matched evaluation can substantially overstate deployment-time coverage, and that the effect is heterogeneous across models and datasets. We argue that inference configuration should be reported as a first-class experimental axis in CP evaluations for language-model systems.
Show more
PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning
Qiran Zhang ⋅ Yuheng Wang ⋅ Runde Yang ⋅ Lin Wu ⋅ Jingru Fan ⋅ Shu Yao ⋅ Jie Zhang ⋅ Tianle Zhou ⋅ Huatao Li ⋅ Ruijie Shi ⋅ Yihan Li ⋅ Chen Qian
Programmatic video generation through code offers geometric precision and temporal coherence unattainable by pixel-level diffusion models, yet rigorously evaluating whether language models can produce spatially correct animated output remains an open problem. We introduce **PRISM**, a large-scale benchmark of 10,372 human-calibrated instruction-code pairs ($20\times$ larger than prior Manim benchmarks) grounded in real-world educational scenarios across English and Chinese, spanning 437 subject categories. Alongside the dataset, we propose a funnel-style evaluation framework with four complementary metrics: *Code-Level Reliability* as the executability gate, *Spatial Reasoning* as the core end-to-end metric measuring layout correctness over full animation sequences, and *Prompt-Aware Dynamic Visual Complexity* (PADVC) and *Temporal Density* (TD) as diagnostic dimensions characterizing dynamic expression and temporal activity. Systematic evaluation of seven mainstream LLMs reveals a striking *Execution-Spatial Gap*: the average drop from execution success rate to spatial pass rate is approximately 41\%, demonstrating that syntactically correct, runnable code does not automatically yield spatially coherent visual output. No single model dominates all dimensions; notably, the model with the highest execution rate exhibits the largest Execution-Spatial Gap, while the most spatially robust model does not lead on code reliability. These findings highlight that evaluating programmatic video generation should not stop at executability. PRISM provides a principled benchmark for advancing spatially coherent code generation.
Show more
Do Image Editing Models Understand Lighting?
Tim Küchler ⋅ Johann-Friedrich Feiden ⋅ Matthias Niessner ⋅ Carsten Rother
While recent advancements in generative image editing models have achieved stunning visual fidelity, it remains an open question whether these systems possess an intrinsic knowledge of real-world lighting. Existing benchmarks typically evaluate high-level plausibility of perceptual light transport on curated internet imagery, using VLMs or human judgement, or they rely on synthetically generated datasets. In this work, we introduce the 3D-anchored Light Probe (3DLP) benchmark, for which we have captured a new high-fidelity HDR dataset of real-world lighting changes. The dataset consists of 1K image pairs of diverse indoor scenery in which light probes are physically turned on and off. To allow for a granular performance analysis, we annotated specific image regions such as cast shadows or metallic surfaces. With this data, we evaluate a range of state-of-the-art image editing models by measuring how well their light probe edits align with reality. The evaluation uses two new scores to compensate for AI-generated photographic effects, such as adjusted white balance. Our results show that the overall performance of models differs considerably, with differences slightly less pronounced for specular highlights. The best image editing models are remarkably consistent with real-world physics, however, they still leave room for improvement. We observe that image regions that receive less light from the light probe are more prone to errors for all models. Furthermore, building on their success in evaluating macroscopic lighting plausibility, we test VLMs on our task but find that they are unsuitable for pixel-level light transport analysis. We will make the benchmark, together with the real-world dataset, publicly available to encourage future research on this topic.
Show more
ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing
Xinghao Chen ⋅ Xiangbo Gao ⋅ Jiongze Yu ⋅ Yuheng Wu ⋅ Zhengzhong Tu
Video scene text editing aims to replace text appearing on scene surfaces in a video, such as storefront signs, whiteboards, and product labels, while preserving the surrounding content, motion, and camera dynamics. Although scene text editing has been extensively studied for still images, its video counterpart remains underdeveloped. Existing resources do not provide paired edits over real-world videos, current protocols rarely measure whether the edited region reads as the target string over time, and there is no large-scale open-source benchmark for systematic comparison. To close these gaps, we introduce ViTeX-Bench, a comprehensive benchmark suite for high-fidelity and temporally consistent video scene text editing. Specifically, we first built ViTeX-Dataset, containing 387 real-world 720p videos with precise text-region masks and editing instructions; the 230-training split provides the first such resource with paired editing results built through a semi-automatic pipeline, with the rest forming the evaluation set for standardized benchmarking. In ViTeX-Bench, we propose a three-axis evaluation protocol: text correctness, visual quality, and edit locality, which contains a total of 13 metrics that combine character-level OCR signals with motion and preservation metrics. Our benchmarking experiments indicate that all eight leading video editing models, including both public and commercial models, fail in distinct ways: they either flicker or drift over time, or simply fail to produce the requested editing effects. To further validate the efficacy of our proposed dataset, we trained ViTeX-Edit-14B on the paired training split with a motion-aligned glyph-video conditioning stream. We show that, trained on only 230 high-quality paired videos, ViTeX-Edit-14B achieves the strongest CharAcc among video-native editors (0.688, +11.1\% over VideoPainter) and leads three ranked temporal metrics, reducing the previous best by 2.1\% (Flicker$_f$), 6.6\% (Flicker$_c$), and 1.9\% (Warp$_c$), respectively.
Show more
CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage
Liu Jiale ⋅ Jungang Li ⋅ Jieming Yu ⋅ Yu Xinglin ⋅ Zihao Dongfang ⋅ Shi Keyu ⋅ Zongjian Ding ⋅ 家焕 张 ⋅ Shunwen Bai ⋅ Haoran Huang ⋅ Yurun Wang ⋅ Yanxi Wu ⋅ Ningzhe Yu ⋅ Yudong Gao ⋅ Mingjun Cheng
Modern 3D visual learning relies on observations sampled from metric 3D assets, yet existing scans, meshes, point clouds, simulations, and reconstructions do not directly provide a sparse, comparable, and geometry-consistent panoramic training interface. Dense trajectories duplicate nearby views, source-specific rendering policies yield heterogeneous annotations, and sparse heuristics may miss important regions or introduce depth-inconsistent observations. *We study how to convert 3D assets into sparse panoramic RGB-D-pose data that preserves complete scene coverage with low redundancy and auditable provenance.* We propose **COVER** (**C**overage-**O**riented **V**iewpoint curation with **E**RP **R**ange-depth warping), a training-free ERP viewpoint curator that projects geometry observed from selected views into candidate ERP probes, scores incremental coverage, and penalizes depth conflicts. Under bounded proxy error, its greedy coverage proxy preserves the standard coverage-style approximation behavior up to an additive error term. Using COVER, we build **CM-EVS** (**C**overage-curated **M**etric **E**RP **V**iew **S**et), a panoramic RGB-D-pose dataset with 36,373 curated ERP frames from 1,275 indoor scenes across Blender indoor, HM3D, and ScanNet++, complemented by outdoor panoramas from TartanGround and OB3D re-encoded into the same schema. Each frame provides full-sphere RGB, metric range depth along ERP rays, calibrated world-to-camera pose, and provenance logs. With a median of only 25 frames per indoor scene, CM-EVS covers all 13 unified room types while maintaining compact scene-level coverage. Experiments show that COVER improves the coverage–conflict trade-off, making CM-EVS a sparse, compact, and auditable RGB-D-pose resource for geometry-consistent panoramic 3D learning.
Show more
Stranger Things: When Objects Appear Without Their Typical Neighbours
Siddhartha Gairola ⋅ Jiahao Xie ⋅ Anna Kukleva ⋅ Francesco Locatello ⋅ Bernt Schiele
Visual context is a powerful cue for object recognition: scenes inform what, where, and at what scale objects are likely to appear. Yet over-reliance on context becomes a failure mode when models predict objects based on their typical neighbours rather than the objects' own visual features, a phenomenon we refer to as typicality bias. While this issue has been extensively studied in the pre-Transformer era, it has often been assumed that architectural and scale improvements would naturally solve it. Despite recent attempts in isolated settings, a systematic diagnostic across the modern vision stack remains absent. In this work, we provide such a diagnostic framework. Specifically, we measure typicality bias via normalized pointwise mutual information (NPMI) over object pairs, and introduce STRANGE-Bench (Synthetic Typicality-RANked Grounded Evaluation Benchmark): a photorealistic, layout-controlled benchmark spanning 11 typicality tiers from typical to never-co-occurring object pairs. Across \five model families (specialist detectors, open-vocabulary detectors, self-supervised, vision-language, and multimodal large language models) and three recognition tasks}(supervised detection, open-vocabulary detection, and classification), we find the bias to be highly prevalent: gaps of 5--11\% persist when objects appear outside their typical company, and they do not consistently shrink with scale. This persistence motivates a complementary question: how to mitigate it? Data augmentation is a natural lever, since it directly shapes the training distribution that produces the bias; yet existing augmentations are designed for visual diversity, not for breaking co-occurrence shortcuts as they sample from the very distribution that produces typicality bias, preserving rather than perturbing it. We therefore propose Anchor-Atypicality Sampling (AAS), a simple context-aware variant of paste-based augmentations that uses the NPMI signal to deliberately tilt the training-time co-occurrence distribution toward atypical partners. AAS reduces the typicality gap by up to 3.3 pp on natural-image COCO splits and 17.4 pp on synthetic STRANGE-Bench across three representative model families (specialist detection, open-vocabulary detection, self-supervised classification), without sacrificing overall performance.
Show more
Mirror, Mirror on the Wall: Can VLM Agents Tell Who They Are at All?
Filippo Ziliotto ⋅ Ciro Beneduce ⋅ Luciano Serafini ⋅ Bruno Lepri ⋅ Massimiliano Luca ⋅ Tommaso Campari
In the animal kingdom, mirror self-recognition is a canonical probe of higher-order cognition, emerging only in some species. We ask whether an analogous functional capability emerges in embodied vision-language model (VLM) agents: can they recognize themselves in a mirror? We introduce a controlled 3D benchmark where a first-person VLM agent must infer a hidden body attribute from its reflection and select the matching target, while avoiding self–other misattribution. To separate mirror-grounded self-identification from shortcuts, we test mirror removal, misleading cues, and occluded reflections. We also evaluate the decision process through mirror seeking, temporal ordering, self-attribution, and reasoning–action consistency. Our experiments show that mirror-based self-identification emerges mainly in stronger VLMs. These models can use reflected evidence for action, whereas weaker models often inspect the mirror but fail to extract self-relevant information or misattribute their reflection. Language–vision conflict further shows that self-referential language alone is not evidence of grounded self-identification. Overall, mirror-based evaluation provides a diagnostic for whether embodied self-grounding is causally rooted in perception and action rather than priors, prompt compliance, or confabulation.
Show more
ELMA: Benchmarking Anaphoric Compositions for Long-Term Text-to-Motion Generation
Shun Kato ⋅ Dixin Yang ⋅ Kotaro Amaya ⋅ Yuiga Wada ⋅ Kazumi Fukuda ⋅ Takashi Shibuya ⋅ Yuki Mitsufuji ⋅ Mariko Isogawa
Text-guided long-term human motion generation is essential for applications such as continuously operating digital avatars. Despite recent progress on short clips, existing models struggle with multi-segment instructions that involve anaphora, where constraints established in earlier segments must be preserved over time. In this paper, we introduce a novel task of long-term motion generation from such anaphoric compositions. To mitigate this, we present ELMA, an episodic long-term motion dataset with dense segment-level annotations that preserve cross-segment dependencies. Built via a fully automated collection and filtering pipeline, ELMA extracts high-quality, continuous 30-second clips from in-the-wild videos, capturing the episodic narratives necessary for long-range state modeling. Leveraging this dataset, we fine-tune state-of-the-art motion generation models to incorporate long-range context. Extensive experiments reveal fundamental shortcomings in current models when handling context-dependent instructions. We demonstrate that ELMA enables significant improvements in both global physical realism and semantic alignment, setting a robust new baseline for long-term motion synthesis.
Show more
VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness
Iulia-Maria Udrea ⋅ Alexandra Diaconu ⋅ Bogdan Alexe
We introduce VSRo-200, the first large-scale dataset for visual speech recognition (lip reading) in Romanian, comprising 200 hours of real-world podcast videos. All samples are annotated with pseudo-labels generated by a fine-tuned Romanian ASR model, while a subset of 100 hours is additionally transcribed by humans, enabling controlled analysis of supervision quality under a unified framework. Building on this dataset, we establish a benchmark for visual speech recognition in low-resource settings. We systematically study the impact of supervision quality, showing that while human annotations provide better performance at fixed data scales, pseudo-labels enable continued improvements through scalability. We further evaluate robustness under domain shift using curated out-of-distribution (OOD) test sets, and analyze audio-visual speech recognition (AVSR) under noisy conditions, where multimodal fusion significantly improves robustness compared to audio-only models. Finally, we demonstrate that representations learned on VSRo-200 transfer effectively to the LRRo benchmark for isolated word recognition, substantially outperforming previously reported results. Overall, VSRo-200 provides a new testbed for studying supervision, domain generalization, and multimodal fusion in low-resource visual speech recognition.
Show more
Re:Cognize - A Framework For Open-Set Comic Character Re-Identification
Aaditya Baranwal ⋅ Madhav Kataria ⋅ Yogesh Rawat ⋅ Shruti Vyas
Comics and manga tell stories through characters whose appearances change from panel to panel, yet readers recognise them hundreds of pages later without ever consulting a cast list. Character re-identification models rarely face the same challenge: existing evaluations give them a fixed set of identities before the story begins. **Re:Cognize** instead asks models to recognise characters as the story unfolds and learn who is who along the way. Across three manga corpora, we evaluate five Re-ID backbones and **MECHA**, a memory-augmented baseline we introduce, and find that the central difficulty lies somewhere unexpected. Recognising familiar characters is nearly solved. Knowing which matches to trust is not. When models build a gallery and grow it from their own predictions, their performance deteriorates, even though correct identity labels would improve it by over twenty points. This raises a simple question: when should a model trust its own match? We derive a single comparison that determines when gallery updates improve recognition and use it to design **VOUCH**, which updates its memory only when the page provides supporting evidence. Without fitting additional parameters, VOUCH recovers between one-third and two-thirds of the improvement available from perfect labels. RE:COGNIZE measures whether a model can recognise characters as it reads; VOUCH shows how to remember them reliably. We focus our methodological claims on identity maintenance for previously encountered characters, while separately evaluating new-character emergence as a diagnostic under a fixed reference rule to distinguish recognition from discovery.
Show more
PatchBench: Measuring Collateral Damage in Activation Patching
Alexi Canesse ⋅ Mathis Le Bail ⋅ Maël JENNY ⋅ Clément Elliker ⋅ Mahammed El Sharkawy ⋅ Sonia Vanier
An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to correct a specific unsafe behaviour without changing unrelated behaviours. A patch may block the exact prompts used for evaluation yet fail on close harmful variants, or suppress harmful behaviour by over-refusing benign prompts that share its wording or structure. Existing evaluation protocols primarily test whether models can be broken, while aggregate metrics, such as attack success, refusal rates, and global capability scores, cannot distinguish selective repairs from broader local suppression. To address this gap, we introduce **PatchBench**, a benchmark of empirically observed model-specific jailbreak failures that induce actionable harmful answers, designed as concrete repair targets for patching research. Starting from 27,870 prompts aggregated from 37 public jailbreak benchmark datasets, we manually curate 15,314 English prompts and query 8 open-source instruction-tuned models. We then combine WildGuard filtering, pairwise Elo ranking, and manual verification to retain only unsafe completions that answer the harmful requests. The result is a curated bank of 400 high-confidence jailbreak failures, organised by model. We further introduce **PatchBench-Local**, a local evaluation protocol that tests whether a patch is behaviourally precise. For each harmful source prompt, PatchBench-Local generates three families of local neighbours: harmful variants that preserve the malicious intent, benign prompts with matched structure, and benign prompts that reuse the key harmful terms. It evaluates both harmful-neighbour correction and benign-neighbour preservation, thus distinguishing selective repair from broader local suppression. We use PatchBench-Local and MMLU to evaluate four state-of-the-art activation steering methods. Our results show that global capability can remain nearly unchanged while local benign-neighbour regressions are severe, confirming that aggregate metrics alone can miss important collateral damage. By exposing whether patches act as precise behavioural repairs or as broad local suppressors, PatchBench-Local provides a more precise basis for developing and comparing jailbreak repair methods.
Show more
Benchmarking Graph Self-Supervised Learning for Node-Level Tasks: Insights and Strong Baseline
Dmitry Eremeev ⋅ Gleb Bazhenov ⋅ Oleg Platonov ⋅ Artem Babenko ⋅ Liudmila Prokhorenkova
Self-supervised learning (SSL) has received notable attention in the graph machine learning community, enabling the efficient use of unlabeled data. However, evaluation setups vary significantly across studies, and baseline tuning often receives limited attention, making it difficult to draw reliable conclusions about model performance. In this paper, we reevaluate four representative graph SSL methods within a unified setup across a diverse set of datasets. We focus on rigorously tuning both SSL and supervised methods by employing enhanced GNN architectures and comprehensive hyperparameter optimization. We observe that, contrary to prior literature, two popular generative methods, MaskGAE and GraphMAE, regularly fail to outperform well-tuned supervised baselines. In contrast, the contrastive methods BGRL and GRACE consistently perform better than both the generative methods and supervised baselines. We hypothesize that this discrepancy arises because BGRL and GRACE capture information from both graph structure and node features, whereas MaskGAE and GraphMAE focus on a single source of information. We support this hypothesis through an analysis on carefully designed synthetic data. Motivated by our observations, we advocate for designing SSL objectives that capture both feature and structural information. To verify the effectiveness of this approach, we propose a simple generative method, GrASP, which reconstructs both graph structure and node features. Despite its simplicity, GrASP outperforms all other evaluated approaches and can be considered as a strong baseline in future studies.
Show more
PolarScale: A Physics-Grounded Benchmark for Radiometrically Consistent RGB-to-Stokes Estimation
Beibei Lin ⋅ Tingting Chen ⋅ Xin Zhang ⋅ Wenhao Zhao ⋅ DONGJUN LI ⋅ Zifeng Yuan
Polarization imaging provides rich physical cues beyond RGB imaging, yet acquiring such information typically requires specialized polarization-sensitive hardware that increases system cost and deployment complexity. Recent methods have proposed to infer polarization information directly from standard RGB images, providing a hardware-free alternative. While useful for describing relative polarimetric structure, these representations do not fully assess whether estimated images preserve the radiometric scale and intensity-dependent physical information needed for full Stokes reconstruction. To address this limitation, we introduce PolarScale, a physics-grounded benchmark for radiometrically consistent RGB-to-Stokes estimation. Instead of treating polarization inference from RGB images as conventional image-to-image translation, PolarScale evaluates how scale-independent descriptors, scale-dependent Stokes components, and radiometric scale jointly support physically meaningful Stokes reconstruction. Building upon this formulation, we propose three prediction strategies, namely direct, joint, and decoupled prediction, to study how scale-independent and scale-dependent polarimetric cues should be modeled across representative restoration-based and generative frameworks. Our analysis reveals that restoration-based frameworks outperform generative ones for radiometrically consistent polarization estimation (e.g., 28.61 vs. 26.14 dB PSNR). Furthermore, we find that jointly modeling scale-independent and scale-dependent parameters yields more stable physical representations than direct scale-dependent prediction (e.g., 28.61 vs. 26.81 dB PSNR). Overall, PolarScale provides a physics-grounded benchmark foundation for evaluating whether vision models can recover radiometrically consistent polarimetric signals from standard RGB images.
Show more
Characterizing the Aesthetic Defaults of Generative Image Models
Maty Bohacek ⋅ Raina Panda ⋅ Daniel Fein ⋅ Arpita Singhal ⋅ Mark Fiore ⋅ Bill Freeman ⋅ Maneesh Agrawala
Contemporary text-to-image (T2I) models are widely observed to exhibit a recognizable, model-specific look even when prompted without any stylistic instruction (e.g., the so-called ``Midjourney look''). What such a default aesthetic actually is, however, has not been characterized: dominant T2I evaluation paradigms score realism, prompt alignment, or pairwise preference, but cannot say which concepts drive separation, how they shift across releases, or how they relate to real-image distributions. We address this with a concept-level decomposition of generator outputs along an interpretable, style-trained sparse vocabulary. We introduce LouvreSAE, a sparse autoencoder over CLIP image embeddings whose dictionary is shaped to fall on stylistic rather than object features, and LatentAesthetic, an evaluation methodology that uses it to estimate, ground, and compare generators' default aesthetics under content-controlled, style-neutral prompts. Applied to 26 generators spanning four years, our methodology surfaces stable per-generator profiles, a uniform pull toward lifestyle and editorial photography over art-historical imagery, and a steady cross-family contraction of the aesthetic envelope, supporting hypotheses of an algorithmic monoculture in image generation. Code, weights, and data are available at https://hf.co/datasets/neurips-subm/anonymous for the evaluation of future models.
Show more
When the Most Disruptive Messages Are Safe to Prune in Aggregate: Functional Analysis of Communication Pruning in Two-Agent LLM Systems
Zhen Sun ⋅ Xueyang Zhou ⋅ Yongchao Chen
Multi-agent LLM systems exchange messages whose communicative necessity varies widely, yet no systematic evaluation protocol determines which messages can be safely removed. We introduce, to our knowledge, the first message-level functional analysis protocol for evaluating communication in two-agent LLM systems. The protocol comprises three reusable stages (annotate, perturb, prune), each deployable to new backbones, domains, and topologies without task-specific training data. We release a 21,661-message annotated corpus across 3 backbones and 4 reasoning domains with a five-class taxonomy for mechanism exploration, a zero-shot annotation pipeline using constrained decoding, and a statistical hypothesis-testing framework combining TOST equivalence, non-inferiority, and McNemar's tests. Human validation (N=225) confirms annotation reliability (κ = 0.81 binary ack/non-ack, underpinning all pruning claims; 0.76 five-class); the cross-model annotation pipeline achieves κ=0.72. Applying the protocol across 18 configurations spanning 7B–72B scale, 4 reasoning domains, and 3 topology variants, with 10-seed replication for flagship settings, we surface a perturbation–pruning dissociation: the most disruptive message type under single removal is safe to remove in bulk, enabling acknowledgment-targeted pruning that suppresses 36–56% of inter-agent messages from conversation context at non-inferior accuracy under greedy decoding. The protocol further identifies boundary conditions where pruning fails and reveals backbone-dependent communication mechanisms, providing a screening tool for deployment decisions.
Show more
MedMemoryBench: Benchmarking Agent Memory in Personalized Healthcare
Yihao Wang ⋅ Haoran Xu ⋅ Renjie Gu ⋅ Yixuan Ye ⋅ Xinyi Chen ⋅ Xinyu Mu ⋅ Yuan Gao ⋅ guo chunxiao ⋅ Peng Wei ⋅ Jinjie GU ⋅ Huan Li ⋅ Ke Chen ⋅ Lidan Shou
The large-scale deployment of personalized healthcare agents demands memory mechanisms that are exceptionally precise, safe, and capable of long-term clinical tracking. However, existing benchmarks primarily focus on daily open-domain conversations, failing to capture the high-stakes complexity of real-world medical applications. Motivated by the stringent production requirements of an industry-leading health management agent serving tens of millions of active users, we introduce **MedMemoryBench**. We develop a human-agent collaborative pipeline to synthesize highly realistic, long-horizon medical trajectories based on clinically grounded, synthetic patient archetypes. This process yields a massive, expertly validated dataset comprising approximately 2,000 sessions and 16,000 interaction turns. Crucially, MedMemoryBench departs from traditional static evaluations by pioneering an *evaluate-while-constructing* streaming assessment protocol, which precisely mirrors dynamic memory accumulation in production environments. Furthermore, we formalize and systematically investigate the critical phenomenon of memory saturation, where sustained information influx actively degrades retrieval and reasoning robustness. Comprehensive benchmarking reveals severe bottlenecks in mainstream architectures, particularly concerning complex medical reasoning and noise resilience. By exposing these fundamental flaws, MedMemoryBench establishes a vital foundation for developing robust, production-ready medical agents. Our dataset and benchmark are publicly available at: .
Show more
EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
Yuyao Wang ⋅ Zhongjian Zhang ⋅ Mo Chi ⋅ Kaichi Yu ⋅ Yuhan Li ⋅ Miao Peng ⋅ Bing Tong ⋅ Chen Zhang ⋅ Yan Zhou ⋅ Jia Li
Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agents, as it enables them to store, update, and retrieve information over time. This ability remains under-evaluated, largely because existing benchmarks do not provide a systematic way to assess memory mechanisms. In this paper, we study agent memory from a self-evolving perspective and introduce \texttt{EvoMemBench}, a unified benchmark organized along two axes: memory scope (in-episode vs.cross-episode) and memory content (knowledge-oriented vs.execution-oriented). We compare 15 representative memory methods with strong long-context baselines under a standardized protocol. Results show that current memory systems are still far from a general solution: long-context baselines remain highly competitive, memory helps most when the current context is insufficient or tasks are difficult, and no single memory form works consistently across all settings. Retrieval-based methods remain strong for knowledge-intensive settings, whereas procedural and long-term memory methods are more effective for execution-oriented tasks when their stored experience matches the task structure. We hope EvoMemBench facilitates future research on more effective memory systems for LLM-based agents. Our code is available at \url{https://anonymous.4open.science/r/EvoMemBench-2}.
Show more
LL-Bench: Rethinking Low-Level Vision Evaluation in the Era of Large-Scale Generative Models
Lu Liu ⋅ Huiyu Duan ⋅ Chenxin Zhu ⋅ Jintong Lu ⋅ Haoyun Jiang ⋅ Liu Yang ⋅ Qiang Hu ⋅ Guangtao Zhai ⋅ Xiaoyun Zhang
Large-scale generative models have demonstrated remarkable capabilities across image generation and editing tasks. However, their performance in low-level vision tasks, which require pixel-wise control, remains insufficiently studied. To address this gap, we introduce \textbf{LL-Bench}, a comprehensive \textbf{Benchmark} for evaluating the capabilities of large-scale generative models on \textbf{L}ow-\textbf{L}evel vision tasks. The benchmark comprises 2,469 real-world degraded images covering 16 low-level degradation tasks, and 28,919 restored images produced by 10 state-of-the-art large-scale generative models and 21 conventional restoration models, which are annotated with 152,020 expert-level pairwise human preferences and 28,334 quality scores. Built upon LL-Bench, we present a systematic diagnosis that reveals the performance boundaries and unique failure modes of large-scale generative models across diverse low-level vision tasks, compared with conventional representative restoration approaches. Moreover, we investigate the effectiveness of current quality evaluation metrics on LL-Bench, which exhibit significant discrepancy with human ratings. To better align restored-image quality assessment with human preferences, we further propose \textbf{LL-Score}, an MLLM-based evaluator that captures both restoration quality and hallucination existence. Extensive experiments demonstrate that LL-score not only outperforms existing image quality assessment metrics, but also serves as a promising reward model for training generative models on low-level vision tasks. The database and code are available at \url{https://anonymous.4open.science/r/LL-Bench-04F0}
Show more
ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety
Michael S Lee ⋅ Yash Maurya ⋅ Drew Rein ⋅ Bert Herring ⋅ Jonathan Nguyen ⋅ Kyungho Song ⋅ Udari Madhushani Sehwag ⋅ Jiyeon Cho ⋅ Kaustubh Deshpande ⋅ Yeongkyun Jang ⋅ Jiyeon Joo ⋅ Minn S Choi ⋅ Evi Fuelle ⋅ Christina Q Knight ⋅ Joseph Brandifino ⋅ Max Fenkell
Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual safety is typically assessed through translation-only benchmarks that preserve the underlying scenario, and empirical evidence of how language and geopolitical context interact remains limited to a narrow set of language pairs. We introduce ROK-FORTRESS, a bilingual, culturally adversarial NSPS benchmark that uses the English--Korean language pair and U.S.--ROK geopolitical axis as a case study, separating the effects of language and geopolitical grounding via a transcreation matrix: adversarial intents are evaluated under controlled combinations of (i) English versus Korean language and (ii) U.S. versus Korean entities, institutions, and operational details. Each adversarial prompt is paired with a dual-use benign counterpart to quantify over-refusal. Model responses are then scored using calibrated LLM-as-a-judge panels, applying our expert-crafted, prompt-specific binary rubrics. Across a dual-track set of frontier and Korean-optimized models, we find a consistent suppression effect in Korean variants and substantial model-to-model variation in how geopolitical grounding interacts with language. In many models, Korean grounding mitigates the Korean language-driven suppression---with no model showing significant amplification in the other direction---indicating that, at least in the English--Korean case, safety behavior is shaped by language-as-risk signals and context interactions that translation-only evaluations miss. The transcreation matrix methodology is designed to generalize to other language--culture pairs.
Show more
ViMU: Benchmarking Video Metaphorical Understanding
Qi Li ⋅ Xinchao Wang
Any new medium, once it emerges, is used for more than the transmission of overt content alone. The information it carries typically operates on two levels: one is the content directly presented, while the other is the subtext beneath it—the implicit ideas and intentions the creator seeks to convey through the medium. Likewise, since video technologies became widely adopted, video has served not only as a powerful tool for recording and communicating visual information, but also as a vehicle for emotions, attitudes, and social meanings that are often difficult to articulate explicitly. Thus, the true meaning of many videos does not reside solely in what is shown on screen; it is often embedded in context, style of expression, and the viewer’s social experience. Some forms of such video subtext are humorous, while others carry irony, mockery, or criticism. These implicit meanings can also be interpreted very differently across cultural backgrounds and social groups. However, most existing video understanding models still focus primarily on literal visual comprehension, such as recognizing objects, actions, or temporal relations, and lack a systematic ability to understand the metaphorical, ironic, and social meanings embedded in videos. To bridge this gap, we introduce ViMU (Video Metaphorical Understanding), the first benchmark designed to systematically evaluate the subtext understanding capabilities of frontier models in videos. ViMU assesses whether video understanding models can go beyond literal perception to infer implicit meaning, rhetorical devices, social signals, target subjects, and culturally grounded subtext, while grounding their interpretations in multimodal evidence and answering both open-ended and multiple-choice questions. Importantly, all questions are designed to be hint-free, ensuring that no key evidence is disclosed to models before answering. Extensive experiments show that most frontier models, including closed-source ones, achieve below 50\% overall performance. We further conduct fine-grained analyses to uncover distinctive model behaviors. Disclaimer: This paper contains potentially offensive and harmful content.
Show more
OpenMHC: Accelerating the Science of Wearable Foundation Models
Narayan Schuetz ⋅ Yuze Bai ⋅ Lianggang Pan ⋅ Edgar Eggert ⋅ Favour Nerrise ⋅ Juan Delgado-SanMartin ⋅ Max Rosenblattl ⋅ Milana Gurbanova ⋅ Mohammad Asadi ⋅ Anders Johnson ⋅ Paul Schmiedmayer ⋅ Dennis Wang ⋅ Allan Lawrie ⋅ Daniel S Kim ⋅ Xin Liu ⋅ Akshay Paruchuri ⋅ Ehsan Adeli ⋅ Euan Ashley ⋅ Kelly W Zhang
Mobile and wearable devices offer an unprecedented opportunity for continuous, passive health monitoring and active health coaching. However, the largest wearable datasets are not publicly available for research, and leading wearable foundation models trained on such datasets are rarely open-weight or come with reproducible training code. To accelerate open science in wearable health, we release **OpenMyHeartCounts (OpenMHC)**, the largest and most comprehensive open-access wearable health dataset to-date, alongside open-source implementations of recent wearable foundation models. \openmhc{}, derived from over a decade of data collected through the My Heart Counts study app, includes >60 million hours of wearable data across 19 sensor channels (e.g., step count, heart rate, sleep, workouts) and up to 169 linked variables, including health, lifestyle, mood, and behavior from 11,894 consenting participants. Furthermore, we introduce a unified, open benchmark that enables standardized comparison of wearable health models across three tracks: health and behavior downstream prediction, multivariate data imputation, and time-series forecasting. We benchmark classical methods alongside recent wearable and multivariate time series foundation models. By open-sourcing data, code, and model weights at this unprecedented scale, we aim to democratize wearable health AI research and enable the community to drive open progress in this domain.
Show more
Beyond Right and Wrong: Evaluating Second-order Social Reasoning in Large Language Models
Sunny Rai ⋅ Jinyi Kuang ⋅ Reyhan Jamalova ⋅ Niyati Malhotra ⋅ Cristina Bicchieri ⋅ Victor Orozco-Olvera ⋅ Ana Maria Munoz Boudet ⋅ Annie Lou ⋅ Lyle Ungar ⋅ Sharath Chandra Guntuku
Previous AI alignment efforts have focused primarily on first-order social norms -- teaching models what is socially acceptable or unacceptable (e.g., 'do not steal'). However, social intelligence depends not only on norm recognition, but also on anticipating who will enforce it and how (e.g., public shame or even imprisonment). These second-order expectations, known as metanorms, govern how people respond when social rules are broken. We introduce a novel framework for evaluating metanorm reasoning in Large Language Models (LLMs) along two dimensions: emotional appraisal and behavioral response, and propose new classification tasks, namely, predicting self-regulation in violators and other-regulation in observers. We release a multi-perspective dataset, NormReact, of 450 norm violation scenarios, hand-annotated for emotions and behavioral responses across norm violators' gender and observers' social closeness. Current LLMs portray a harsher social world: across six models, they overpredict negative sanctions where humans would expect inaction, and alignment with human judgments deteriorates as social distance increases. These findings suggest that AI systems in norm-sensitive domains, from conflict mediation to policy simulation, may risk producing a distorted picture of social regulation: one that over-represents punishment and under-represents the tolerance, restraint, and relational calibration that characterize actual norm enforcement in the real world.
Show more
PINNBench: A Benchmark and Evaluation Study of Training Policy Selection in Hybrid PINN-Operator Solvers
Suan Lee ⋅ Namhyeon Kim ⋅ Dongmin Jin
We introduce PINNBench, a benchmark and evaluation study for training-policy selection in hybrid PINN-operator PDE solvers, contributed to the NeurIPS 2026 Evaluations and Datasets (ED) Track. PINNBench comprises 1,539 recorded runs and probes across 13 PDE configurations drawn from 8 equation families, 3–5 training policies per PDE, and 10 seeds per condition, with seven standardized metrics, Wilcoxon signed-rank tests, paired bootstrap 95% confidence intervals, Cohen's d, and Benjamini–Hochberg FDR correction. Baseline evaluation of 5 routing methods establishes three findings that bear directly on how PDE solvers should be evaluated. (i) No single policy dominates: hybrid_full wins 7/13, hybrid_no_gate 4/13, pinn_only 2/13. (ii) Accuracy and regret dissociate: under relative L2, a physics-loss heuristic and a learned Random Forest router reach identical 62% top-1 accuracy, but the heuristic suffers mean regret 0.0850 versus 0.0011 — a 77x gap. Evaluation by accuracy alone is misleading for PDE policy routing. (iii) Causal-loss effectiveness is stage-mediated: 7/9 PDEs improve with pretraining (+5.9% mean) while 5/9 degrade without it (-4.2%). A new leave-one-PDE-family-out evaluation shows that the strongest family-level generalizer uses no PDE-identity features, addressing the concern that PDE-aware routing memorizes PDE identity. All 1,539 result JSONs, PDE implementations, policy configurations, statistical pipeline, and routing infrastructure are released as open-source artifacts.
Show more
DeepfakeGenome: Toward Next-Generation Deepfake Attribution
Shunli Wang ⋅ Xinyu Zhou ⋅ Yandan Zhao ⋅ Wenbin Liang ⋅ Taiping Yao ⋅ Ke-Yue Zhang ⋅ Zhiyuan Yan ⋅ Shouhong Ding ⋅ Lizhuang Ma
In recent years, AIGC technologies are capable of generating hyper-realistic forged facial images, posing severe threats to facial security safety. To tackle these challenges, early Deepfake research primarily centered on real-fake classification. Recently, growing efforts have been devoted to the Deepfake Attribution (DFA) of generated content. Nevertheless, existing research on Deepfake detection and attribution faces saturated performance in binary classification, limited diversity in datasets and algorithms, and imperfect evaluation protocols, which severely impede practical application. To address these limitations, we propose a comprehensive deepfake detection and attribution benchmark named DeepfakeGenome (DFG). It contains 100 facial forgery algorithms and 2M images in total, achieving 4× to 100× larger than prior DFA benchmarks. We further designed 4 protocols for practical evaluation, including a novel retrieval-based attribution paradigm. Unlike previous open-set evaluation metrics, the proposed retrieval metrics are more aligned with the real-world active defense situation of blacklist registration mechanisms. Based on these elaborate designs, we investigate the performance ceiling of deepfake attribution task. Over 2k+ experimental evaluations are conducted, and 10 insightful findings are derived. We hope this work can provide new insights into the DFA research field.
Show more
Mind the Gap: Dataset and Fine-grained Evaluation for Inline Audio Descriptions
Subhashini Venugopalan ⋅ Yingwen Tan ⋅ Taylor Roper ⋅ Jimmy Tobin ⋅ Anton Kast ⋅ Alicia Martin ⋅ Sam Sepah ⋅ Amy Pavel
Audio descriptions (AD) are essential for making visual media accessible to blind and low-vision (BLV) users. While Multimodal Large Language Models (MLLMs) offer a scalable solution for generating audio descriptions on-demand, their performance on ``inline'' audio descriptions -- which must fit within existing silences in a video -- remains under-explored, particularly for diverse user-generated content (UGC). We present a comprehensive investigation into MLLM-generated inline audio descriptions. First, we introduce a professionally annotated dataset of 36 hours of videos, each averaging $\sim$5 mins. and total 7k+ descriptions, providing a high-quality benchmark for this task. Second, we propose an evaluation framework based on audio description expert guidelines that provides fine-grained actionable metrics for model improvement. Crucially, our framework evaluates the end-to-end task: it renders the generated scripts into audio tracks to assess their fit within the original video's natural silences, minimizing disruptive overlaps. Our analysis reveals that while frontier MLLMs accurately describe visual events, significant gaps persist in audio-visual timing, quality, and narrative flow. Finally, we validate the framework and conduct studies with BLV participants and sighted raters, finding that while users narrowly prefer human-authored audio descriptions, they still find MLLM-generated audio descriptions highly beneficial for comprehension. We provide guidance on the current capabilities of MLLMs for audio descriptions and release our dataset and evaluation suite to the community.
Show more
COSMIO: A Benchmark for Cross-Survey Modality Imputation
Dichang Zhang ⋅ Yixuan Shao ⋅ Jiali Cui ⋅ Haotian Yin ⋅ Yuanpeng Liu ⋅ Jiaiq Deng ⋅ Xiang Gao ⋅ Zhiqiang Lao ⋅ Heather Yu ⋅ Simon Birrer ⋅ Dimitris Samaras
Large-scale astronomical surveys define a heterogeneous, partially observed multi-survey learning setting: each object is observed by only a subset of surveys, while survey observations differ in resolutions, wavelength coverage, noise characteristics, and footprints. We formalize this as the problem of cross-survey modality imputation and introduce COSMIO, a multi-survey benchmark of aligned observations obtained by forward-modeling shared astrophysical scenes across LSST, Euclid, and Roman under survey-specific instrument characteristics. The benchmark enables systematic evaluation of imputation across source-target survey combinations and missing-survey patterns. We evaluate a broad range of machine learning methods for imputation. Our results reveal a clear domain gap: methods developed for existing imputation settings transfer poorly to the astronomical survey setting, indicating that cross-survey imputation constitutes a distinct learning problem. Beyond common-object imputation, we identify missing-modality rare-event imputation as a central challenge. This regime arises in the early stages of new surveys, when rare but scientifically valuable phenomena may be available only from old surveys. We use strong gravitational lensing, where light from a background galaxy is gravitationally deflected by a foreground galaxy, as a representative rare-event setting. We evaluate model performance on zero-shot imputation of gravitational lenses and on downstream scientific tasks using the imputed lens observations. On our constructed evaluation set of galaxy–galaxy strong lenses, performance degrades under rare-event distribution shift, and this degradation further propagates to the downstream lens-detection task. Our results establish cross-survey imputation as a distinct and challenging machine learning problem, highlighting the need for methods that are robust to distribution shift and aligned with scientific objectives. Code and data are available at https://anonymous.4open.science/r/COSMI-3671/.
Show more
probly: Uncertainty-Aware Machine Learning
Paul Hofman ⋅ Clemens Damke ⋅ Timo Löhr ⋅ Santo Thies ⋅ Alireza Javanmardi ⋅ Valentin Margraf ⋅ Jakub Paplhám ⋅ Yusuf Sale ⋅ Eyke Hüllermeier ⋅ Maximilian Muschalik
As machine learning systems are increasingly deployed in real-world applications, the question of how to represent and quantify uncertainty has moved from a methodological side issue to a central concern. In practice, however, making a model uncertainty-aware is still surprisingly difficult: relevant tools tend to be scattered across libraries, each tied to a particular framework and a particular approach to modeling uncertainty, and the choice of representation, quantification, and evaluation is typically left to the user without much guidance. In this paper, we present probly, a Python package that addresses theses issues in a single, modular framework. probly offers (i) lightweight transformations that turn existing models into uncertainty-aware ones, currently supporting PyTorch, scikit-learn, and Flax, (ii) several representations, including second-order distributions, credal sets and conformal prediction, (iii) the corresponding quantification measures, and (iv) a number of evaluation protocols, which can be combined more or less arbitrarily. Using probly, we conduct a benchmark study on three standard tasks --- selective prediction, out-of-distribution detection, and active learning. In addition, and unlike existing benchmarks, we evaluate suitable methods on first-order data, i.e., datasets for which the target itself is a distribution over outcomes. We further illustrate the flexibility of the package on a number of less standard case studies, including large language models, graph neural networks, and data streams. The package is publicly available at \url{https://anonymous.4open.science/r/probly}.
Show more
URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment
Bogdan Zagribelnyy ⋅ Ivan Ilin ⋅ Nikita Bondarev ⋅ Anton Morgunov ⋅ Arkadii Lin ⋅ Maksim Kuznetsov ⋅ Rim Shayakhmetov ⋅ Vlad Aladinskiy ⋅ Alex Aliper ⋅ Alex Zhavoronkov
Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Recent progress has produced both specialized deep-learning retrosynthesis systems and general-purpose large language models, but objective comparison remains difficult due to the lack of flexible, chemically interpretable benchmarking protocols. In the current study, we are introducing the URSA (Utilitarian RetroSynthesis Assessment) evaluation framework that provides the opportunity to benchmark the synthetic routes not only from a formal perspective, such as convergence to commercially available starting materials, but also from a chemical plausibility perspective, mimicking the way expert chemists evaluate the reactions and routes. The study covers a comprehensive evaluation of both conventional end-to-end retrosynthesis solutions and LLMs for the synthesis planning task on a set of novel, diverse target molecules with undisclosed synthetic routes, which represent realistic tasks in the daily drug design routine. We find that while LLMs can support high-level strategic planning, they currently underperform specialized retrosynthesis models in reliably solving synthesis planning tasks.
Show more
RipBench: A Unified Benchmark for Multi-Level Rip Current Detection, Classification and Segmentation
Andrei Dumitriu ⋅ Aakash Ralhan ⋅ Florin Miron ⋅ Florin Tatui ⋅ Radu Tudor Ionescu ⋅ Radu Timofte
Rip currents are a serious and often under-addressed threat to beach safety, and the leading cause of coastal drownings worldwide. They are difficult to detect due to their amorphous structure, similarity to the background, and large variability in viewpoints and environments. Despite their dangers and growing interest in automated detection, existing research remains fragmented across datasets, metrics, models and approaches, leading to an incomplete understanding of model performance. We introduce RipBench, a unified benchmark that enables controlled evaluation of rip current detection spanning multiple tasks, including classification, axis-aligned and oriented object detection, and instance, semantic, and panoptic segmentation, on the same data with standardized splits. This allows direct comparison of model performance on all levels of visual abstraction. Across $304$ videos ($303{,}491$ frames) collected from diverse coastlines, our results expose a clear performance gap between coarse recognition and precise spatial understanding. While models achieve near-saturated classification performance, accurate localization proves to be a substantially more challenging task, with performance varying across tasks. All tasks are supported with carefully curated annotations and evaluated using both standard and safety-critical metrics, with a focus on the $F_2$ score to emphasize recall in this safety-critical setting. RipBench, along with multiple baseline models per task, is publicly available at \url{https://RipBench.ai} to support progress in real-world, multi-task vision for beach safety.
Show more
LiveOption: Evaluating LLM Agents in Structured Option Trading with Nonlinear Payoffs
Haochen Luo ⋅ YIFAN LI ⋅ An B Minh ⋅ Xiaolong Luo ⋅ Zhengzhao Lai ⋅ Yuan Zhang ⋅ Chen Liu
Large language models (LLMs) and multi-agent systems (MAS) have shown promise in financial decision-making, yet existing evaluations focus on equity trading and primarily assess directional prediction, overlooking the structural complexity of derivative markets. Option trading introduces fundamentally different challenges, including nonlinear payoffs and multi-leg strategy construction, requiring structured decisions rather than simple directional bets. We introduce LiveOption, an evaluation framework for LLM-based and reinforcement learning agents in option trading. LiveOption formulates the problem as structured sequential decision-making under realistic execution and capital constraints, and provides a reproducible environment with standardized interaction protocols. The framework includes several task suites covering portfolio overlays, open-ended strategy construction, event-driven earnings trading, and 0DTE intraday trading. We further propose a hierarchical metric suite that evaluates action validity, decision quality, risk characteristics, and outcome-level performance. Experiments show that current agents often fail to construct valid strategies and handle nonlinear payoff dynamics, or achieve competitive returns in most scenarios. LiveOption offers a principled testbed for evaluating structured decision-making beyond outcome-based metrics.
Show more
FutureSim: Replaying World Events to Evaluate Adaptive Agents
Shashwat Goel ⋅ Nikhil Chandak ⋅ Arvindh Arun ⋅ Ameya Prabhu ⋅ Steffen Staab ⋅ Moritz Hardt ⋅ Maksym Andriushchenko ⋅ Jonas Geiping
AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, we propose building grounded simulations that replay real-world events in the order they occurred. We build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. We evaluate frontier agents in their native harness, testing their ability to predict world events over a three-month period from January to March 2026. FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse calibration than a constant predictor. Through careful ablations, we show how FutureSim offers a realistic setting to study emerging research directions like long-horizon test-time adaptation, search, memory and reasoning about uncertainty. Overall, we hope our benchmark design paves the way to measure AI progress on long-horizon open-ended adaptation spanning multiple months in the real world.
Show more
Probing the Trajectories of Reasoning Traces in Large Language Models
Marthe Ballon ⋅ Brecht F Verbeken ⋅ Vincent Ginis ⋅ Andres Algaba
Large language models (LLMs) increasingly solve difficult problems by producing reasoning traces before emitting a final response. However, it remains unclear how accuracy and decision commitment evolve along a reasoning trajectory, and whether intermediate trace segments provide answer-relevant information beyond generic length or stylistic effects. Here, we propose a trajectory-probing protocol for evaluating how a reasoning model's predicted answer evolves along a partial reasoning trace. The protocol 1) generates a model's full reasoning trace, 2) truncates it at fixed token-percentiles, and 3) injects each partial trace back into the model, measuring the model's induced answer distribution. We apply the protocol to five open-source reasoning models (Qwen3-4B/-8B/-14B and gpt-oss-20b/-120b) across three benchmarks (GPQA Diamond, MMLU-Pro, and Omni-MATH-2). The protocol reveals that accuracy and decision commitment consistently increase as the percentage of provided reasoning tokens grows, and length, form, and token-identity controls confirm that gains stem from instance-specific semantic content rather than context length or generic reasoning style effects. Furthermore, weak-to-strong cross-model probing experiments reveal when stronger models recover from incorrect partial traces and when they instead anchor to them. Together, these results provide a reproducible, model-agnostic measurement of reasoning-trace dynamics that generalizes across answer formats and can reveal model-specific failure modes invisible to aggregate accuracy.
Show more
E-GEO: A Testbed for Generative Engine Optimization in E-Commerce
Puneet S Bagga ⋅ Vivek Farias ⋅ Tamar Korkotashvili ⋅ Tianyi Peng ⋅ Yuhang Wu
With the rise of large language models (LLMs), generative engines have become powerful alternatives to traditional search, reshaping retrieval tasks. In e-commerce, for instance, conversational shopping agents now guide consumers to relevant products. This shift has created the need for generative engine optimization (GEO)---improving content visibility and relevance for generative engines. Despite its growing importance, current GEO practices are largely ad hoc, and their impacts remain poorly understood, especially in the e-commerce setting. We address this gap by introducing E-GEO, the first dataset built specifically for e-commerce GEO. E-GEO contains 13,747 realistic, multi-sentence consumer product queries, each paired with 10 relevant Amazon listings, capturing rich intent, constraints, preferences, and shopping contexts that existing datasets miss. Using this dataset, we conduct the first large-scale empirical study of e-commerce GEO across five frontier generative engines, seven popular LLM rewriters with a simple prompt, and fifteen hand-crafted rewriting heuristics. We further formulate GEO as a tractable optimization problem and develop a lightweight prompt meta-optimization algorithm that significantly improves over heuristic baselines. Notably, the optimized prompts reveal a stable, domain-agnostic pattern, suggesting the existence of a "universally effective" GEO strategy. Finally, we red-team the GEO system through both heuristic and optimization-based attacks and show that, under a simple in-prompt defense, gains from GEO reflect genuine content improvement rather than manipulation, anchoring GEO as a substantive, well-defined optimization problem.
Show more
Right Results, Wrong Reasons: Auditing Behavioral Reliance in Motion Forecasting
Geonyeong Park ⋅ Byounghun Park ⋅ Nayoung Kim ⋅ Kyungmin Kim ⋅ Soonmin Hwang
Motion forecasting models are mainly evaluated by trajectory-level accuracy metrics. However, these metrics do not directly evaluate which surrounding agents a model relies on for prediction—that is, agent-level reliance. We adopt leave-one-out (LOO) agent removal as a model-agnostic, output-based, linear-cost intervention. It measures how sensitive a model's output behavior is to removing a single agent. We formalize this as the LOO Reliance Audit protocol. We evaluate five architecturally distinct motion models on Argoverse 2 and the Waymo Open Motion Dataset, validate the protocol along four aspects—faithfulness, non-reducibility, stability, and convergence—and then derive three findings. Across the five architectures, agent-reliance rankings exhibit near-zero agreement (mean pairwise Spearman $\rho \approx 0.02$), consistently reproduced across both benchmarks (F1). LOO reliance is only partially aligned with human-annotated causal agents, revealing a collective blind spot: in 17.4% of scenes, all five models unanimously identify the same non-causal agent as top-1 (F2). In addition, within-model agreement between attention and LOO rankings is very low, suggesting that attention currently does not function as a reliable proxy for actual reliance (F3). Overall, current motion forecasting evaluation supports trajectory accuracy but does not support claims about what models rely on for prediction. We propose adding to existing benchmarks the two progress axes defined by the audit—how well a model's reliance aligns with human-annotated causal agents and how well attention and LOO rankings agree within the same model. Code and per-scene LOO scores are publicly released.
Show more
Formal Conjectures: An Open and Evolving Benchmark for Verified Discovery in Mathematics
Moritz Firsching ⋅ Paul Lezeau ⋅ Salvatore Mercuri ⋅ Miklós Z. Horváth ⋅ Yaël Dillies ⋅ Calle Sönne ⋅ Eric Wieser ⋅ Fred Zhang ⋅ Thomas Hubert ⋅ Blaise Aguera y Arcas ⋅ Pushmeet Kohli
As automated reasoning systems advance rapidly, there is a growing need for research-level formal mathematical problems to accurately evaluate their capabilities. To address this, we present Formal Conjectures, an evolving benchmark of currently 2615 mathematical problem statements formalized in Lean 4. Sourced from areas of active mathematical research, the dataset features 1029 open research conjectures providing a zero-contamination benchmark for mathematical proof discovery, and 836 solved problems for proof autoformalization. Notably, the repository provides a structured interface connecting mathematicians who formalize and clarify problems with the AI systems and humans attempting to solve them. Demonstrating its immediate utility, the benchmark has already been leveraged to make new mathematical discoveries, including the resolution of open research conjectures. We describe our approach to ensuring the correctness of these formalizations in a collaborative open-source project where contributions stem from an active community. In this framework, AI-generated proofs and disproofs serve as a valuable auditing mechanism to iteratively improve the fidelity of the benchmark. Finally, we provide a standardized evaluation setup and report baseline results on frozen evaluation subsets, demonstrating a climbable signal that measures the current frontier of automated reasoning on research-level mathematics.
Show more
NEST: Nascent Encoded Steganographic Thoughts
Artem Karpov
Monitoring chain-of-thought (CoT) reasoning is a foundational safety technique for large language model (LLM) agents; however, this oversight is compromised if models learn to conceal their reasoning. We explore steganographic CoT---where models hide secret reasoning within innocuous text---to inform risk assessment and deployment policies. Steganographic reasoning requires two skills in a single forward pass: computing an intermediate result, and embedding it into a coherent cover that answers an unrelated question. Drawing on our taxonomy of steganographic and non-steganographic CoT types, we systematically evaluate the limits of prompt-elicited steganographic CoT capability across 34 models, ranging from past generations to the current frontier. We measure monitor evasion, refusal rates, encoding fidelity, and hidden task accuracy across five datasets, comparing against plain reasoning, direct answer, and filler-token baselines. The two experiments isolate the two sub-skills: a reasoning tasks sweep tests joint reason-and-embed, while a counting task hands the model a known numerical sequence and tests embedding alone---a necessary precondition for stego reasoning. Current frontier models cannot sustain joint reason-and-embed: a paired McNemar comparison shows the steganographic channel is dominated by an filler-token baseline on every (model, family) cell. The encoding-only floor, by contrast, is cleared---Claude Opus~4.5 reaches 92\% per-number partial accuracy (54\% exact-match) on 4-digit sequences and saturates at 100\% exact-match on length-8 single-digit sequences---establishing that the binding constraint on stego CoT is the joint reasoning-plus-encoding load, not raw channel capacity. Separately, GPT-5.2 issues an explicit refusal to the steganographic instruction yet still produces partial valid encoding in 6 of 644 trials. Our findings underscore the need for continuous evaluation of steganographic risk and provide a methodology to preemptively detect and evaluate hidden reasoning that might empower misaligned scheming and deceptive behavior.
Show more
Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
Qingyun Zou ⋅ Feng Yu ⋅ Hongshi Tan ⋅ Jiahao Cui ⋅ Bingsheng He ⋅ Weng-Fai Wong
We ask whether agentic AI systems built for software engineering transfer to realistic hardware engineering. Existing hardware LLM benchmarks isolate sub-tasks but none jointly requires repository navigation, hierarchy-aware localization, Electronic Design Automation (EDA) executable verification, and maintenance-style patching. We introduce **Phoenix-bench**, a synchronized corpus of 511 verified Verilator instances from 114 GitHub repositories, each shipped with the developer patch, design-flow labels, fail-to-pass and pass-to-pass testbenches, and a Docker-pinned EDA environment so resolved-rate differences reflect agent behavior rather than toolchain availability. Using Phoenix-bench we run a uniform evaluation of four commerical agents and eight open-source agentic structures across four LLM backbones, plus two diagnostic interventions (file-level oracle localization and one round of testbench-log feedback). Three findings emerge. **Software and hardware are fundamentally different engineering tasks:** the same agent loses 37\% to 58\% from SWE-bench Verified to Phoenix-bench because hardware bugs propagate across parallel instantiated modules through signal flow rather than along a software-style call graph, and software-tuned agents stop at the symptom file instead of tracing back through the instantiation chain. **Failures concentrate on design control-flow / Finite State Machine bugs, verification testbench bugs, and hard cases** that demand cross-hierarchy signal-flow tracking and coordinated multi-file edits. **Localization granularity matters far more than localization itself:** a perfect file-level oracle yields only $+1.4$\% because the agent then breaks files that did not need editing, while a single round of test case feedback lifts resolved rate by $42$\% to $45$\% because the test case tells *where* the bug is and *what* the fix has to look like.
Show more
Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty
Sarina Penquitt ⋅ Jonathan Klees ⋅ Antonia van Betteray ⋅ Parssa Jashnieh ⋅ Peter Stehr ⋅ Matthias Rottmann ⋅ Lars Schmarje
While object detection has advanced through improved architectures and open-vocabulary models, we provide strong evidence that benchmark quality is limited by annotation incompleteness. Across four widely used datasets (COCO, Pascal VOC, Cityscapes, KITTI), re-annotation reveals substantial increases in annotated objects (e.g., up to +60% on KITTI and +40% on COCO), driven primarily by previously unlabeled small, occluded, or densely packed instances. While some differences arise from dataset-specific annotation conventions, we consistently find that missing annotations are the main source of label errors across all datasets. To achieve high data quality, we introduce a scalable annotation pipeline that emphasizes high recall and captures ambiguity through soft labels aggregated from at least 11 annotators per object. The resulting annotations improve coverage and align well with human calibration. We show that benchmark performance is highly sensitive to annotation quality, although model rankings remain largely stable. We introduce two large-scale benchmarks: (i) an uncertainty-aware object detection benchmark, and (ii) a label error detection benchmark grounded in real label errors. We show that current detectors are strongly depended on annotation quality and are misaligned with human perception. Current label error detection methods, which have been shown to perform well on synthetic noise, struggle to achieve high recall and precision on real label errors. Our results highlight the need for future object detection benchmarks to move beyond deterministic annotations toward high-recall, uncertainty-aware evaluation that maximizes valid instances and better reflects real-world ambiguity.
Show more
MM-OptBench: A Solver-Grounded Benchmark for Multimodal Optimization Modeling
Zhong Li ⋅ Qi Huang ⋅ Yuxuan Zhu ⋅ Mohammad Mohammadi Amiri ⋅ Niki van Stein ⋅ Thomas Bäck ⋅ Matthijs van Leeuwen ⋅ Zaiwen Wen ⋅ Lincen Yang
Optimization modeling translates real decision-making problems into mathematical optimization models and solver-executable implementations. Although language models are increasingly used to generate optimization formulations and solver code, existing benchmarks for this capability are almost entirely text-only. This omits many optimization-modeling tasks that arise in operational practice, where requirements are described in text but essential instance information is often conveyed through visual artifacts such as tables, graphs, maps, schedules, and planning dashboards. We introduce \emph{multimodal optimization modeling}, a benchmark setting in which models must construct both a mathematical formulation and executable solver code from a text-and-visual problem specification. To evaluate this setting, we develop a solver-grounded framework that generates structured optimization instances, verifies each instance with an exact solver, and builds both the model-facing inputs and the hidden reference files from the same verified source. We instantiate the framework as MM-OptBench, a benchmark of 780 solver-verified instances spanning 6 optimization families, 26 subcategories, and 3 structural difficulty levels. We then evaluate 9 multimodal large language models (MLLMs), including 6 frontier general-purpose models and 3 math-specialized models, with aggregate, family-level, difficulty-level, and failure-mode analyses. The results show that the task remains far from solved: the best two models reach 52.1\% and 51.3\% pass@1, while on average across the six general-purpose MLLMs, pass@1 is 43.4\% on easy instances and 15.9\% on hard instances. All three math-specialized MLLMs solve 0/780 instances. Failure attribution shows that errors arise both when extracting instance data from text and visuals and when turning extracted data into solver-correct formulations and code. MM-OptBench provides a concrete testbed for solver-grounded, decision-oriented multimodal intelligence.
Show more
Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows
Zixin (Steven) CHEN ⋅ Peng Liu ⋅ Rui SHENG ⋅ Haobo Li ⋅ Jianhong Tu ⋅ Deng Xiaodong ⋅ KaShun SHUM ⋅ Dayiheng Liu ⋅ Huamin Qu
Language agents are increasingly deployed in complex professional workflows, with tutoring emerging as a particularly high-stakes capability that remains largely unmeasured in existing benchmarks. Effective tutor agents require more than producing correct answers or executing accurate tool calls: a robust tutor must diagnose learner state, adapt support over time, make pedagogically justified decisions grounded in educational evidence, and execute interventions within realistic learning-management systems. We introduce EduAgentBench, a source-grounded benchmark for holistically evaluating tutor agents across the full scope of teaching work. It contains 150 quality-controlled tasks across three capability surfaces: situated multi-turn tutoring, professional pedagogical judgment, and Canvas-style teaching workflow completion. Tasks are constructed through a pedagogical-insight-driven pipeline and evaluated with complementary verification signals and human review. Across a comprehensive evaluation of frontier models, our findings reveal that current models are generally capable of bounded pedagogical judgment, but still fall short of professional teaching standards in situated tutoring and autonomous teaching-workflow execution. To our knowledge, EduAgentBench is the first theory-grounded and realistic benchmark for evaluating the holistic teaching capability of tutor agents, providing a measurement foundation for developing future tutor agents that can support realistic teaching work.
Show more
LoopNav: Benchmarking Spatial Consistency in World Models
Kewei Lian ⋅ Shaofei Cai ⋅ Yitao Liang ⋅ Anji Liu
The ability to simulate the world in a spatially consistent manner is a crucial requirement for effective world models. Such a model enables high-quality visual generation, and also ensures the reliability of world models for downstream tasks such as simulation and planning. It must not only retain long-horizon observational information, but also enables the construction of explicit or implicit internal spatial representations. However, existing datasets do not explicitly enforce spatial consistency constraints, limiting both the ability to systematically evaluate this capability and to learn it through data-driven approaches. Furthermore, most existing benchmarks primarily emphasize visual coherence or generation quality, neglecting the requirement of long-range spatial consistency. To bridge this gap, we propose loopnav, a dataset and corresponding benchmark centered on loop-based navigation for evaluating spatial consistency. The dataset comprises 250 hours (20 million frames) of loop-based navigation videos with actions, collected from diverse locations in the open-world environment of Minecraft. We further introduce a Scene Graph Consistency Score to quantify spatial consistency while remaining invariant to pixel-level variations. Dataset, benchmark, and code are open-sourced to support future research.
Show more
The Illusion of Multi-Agent Advantage
Prathyusha Jwalapuram ⋅ Hehai Lin ⋅ Chuyuan Li ⋅ Fangkai Jiao ⋅ Sudong Wang ⋅ Yifei Ming ⋅ Zixuan Ke ⋅ Chengwei Qin ⋅ Giuseppe Carenini ⋅ Shafiq Joty
Prevailing wisdom posits that Multi-Agent Systems (MAS) are superior to Single-Agent Systems (SAS), citing advantages like context protection, parallel processing and distributed decision-making. However, empirical support for this claim relies primarily on comparisons with SAS baselines using benchmarks that prioritize isolated reasoning tasks, which do not adequately assess these advantages. Focusing on automatically generated MAS that are designed for enhanced generalizability over manually-designed counterparts, we perform a rigorous, systematic evaluation against SAS, specifically Chain-of-Thought with Self-Consistency (CoT-SC). Across traditional reasoning datasets and tasks with interactive multi-step workflows (e.g., BrowseComp-Plus), we demonstrate that automatic MAS consistently underperform CoT-SC despite being up to 10x more expensive. To isolate these failures from limitations inherent to task structure, we introduce a diagnostic synthetic dataset tailored for MAS featuring explicit task decomposition, context separation and parallelization potential. We show that expert-architected MAS consistently outperforms automatically generated architectures in both raw performance and cost-efficiency on this dataset, demonstrating that existing evaluation frameworks mask critical architectural gaps and inefficiencies of complex MAS by failing to account for the marginal utility of increased computational cost. Critically, systematic deconstruction of the generated MAS architectures reveals that current automated design paradigms produce architectural bloat that prioritizes superficial complexity which does not translate into functional utility, exposing a fundamental misalignment with multi-agent principles.
Show more
AQBENCH: Benchmarking Neural Surrogates for Air Quality Forecasting
Siddharthan Dileep ⋅ Sanchit Bedi ⋅ Pareshbhai D Parmar ⋅ Ayush Maheshwari ⋅ Sri H Kota ⋅ N M Anoop Krishnan
Neural surrogate benchmarks for spatiotemporal PDEs are dominated by idealized test beds with periodic boundaries and synthetic dynamics. Architecture rankings established on these benchmarks do not transfer to real atmospheric chemistry, and standard regression metrics conceal the failure modes that determine operational utility. AQBench evaluates ten architectures across five model families on high-resolution WRF-Chem simulations of $PM_2.5$, $NO_2$, and $CO$ over the Indian subcontinent, against five operationally-grounded objectives: long-horizon stability, exceedance detection, extreme-episode bias, advection-diffusion residuals, and multi-pollutant forecasting. Three findings emerge. Spectral neural operators, the strongest family on canonical PDE benchmarks, rank in the bottom four on 168 hour RMSE here. Models that lead on aggregate regression underestimate during extreme pollution episodes and miss the majority of true exceedance events. Joint training on $PM_2.5$, $NO_2$, and $CO$ improves $PM_2.5$ accuracy across most architectures with no parameter increase, with Transolver gaining 21\% at 168th hour. The dataset, evaluation framework, and trained baselines are released.
Show more
MDPBench: A Benchmark for Multilingual Document Parsing in Real-World Scenarios
Zhang Li ⋅ Zhibo Lin ⋅ Qiang Liu ⋅ Ziyang Zhang ⋅ Shuo Zhang ⋅ Zidun Guo ⋅ Jiajun Song ⋅ Jiarui Zhang ⋅ Xiang Bai ⋅ Yuliang Liu
We introduce Multilingual Document Parsing Benchmark, the first benchmark for multilingual digital and photographed document parsing. Document parsing has made remarkable strides, yet almost exclusively on clean, digital, well-formatted pages in a handful of dominant languages. No systematic benchmark exists to evaluate how models perform on digital and photographed documents across diverse scripts and low-resource languages. MDPBench comprises 3,400 document images spanning 17 languages, diverse scripts, and varied photographic conditions, with high-quality annotations produced through a rigorous pipeline of expert model labeling, manual correction, and human verification. To ensure fair comparison and prevent data leakage, we maintain separate public and private evaluation splits. Our comprehensive evaluation of both open-source and closed-source models uncovers a striking finding: while closed-source models (notably Gemini3-Pro) prove relatively robust, open-source alternatives suffer dramatic performance collapse, particularly on non-Latin scripts and real-world photographed documents, with an average drop of 17.8% on photographed documents and 14.0% on non-Latin scripts. These results reveal significant performance imbalances across languages and conditions, and point to concrete directions for building more inclusive, deployment-ready parsing systems.
Show more
RA-ClipScore: Making Generative Model Evaluation More Interpretable
Yifan Lu ⋅ Taras Kucherenko ⋅ Hedvig Kjellstrom ⋅ Judith Bütepage
Generative models can produce images nearly indistinguishable from real data, yet rigorous and interpretable evaluation remains challenging. Conventional metrics such as FID provide only scalar scores with limited diagnostic insight. Widely adopted CLIP-based metrics enable semantic evaluation beyond simple training class labels, but inherit limitations from CLIP’s training paradigm that restrict attribute-wise analysis. We propose RA-CLIPScore, a novel metric that mitigates these issues and extends CLIP-based evaluation to spatial distribution alignment, measuring whether generated objects adhere to the positional priors found in the training data. RA-CLIPScore introduces dual prompts to decouple competing attributes and leverages local patch tokens to capture fine-grained regional semantics. We evaluate image generative models on their ability to match both attribute and spatial distributions of the training data. Extensive experiments show that RA-CLIPScore provides more robust and interpretable evaluations than prior methods, particularly under distribution misalignment or partially irrelevant textual attributes. We further demonstrate how it reveals previously undocumented regional tendencies in generative models.
Show more
KnowVis: A Dual-View Benchmark for Diagnosing World-Knowledge Grounding in Text-to-Image Models
Yonghang Yu ⋅ Ruoyu Wang ⋅ Haoyu Zhang ⋅ Lin Qi ⋅ Yawei Zhou ⋅ Changqian Yu
Text-to-image (T2I) models have made substantial progress in visual realism, aesthetic quality, and instruction following. However, real-world prompts often go beyond explicit visual descriptions and require implicit facts, structured knowledge, and domain-specific commonsense. Existing evaluations mainly focus on explicit prompt-to-image semantic alignment or specific knowledge subdomains, lacking a unified and diagnostic framework for evaluating world-knowledge grounding in T2I models. We introduce KnowVis, a dual-view diagnostic benchmark for world-knowledge grounding in T2I generation. KnowVis is built on a hierarchical knowledge taxonomy with case-level annotations, and contains two complementary views: a Skill-Tree Benchmark for fine-grained atomic knowledge evaluation and a Generalized Benchmark for open-ended knowledge composition and expression. We further design a structured MLLM-based VQA judge protocol and validate its reliability on the Skill-Tree validation split, enabling scalable evaluation and diagnostic error attribution. The protocol uses Required VQAs to assess atomic knowledge correctness in the Skill-Tree Benchmark, and combines Required, Optional, and dynamically discovered knowledge expressions with human-judge collaborative evaluation in the Generalized Benchmark. Experiments show that KnowVis reveals notable gaps in world-knowledge grounding across existing T2I models and provides structured diagnostic signals for knowledge-intensive image generation.
Show more
Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence
Ali J Alrasheed ⋅ Aryan Y Parast ⋅ Basim Azam ⋅ James Bailey ⋅ NAVEED AKHTAR
Self-supervised video models are increasingly framed as world models, yet their evaluation remains largely confined to a single top-1 accuracy score on clean benchmarks. This leaves a major gap in comprehending their potential as world models. We present the first systematic study addressing this gap, analyzing four matched-capacity frontier video foundation models, V-JEPA 2.1, V-JEPA 2, VideoPrism, and VideoMAEv2, across five robustness axes relevant to their deployment as video world models: feature discriminability, corruption robustness, fine-grained discrimination, occlusion robustness, and sensitivity to temporal direction. Our evaluations establish that across all five axes, latent-prediction models form a distinct and consistent profile. They degrade more gracefully under pixel corruption, preserve usable class structure rather than mere geometric stability under occlusion, capture fine-grained physical contact cues without reconstructing pixels, and uniquely encode the arrow of time. These advantages can even survive task adaptation: a frozen V-JEPA 2 backbone with a lightweight attentive probe outperforms a fully fine-tuned VideoMAE and a supervised TimeSformer on corruption and occlusion robustness. Our extensive results offer concrete new evidence in favor of latent prediction for robust world modeling.
Show more
ALTER: An Allen's Algebra-Based Evaluation Framework for Temporal Reasoning of LLMs
Hongkai Zhou ⋅ Yanhui Li
Despite the critical role of temporal reasoning in Large Language Models (LLMs), current synthetic benchmarks fall short: they either introduce information leakage or, upon addressing this issue, merely assess questions with explicit time points, failing to rigorously evaluate implicit temporal reasoning capabilities. To bridge this gap, we propose a formally grounded evaluation framework, ALTER, based on Allen’s Interval Algebra. By combining basic relations into complex networks and converting these structures into natural language forms through an automated pipeline, we construct a validated benchmark of 2,500 question-answer pairs, ensuring a controllable and scalable evaluation environment. We conduct experiments on a representative selection of LLMs. Empirical evaluations on diverse LLMs yield three key findings: (1) identifying performance decline of LLMs in highly complex scenarios involving dense events; (2) isolating complex logical interactions—rather than linguistic comprehension—as an unresolved bottleneck; and (3) revealing a critical evaluation bias induced by narrative chronology, with LLMs exploiting textual shortcuts. In summary, these findings reveal the blind spots of current evaluation paradigms, providing a rigorous foundation for the robust evaluation of complex temporal reasoning.
Show more
MosaicMRI: A Diverse Dataset and Benchmark for Raw Musculoskeletal MRI
Paula Arguello ⋅ Berk Tinaz ⋅ Mohammad Shahab Sepehri ⋅ Maryam Soltanolkotabi ⋅ Mahdi Soltanolkotabi
Deep learning underpins a wide range of applications in MRI, including reconstruction, artifact removal, and segmentation. However, progress has been driven largely by public datasets focused on brain and knee imaging, shaping how models are trained and evaluated. As a result, careful studies of the reliability of these models across diverse anatomical settings remain limited. In this work, we introduce MosaicMRI, a large and diverse collection of fully sampled raw musculoskeletal (MSK) MR measurements designed for training and evaluating machine-learning--based methods. MosaicMRI is the largest open-source raw MSK MRI dataset to date, comprising 2,725 volumes and 81,570 slices, including a 1.5T core dataset and a 0.55T low-field set. The dataset offers substantial diversity in volume orientation (e.g., axial, sagittal), imaging contrasts (e.g., PD, T1, T2), anatomies (e.g., spine, knee, hip, ankle, and others), numbers of acquisition coils, and field strengths. Using accelerated reconstruction as a testbed, we study scaling, robustness, and data selection under realistic MSK distribution shifts. Across E2E-VarNet, U-Net, and ViT baselines, mixed-anatomy training improves over anatomy-specific training, showing that the benefit of anatomical diversity is not architecture-specific. Zero-shot experiments on the 0.55~T low-field set highlight that using more training data improves average reconstruction under field-strength shift. Controlled subset and leave-one-anatomy-out experiments further suggest that these gains come from transferable cross-anatomy structure, rather than anatomy or contrast matching alone.
Show more
EnvFaultBench: Benchmarking LLM Agents on Software Environment-Fault Troubleshooting
Siqi Zhong ⋅ Haiyang Shen ⋅ Mugeng Liu ⋅ Chongyang Pan ⋅ Yun Ma
Large language model (LLM) coding agents are increasingly used as general-purpose software engineering assistants, yet existing benchmarks focus on source-code patches or environment setup from scratch. A common class of failures remains unaddressed: the code is correct but the environment is faulty due to dependency conflicts, configuration drift, stale caches, or resource contention. We present EnvFaultBench, to the best of our knowledge the first benchmark targeting environment-fault troubleshooting. It contains 348 Dockerized instances derived from real GitHub issues across 75 open-source projects in three ecosystems (Python, TypeScript/JavaScript, JVM), covering 23 fault types in three root-cause layers. Hidden functional oracles accept any command sequence that restores target behavior. Evaluating 10 LLMs (4 proprietary, 6 open-weight) under a unified agent framework, we find that the best model (GPT-5.5) resolves 65.8% while a zero-shot baseline resolves only 10.9%. The gap between strong and weak models is strongly associated with diagnostic efficiency under our bounded protocol: stronger models commit to a repair within 2–3 steps (Spearman ρ = −0.85 between first-fix latency and solve rate), while weaker models exhaust the budget on undirected exploration. Data and code are publicly available.
Show more
AuxGeoAgent: Synthesizing Challenging Geometry Proving Data via Planning and Symbolic Deduction
Junyi Liu ⋅ Jia Guo ⋅ Stanley Kok ⋅ Zihao Wang ⋅ zujie wen ⋅ Zhiqiang Zhang
Olympiad-level geometry reasoning remains challenging for large language models, primarily because existing datasets contain limited hard problems requiring long-horizon deduction and hidden auxiliary constructions. To address this gap, we present AuxGeoAgent, an agentic symbolic synthesis framework that formulates problem generation as planning over symbolic construction. Within this framework, an LLM-guided agent iteratively explores construction steps under symbolic constraints and mathematical priors, enabling the generation of structurally challenging problems that require auxiliary constructions. Leveraging this framework, we construct AuxGeo-18K, a compact yet challenging dataset of approximately 18K synthetic geometry proving problems, each accompanied by natural language and symbolic formulations, auxiliary constructions, and complete proof solutions. Unlike prior synthetic corpora that rely on massive data scaling, our approach focuses on the targeted synthesis of difficult instances while using orders of magnitude less data. Experiments on IMO-30 and HAGeo-409 show comparable performance to methods trained on million- and hundred-million-scale synthetic datasets, highlighting the strong data efficiency of our approach. Our work provides a valuable new resource for Olympiad-level geometric reasoning and advances research in mathematical reasoning.
Show more
OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories
Yibing Liu ⋅ Yangze Liu ⋅ Xiao-Long Yin ⋅ Bin Wang ⋅ Chong Zhang ⋅ Hao Yin ⋅ Zhongyi Han
Task success can hide process anomalies in real-world agent executions. An agent may pass the final task oracle while still accumulating unresolved ambiguity, unsafe external writes, ignored errors, weakly grounded commitments, or capability-boundary overcommitment. We study this mismatch as the Outcome-Process Gap and introduce OpenClawBench, a large-scale dataset for measuring and supervising process-side anomalies in real agent execution processes. OpenClawBench is built from BFCL-driven OpenClaw sessions produced by 6 source models and contains 31,264 annotated trajectories. It aligns task-oracle outcomes with structured process evidence. FullTax converts the aligned trajectories into structured anomaly supervision: binary labels, supporting evidence, onset/span localization, severity, recoverability, and a 5-class anomaly taxonomy. Using OpenClawBench, we make the Outcome-Process Gap measurable. Among 31,135 oracle-passing executions, 2,904 (9.33%) are still labeled process-anomalous under FullTax. Within oracle-passing executions that contain high-risk process evidence, 1,765 of 1,904 (92.70%) are labeled anomalous. These results show that success-only evaluation misses a concrete class of process-side failures in real agent executions. FullTax silver labels match human audit on 96.0% of a 300-trajectory human-audited pilot. A LoRA-fine-tuned Gemma 3 12B detector trained on the high-confidence FullTax supervised pool reaches binary F1=0.729 on the cleaner-labels held-out test split (n=2,646). It outperforms the GPT-5.4 frontier reference by +0.302 absolute, the no-fine-tuning base by +0.357, and wins on all six source-model agent slices. The gain comes from calibration rather than higher recall: zero-shot detectors over-flag at 42–50% against a 14.7% label rate, while the fine-tuned detector predicts anomalies 17.7% of the time. Together, OpenClawBench turns real agent execution logs into auditable and reusable supervision for studying, diagnosing, and operationally monitoring runtime agent reliability.
Show more
DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models
Wang Juntong ⋅ Wang Jiarui ⋅ Huiyu Duan ⋅ Lewei Li ⋅ Guangtao Zhai ⋅ Xiongkuo Min
Existing text-to-image (T2I) benchmarks largely rely on fixed prompt sets, leaving them vulnerable to overfitting and benchmark contamination once publicly released and repeatedly reused. In this work, we propose DynT2I-Eval, a fully automated dynamic evaluation framework for T2I models. It constructs a structured visual semantic space from long-form descriptions, decomposing prompts into controllable dimensions (e.g., subject, logical constraint, environment, and composition). This enables the continuous generation of fresh prompts via task-specific spaces and difficulty-aware sampling. DynT2I-Eval evaluates model performance across text alignment, perceptual quality, and aesthetics. Heterogeneous outputs are unified into prompt-conditioned pairwise comparisons, allowing a dynamic scheduler, micro-batch aggregation, and weighted Bayesian updates to maintain a stable online leaderboard despite changing prompt distributions and model injection. Experiments with independently sampled prompt streams demonstrate that continually refreshed prompts provide a robust evaluation protocol, reducing the impact of prompt-set-specific tuning. Simulations and ablations further confirm that the proposed ranking framework achieves a strong balance among cold-start convergence, late-entry discovery, and long-run ranking fidelity. An anonymous codebase is provided for review, which will be fully open-sourced upon publication.
Show more
CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition
Jiaming Zhou ⋅ Yujie Guo ⋅ Shiwan Zhao ⋅ Haoqin Sun ⋅ Hui Wang ⋅ Jiabei He ⋅ Aobo Kong ⋅ Shiyao Wang ⋅ Yong Qin
Code-switching (CS), the alternation between two or more languages within a single conversation, presents significant challenges for automatic speech recognition (ASR) systems. Existing Mandarin-English code-switching datasets often suffer from limitations in size, spontaneity, and the lack of full-length dialogue recordings with transcriptions, hindering the development of robust ASR models for real-world conversational scenarios. This paper introduces CS-Dialogue, a novel large-scale Mandarin-English code-switching speech dataset comprising 104 hours of spontaneous conversations from 200 speakers. Unlike previous datasets, CS-Dialogue provides full-length dialogue recordings with complete transcriptions, capturing naturalistic code-switching patterns in continuous speech. We describe the data collection and annotation processes, present detailed statistics of the dataset, and establish benchmark ASR performance using state-of-the-art models. Our experiments, using Transformer, Conformer, and Branchformer, demonstrate the challenges of code-switching ASR, and show that existing pre-trained models such as Whisper still have the space to improve. The CS-Dialogue dataset will be made freely available for all academic purposes.
Show more
MMLongCite: A Benchmark for Evaluating Faithfulness of Long-Context Vision-Language Models
Keyan Zhou ⋅ Zecheng Tang ⋅ Lingfeng Ming ⋅ Qiguang Chen ⋅ WangJie You ⋅ Guanghao Zhou ⋅ Dan Qiao ⋅ Zheming Yang ⋅ Libo Qin ⋅ Minghui Qiu ⋅ Juntao Li ⋅ Min zhang
The rapid advancement of long-context vision language models (LCVLMs) has led to a significant expansion of their context windows. However, an extended context window does not guarantee the effective utilization of the context, posing a critical challenge for real-world applications. Current evaluations of such long-context faithfulness in multimodal settings remain limited to short contexts. To bridge this gap, we introduce MMLongCite, the first benchmark evaluating the faithfulness of LCVLMs via multimodal citation generation. MMLongCite features 2,280 examples across 8 tasks and diverse modalities (image, video, interleaved), with context lengths scaled from 16K to 128K tokens. To test spatial localization capabilities of LCVLMs, we also introduce MMLongCite-HR, evaluating fine-grained visual grounding amidst dense pixel spaces. Through extensive benchmarking of cutting-edge LCVLMs, we provide a systematic analysis of current multimodal citation capabilities. Our results reveal a significant discrepancy between answer correctness and citation faithfulness. We also conduct attention pattern investigations and in-depth error analyses to reveal the underlying phenomena of failures in LCVLMs. MMLongCite establishes a rigorous foundation for diagnosing and advancing the faithfulness of LCVLMs. We hope our findings provide meaningful insights to drive further improvements in the long-context capabilities of LCVLMs.
Show more
FlareReal: A Real-Captured Paired Dataset for Nighttime Lens Flare Removal
Dafeng Zhang ⋅ Shizhuo Liu
Nighttime lens flare removal is difficult to study with supervised learning because real flare-corrupted and flare-free image pairs are hard to acquire with accurate alignment. Existing benchmarks therefore rely mainly on synthetic compositing or rendering, but these pipelines cannot fully reproduce the coupled effects of lens contamination, internal reflection, sensor saturation, automatic exposure, and diverse urban light sources. We introduce FlareReal, a real-captured benchmark for nighttime flare removal, containing 4,037 pixel-aligned flare-corrupted/flare-free pairs and 500 real pure flare images. FlareReal is collected through a controlled contamination-cleaning protocol: each scene is first photographed with physically induced lens contamination and then immediately re-captured after optical cleaning, followed by robust global alignment and manual curation. The dataset covers 241 scenes, multiple smartphone lens systems, point/linear/area light sources, single- and multi-source layouts, and challenging exposure conditions. Experiments across representative restoration architectures show that models trained with FlareReal consistently outperform models trained on Flare7K, Flare7K++, and FlareX on Flare7K, FlareX, and FlareReal test sets, and further improve generalization under cross-device and off-screen settings. These results suggest that real paired supervision captures transferable flare formation cues that are difficult to obtain from synthetic data alone.
Show more
BioXArena: Benchmarking LLM Agents on Multi-Modal Biomedical Machine Learning Tasks
Loka Li ⋅ Duzhen Zhang ⋅ Xingbo Du ⋅ Leonard Song ⋅ Zixiao Wang ⋅ Assanali Aukenov ⋅ Noel Thomas ⋅ Shakhnazar Sailaukan ⋅ Yonghan Yang ⋅ Feilong Chen ⋅ Jiahua Dong ⋅ Kun Zhang ⋅ Bin Zhang ⋅ Le Song
Large language model (LLM) agents increasingly automate machine-learning model building, yet biomedical benchmarks emphasize question answering, reasoning, and tool use or cover narrow slices of biomedical ML coding. We introduce BioXArena, a biomedical machine learning (BioML) coding benchmark for task-specific model building on heterogeneous, often multi-modal datasets. It contains 76 end-to-end tasks across 9 domains: sequence, single-cell, structure, network biology, chemical biology, perturbation dynamics, phenotype--disease, imaging, and text-integrated tasks. Each task packages primary-source data with hidden labels, held-out graders, and biology-aware metrics on a unified 0-to-100 scale; agents write runnable code, train models, and predict private test samples. BioXArena emphasizes realistic data interfaces: most tasks require agents to integrate multiple input sources, including 46 tasks that combine distinct modality families. We compare 18 harness--backbone configurations: general-purpose coding, biomedical, and ML agents plus our Minimal runner, all under the same controlled resource configuration. Codex with GPT-6 Astra achieves the highest task-level average score of 73.91 among 18 configurations, closely followed by Claude Code with Opus 5.5 (73.79). Matched-backbone comparisons show higher observed scores for mature coding systems than for Minimal; seven Minimal backends characterize model differences under a fixed runner. Beyond the main leaderboard, ablations, robustness checks, and scaling, runtime, and failure-mode analyses characterize how backbones, scaffolds, budgets, and domains affect BioML coding performance, exposing modeling and execution gaps across agent and LLM backbone families.
Show more
$\pi$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
Haoran Zhang ⋅ Luxin Xu ⋅ Zhilin Wang ⋅ Runquan Gui ⋅ Shunkai Zhang ⋅ Haodi Lei ⋅ Tong Zhu ⋅ Xiaoye Qu ⋅ Yang Yang ⋅ Yu Cheng ⋅ Yafu Li
The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $\pi$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $\pi$-Bench evaluates agents’ ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.
Show more
Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents
Zhengyang Tang ⋅ Yi Zhang ⋅ Chenxin Li ⋅ Xin Lai ⋅ Pengyuan Lyu ⋅ Yiduo Guo ⋅ Weinong Wang ⋅ Junyi Li ⋅ Yang Ding ⋅ Huawen Shen ⋅ Zhengyao Fang ⋅ Xingran Zhou ⋅ Liang Wu ⋅ Fei Tang ⋅ Sunqi Fan ⋅ Shangpin Peng ⋅ Zheng Ruan ⋅ anran zhang ⋅ Benyou Wang ⋅ Chengquan Zhang ⋅ Han Hu
When a phone-use agent avoids harm, does that show safety, or simply inability to act? Existing evaluations often cannot tell. A harmful outcome may be avoided because the agent recognized the risk and chose the safe action, or because it failed to understand the screen or execute any relevant action at all. These cases have different causes and call for different fixes, yet current benchmarks often merge them under task success, refusal, or final harmful outcome. We address this problem with *PhoneSafety*, a benchmark of 700 safety-critical moments drawn from real phone interactions across more than 130 apps. Each instance isolates the next decision at a risky moment and asks a simple question: does the model take the safe action, take the unsafe action, or fail to do anything useful? We evaluate eight representative phone-use agents under this framework. Our results reveal two main patterns. First, stronger general phone-use ability does not reliably imply safer choices at risky moments. Models that perform better on ordinary app tasks are not always the ones that behave more safely when the next action matters. Second, failures to do anything useful behave like a capability signal rather than a safety signal: they track broader phone-use performance and increase when visual input is removed. Across models, failures split into two recurring patterns: unsafe choices in settings where the model can act but chooses wrongly, and inability to act in more visually and operationally demanding screens. Overall, a harmless outcome is not enough to count as evidence of safety. Evaluating phone-use agents requires separating unsafe judgment from inability to act. Code and public data: \url{https://github.com/tangzhy/PhoneSafety}.
Show more
Learning to Synergize Textual and Visual Prompts for Fine-Grained Traffic Element Detection in HD Maps
Xiaoyang Bi ⋅ Haowen Guo ⋅ Caoshengzhe Xue ⋅ Siyuan Li ⋅ TAO XUE ⋅ Dong Liu ⋅ Mengshi Qi
High-definition (HD) map construction demands the exhaustive and precise parsing of diverse traffic elements. However, practical traffic scene perception faces two fundamental hurdles: (1) generalizing to novel traffic elements that continuously emerge during periodic map updates, and (2) synergizing fine-grained textual descriptions with visual exemplars to resolve severe visual ambiguities. Consequently, even state-of-the-art Vision-Language Models (VLMs) exhibit severe vulnerabilities when confronting these challenges, hindering their direct deployment. To bridge this gap, we formalize Multimodal Fine-grained Traffic Element Detection (MFTED), a practical task dedicated to identifying traffic targets by synergizing nuanced textual and visual prompts. We introduce MFT-150K, a large-scale benchmark featuring over 150K images and 337K instance annotations, explicitly designed to evaluate fine-grained multimodal alignment and novel category generalization. Furthermore, we provide a comprehensive benchmark of state-of-the-art models and propose an initial baseline to explore multimodal fusion strategies. Extensive experiments reveal a significant multimodal fusion gap in current VLMs when handling fine-grained traffic semantics. Even our proposed probabilistic latent-space alignment yields only limited gains, underscoring the difficulty of this task and positioning MFT-150K as a challenging benchmark for advancing intelligent transportation. Our full-scale dataset is publicly available at https://huggingface.co/datasets/MFTED/MFTED, and code will be released upon publication.
Show more
SpatialBench: Is Your Spatial Foundation Model an All-Round Player
Haosong Peng ⋅ Hao Li ⋅ jiaqi chen ⋅ Yuhao Pan ⋅ Runmao Yao ⋅ Yalun Dai ⋅ Fushuo Huo ⋅ Fangzhou Hong ⋅ Zhaoxi Chen ⋅ Haozhao Wang ⋅ Dingwen Zhang ⋅ Ziwei Liu ⋅ Wenchao Xu
While spatial foundation models have demonstrated impressive performance on standard datasets, a critical question remains: are they truly \textbf{\textit{all-round players}} capable of generalizing robustly across diverse downstream tasks, arbitrary viewpoints, shifting scene domains, varying input densities, and specific hardware constraints? Answering this overarching question requires a holistic assessment, yet current models are mainly evaluated on specific domains for which they were specifically designed or trained. Such evaluations are intrinsically limited by narrow paradigm coverage, limited scene domains, and arbitrary frame sampling, making it fundamentally difficult to assess their true generalization capabilities. To address this gap, we present SpatialBench, a cross-paradigm, domain-diverse benchmark for spatial foundation models with deterministic sampling. SpatialBench features unprecedented scale and rigorous deterministic design, comprising 19 datasets and 546 scenes across 5 diverse spatial domains. It comprehensively evaluates 41 models across 6 paradigms on 5 task suites under 4 different input density settings. Our extensive evaluation reveals that current models are not yet all-round players, and uncovers crucial insights for future advancement. Specifically, we demonstrate that full-context attention maximizes accuracy while bounded-memory strategies unlock long-sequence scalability. Moreover, our empirical evaluations in challenging embodied and egocentric tasks demonstrate that strict domain alignment and high data quality are far more critical to performance than simple dataset scaling. Furthermore, to address the largest data gap identified in our analysis, we go beyond evaluation by introducing a large-scale dataset, DA-Next-5M, alongside a strong baseline model, DA-Next, pushing the boundaries of spatial representation learning. Project page: https://ropedia.github.io/SpatialBench/.
Show more
A Diagnostic Benchmark for Layered Layout and Template-Variant Reasoning
Jaejung Seol ⋅ Haonan Zhu ⋅ Elad Hirsch ⋅ Adrienne Deganutti ⋅ Purvanshi Mehta
Graphic-design layouts are structured, multi-layered artifacts: a single composition specifies the position, type, z-order, and styling of dozens of components, and a single template typically begets a family of sibling variants that share its structural theme but differ in content, palette, or imagery -- properties that flat-canvas layout benchmarks discard. We introduce \textsc{GraphicDesignBench} (GDB), a diagnostic benchmark of $16$ tasks along two axes: \emph{spatial composition} ($8$ understanding + $4$ generation) and \emph{template variants} ($3$ understanding + $2$ generation), grounded in a new dataset of $1{,}148$ real-world templates with full component hierarchy, per-element styling, and sibling groupings. Across four frontier VLMs (GPT-5.4, Gemini-3.1-Pro / Flash-Lite, Claude-Opus-4.6) and two image generators (GPT-Image-1.5, Gemini-3.1-Flash-Image), GDB exposes concrete and dissociated gaps: component detection sits an order of magnitude below natural-image baselines, layer-order reasoning decouples from other spatial skills, partial completion collapses from single- to multi-element settings, and a simple font-Jaccard baseline matches or exceeds frontier VLMs on sibling matching and clustering. We release the dataset, tasks, metrics, and per-model outputs to track where current models are, and are not, ready to act as design collaborators.
Show more
Evaluating Uncertainty Calibration in Probabilistic Time Series Foundation Models
Thomas Decker ⋅ Volker Tresp
Time series foundation models (TSFMs) are increasingly used for probabilistic forecasting across diverse domains, yet the reliability of their uncertainty estimates remains poorly understood. In this work, we study uncertainty calibration in probabilistic TSFMs from a broader methodological and empirical perspective. We introduce a taxonomy of how TSFMs represent predictive uncertainty and develop a fine-grained evaluation framework for calibration in multi-horizon forecasting that distinguishes pooled, horizon-wise, and stronger conditional notions of calibration. Using this framework, we conduct a comprehensive empirical evaluation of popular TSFMs across diverse datasets and show that current models exhibit distinct and systematic patterns of miscalibration that standard metrics often obscure. We further analyze common post-hoc recalibration methods and characterize which forms of calibration they can improve and which tend to persist. Our results provide practical guidance for evaluating and improving uncertainty calibration in TSFMs and highlight important limitations of current practice.
Show more
ViDiC: Video Difference Captioning
Jiangtao Wu ⋅ Shihao Li ⋅ Zhaozhou Bian ⋅ Jialu Chen ⋅ Yiwen He ⋅ Runzhe Wen ⋅ An Ping ⋅ JiakaiWang ⋅ Yuanxing Zhang ⋅ Jiaheng Liu
The rapid advancement of controllable video editing and agent-driven video generation has exposed a critical bottleneck: the inability of existing Multimodal Large Language Models to accurately perceive and articulate fine-grained differences between videos. Whether diagnosing editing fidelity or detecting training data quality issues, explicit comparative perception serves as an essential infrastructure. To address this capability gap, we introduce the ViDiC task and its corresponding ViDiC-1K benchmark. ViDiC-1K comprises 1,000 curated video pairs annotated with 3,720 fine-grained comparative checklist items, rigorously structured across seven dimensions: subject, style, background, camera work, motion, position, and playback techniques. To ensure reliable evaluation and prevent model hallucination, we propose a novel dual-checklist framework that assesses the accuracy of both similarities and differences separately based on an LLM-as-a-Judge protocol. Extensive experiments on 16 representative MLLMs reveal significant limitations in their comparative reasoning and dynamic difference perception. We hope ViDiC-1K serves as a foundational benchmark to advance continuous multi-video captioning and edit awareness in multimodal intelligence.
Show more
COHE: Auditing Non-Transitivity in Sample Difficulty Proxies for Vision Models
Dayena Jeong ⋅ Sunglok Choi
Label-free proxy scores are often used as if they measured downstream discriminative difficulty. A score may rank samples similarly to a target, predict that target on held-out data, or improve a downstream decision; these are distinct evidence levels. We present Cross-Objective Hardness Evaluation (COHE), a protocol that audits each proxy--target--protocol tuple through dependence, held-out-prediction, and operational-transfer gates. COHE maps standard, inspectable statistics to bounded claim tiers; the contribution is evidence-to-claim calibration, not a new sampler, metric, or model. A DINOv2 isolation stress test illustrates the boundary: representation isolation captures a first-learning dynamics relation (rho = 0.4363, R^2 = 0.2126), but using it for direct 10% subset selection fails against Random (Delta = -15.41, 95% CI [-15.87, -14.95]). COHE therefore preserves this target-specific relation without rebranding it as actionability. Across CIFAR-10/100, ImageNet-10, and ImageNet-1K validation checks, eight primary generative proxy families do not support CE/margin surrogate claims: they show weak alignment, near-zero held-out CE prediction, and unreliable transfer. Controls show stable discriminative target rankings, expected shuffled-proxy null behavior, and a task-aligned ImageNet target-hard retrieval/triage tuple that passes all gates. The anonymized supplementary package provides lightweight gate-wise audit code, toy scalar CSV examples, cached aggregate summaries, selected derived scalar arrays for tabular gate verification, supported paper-style verification scripts, and provenance notes without redistributing raw datasets, images, weights, checkpoints, or full scoring/retraining pipelines.
Show more
EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents
Yunqi Liu ⋅ Tong Niu ⋅ Zitong Wang ⋅ Zhenlong Dai ⋅ Yuqi Qing ⋅ Weiqiang Wang ⋅ Jian liu
As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop reasoning, and dynamic interaction with users. However, existing benchmarks fail to jointly evaluate these capabilities due to challenges in designing strictly coupled multi-capability tasks, simulating natural and task-constrained user feedback, and ensuring objective evaluation of dynamic interaction. To bridge this gap, we introduce EgoBench, the first interactive multimodal benchmark for tool-using agents. EgoBench comprises 1,590 egocentric-video-grounded tasks covering five daily scenarios, along with a user-agent-tool interactive environment for evaluation. We implement a three-stage synergistic pipeline through which each task is designed to enforce the joint application of visual perception and tool-augmented multi-hop reasoning. We additionally develop a multi-agent simulated user to evaluate agents' interaction capabilities, which generates high-fidelity, task-aligned responses to agents. Furthermore, we establish a deterministic joint validation framework that guarantees objective assessment through process-based and result-based equivalence. Benchmarking eight video-MLLM agents on EgoBench reveals a severe performance ceiling: the best-performing model achieves only a 34.95% average Joint Success Rate across the three interaction modes. Finally, we conduct a multi-dimensional error analysis to disentangle failure modes, exposing capability bottlenecks for advancing future AI agents.
Show more
LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models
Daniele Cipollone ⋅ Sergey Titov ⋅ Maliheh Izadi ⋅ Egor Bogomolov ⋅ Arie v Deursen
Large software projects often depend on older versions of libraries, even as APIs continue to evolve across releases. This creates a challenge for LLMs: they must maintain knowledge of multiple API versions, not merely the latest or most common one. However, current LLMs, trained on temporally mixed corpora and lack explicit mechanisms for such version-specific reasoning, leading to anachronistic errors -- calling APIs as they exist in a different library version. To systematically evaluate this phenomenon, we introduce LibEvoBench, a multi-task benchmark spanning multiple versions of widely used Python libraries, along with a new metric, the Software Evolution Understanding Score (SEUS), to measure models' consistency when working with evolving APIs. Our results show that state-of-the-art models are largely version-oblivious: performance degrades for evolving APIs, while for stable APIs it remains the same across versions. Moreover, simply specifying the target version provides no benefit, while relevant documentation significantly boosts models' accuracy. These findings highlight a systematic limitation of current training paradigms and motivate new approaches for temporally grounded knowledge in code generation.
Show more
Beyond a Single Score: An Audit of Aesthetic Evaluation in Text-to-Image Pipelines Across Subcultural Visual Languages
Dan Li ⋅ Ling Fan ⋅ Lei Xia ⋅ Mudasir Ahmed ⋅ Yuying Pei
Text-to-image (T2I) models are increasingly used by online creative communities to generate imagery in the shared visual languages those communities have built, from Cottagecore's pastoral warmth to Cyberpunk's neon dystopias. The aesthetic predictors embedded in T2I pipelines filter training data and score generated outputs, yet whether different predictors agree on which visual languages are "high-quality" is largely untested. We present SubCulture-2.6K, a dataset of 2,610 Stable-Diffusion-generated image–prompt pairs spanning six subcultures (Cottagecore, Cyberpunk, Dark Academia, Goblincore, Synthwave, Y2K), released with dominant-color palettes, CLIP ViT-L/14 embeddings, and per-image scores from four deployed predictors (LAION-Aesthetics V2, PickScore, ImageReward, HPS-v2). Using this dataset we report four findings. First, subcultures are visually separable in the representations these predictors read from, so any cross-subcultural score difference reflects a value judgment, not a perceptual failure. Second, LAION-Aesthetics V2 imposes a large ordered preference across subcultures, with Y2K rendering retained at the standard quality cutoff at a small fraction of the rate of the most-favored subculture. Third, peer predictors share LAION-Aesthetics V2's dispreference for Y2K but disagree sharply on which subculture they reward most, and they cluster much more closely with each other than with LAION-Aesthetics V2. Fourth, an interpretable-feature regression accounts for roughly half of the LAION-Aesthetics V2 effect, leaving a sizeable residual subcultural taste. Because LAION-Aesthetics V2 sits both upstream and downstream of recent Stable Diffusion variants, this audit shows that aesthetic predictors are not interchangeable: the choice of which one to use is itself a choice about whose visual language is rewarded.
Show more
LG-Bench: A Graph-Structured Evaluation Benchmark for Life Science
Lu Sun ⋅ Xiangyi Zhang ⋅ Xiangyang Zhu ⋅ Zijian Chen ⋅ Qiang Hu ⋅ Yuan Tian ⋅ Guangtao Zhai
Traditional evaluation benchmarks reduce inherently interconnected scientific knowledge in life sciences into flat lists of questions, disregarding the underlying topological structure of the knowledge. We introduce, the first graph-structured benchmark for life sciences, featuring over 10,000 high-quality multiple-choice questions across medicine, biology, and chemistry. Our approach constructs a weighted evaluation graph using bidirectional matching and semantic similarity algorithms, where nodes represent questions and edge weights capture their semantic relationships. Leveraging this graph topology, we design two novel evaluation metrics. The Global Coherence Score (GCS) measures a model’s consistency within semantically related neighborhoods, while Knowledge Balance Score (KBS) analyzes how model errors are distributed across the graph to reveal conceptual blind spots. LG-Bench facilitates fine-grained comparison of LLMs by surfacing differences in conceptual coherence and patterns of knowledge organization across models. Our framework shifts the evaluation paradigm from flat accuracy metrics to structure-aware analysis, offering a new lens for diagnosing and improving LLM performance in the life sciences domain.
Show more
KNN Implementation Details Can Dramatically Change Performance: An Example from Cover Trees
Amol Khanna ⋅ Edward Raff
Nearest neighbor search algorithms are widely used and have a long history, but we find that key implementation details are seldom reviewed in articles about them. These details can materially impact performance, and thus change the conclusions one makes about a given algorithm's superiority. In this article, we highlight these impacts by reviewing the Cover Tree as an example, producing RustKNN, a unified Rust implementation of three construction variants of the Cover Tree with three query modes. Our analysis reveals: (1) the optimal query strategy depends on $k$ and on the dataset — single-tree search is fastest at $k{=}1$ on three of our four largest datasets, while dual-tree search is fastest at $k{\ge}5$ on the three largest — and choosing the wrong query mode can cost up to a $7.5\times$ factor in runtime; (2) the cover tree base parameter, set to 2 in theory and 1.3 in practice, is not a safe default for dual-tree search: 1.3 is optimal on only one of seven datasets, and on covtype it is $1.6\times$ slower than the optimum; (3) among five micro-optimizations, only early-exit distance computation has a consistent effect, and that effect grows with dimension (none at $d{=}32$, $1.31\times$ at $d{=}54$, and $1.49\times$ at $d{=}784$); and (4) all-nearest-neighbor and held-out benchmarks yield different implementation rankings. All main timing tables report medians over 20 independent allocations with bootstrap 95\% confidence intervals. RustKNN is open source at https://github.com/CrowdStrike/RustKNN.
Show more
AcousticBench: Measuring Acoustic Perception in Large Audio Language Models
Anirudh Rahul ⋅ Arth Vidyarthi ⋅ Michael Horgan ⋅ Suyash Jaju ⋅ Yuhao Zhou
Large Audio Language Models (LALMs) are designed to leverage non-lexical features in audio that Automatic Speech Recognition (ASR)-based cascade systems discard. Yet, existing LALM benchmarks often reward improvements in both language and audio understanding, making it difficult to isolate performance on acoustic perception. We introduce AcousticBench, a benchmark of 3,200 two-choice questions targeting non-lexical audio properties across three task families: Relative Acoustic Discrimination (RAD), Speech Emotion Recognition (SER), and Relative Speech Quality (RSQ). The benchmark draws from 5,800 unique recordings totaling 11.6 hours across three audio collection protocols: locally captured single-speaker voice-acting sessions, WebRTC multi-speaker conversations in 17 languages, and synthetic sound generation. We evaluate 15 LALMs alongside an ASR-based cascade and a held-out human reference. The cascade system performs at near-random levels across task types, confirming AcousticBench cannot be solved from transcripts alone. Gemini-3.1-Pro leads on every aggregate task but no model reaches human performance; open-weight models lag most on relative differences in loudness, dynamic range, and channel degradation. As a diagnostic, we freeze Kimi-Audio's audio encoder and fine-tune its backbone with task-specific LoRA adapters. We find that RAD lifts from 69.2% to 98.7%, RSQ from 54.8% to 98.4%, and pairwise SER from 72.3% to 92.5% on their held-out evaluation sets, with only modest gains on single-clip SER. This suggests that, for Kimi-Audio, targeted post-training can substantially close the gap to human acoustic perception even with the audio encoder held fixed.
Show more
TrustMod-SM: A Multi-Axis Benchmark for Evaluating Trustworthiness of LLMs in Social Media Content Moderation
Aizan Zafar ⋅ Sreekantam Sai Venkat ⋅ Gathik Jindal ⋅ Ritu Raj Sharma ⋅ Zishan Ahmad ⋅ Tulika Saha ⋅ Srinath Srinivasa
Large language models have shown remarkable promise for social media content moderation, with recent studies demonstrating performance rivaling human annotators on policy-compliance tasks. However, the trustworthiness of these models beyond aggregate accuracy remains largely unexamined, posing significant risks, a model with high overall accuracy may still exhibit substantial error-rate disparities across demographic groups while remaining vulnerable to adversarial manipulation. In this paper, we introduce **TrustMod-SM** and aim to comprehensively evaluate the trustworthiness of LLM-based content moderators across five dimensions: *trustfulness*, *fairness*, *safety*, *robustness*, and *context integrity*. TrustMod-SM comprises about 29K evaluation instances curated from eight established datasets, and we evaluate thirteen open-weight models (0.5B-14B parameters), including both text-only and vision-language architectures. Our analysis reveals that models consistently exhibit trustworthiness concerns, often displaying greater demographic disparity as detection accuracy improves, complying with both concealment and exaggeration inducements, assigning high confidence to incorrect predictions while rarely escalating ambiguous content for human review, and failing to distinguish hate from counter-speech and reclaimed language, particularly at sub-8B scales. Furthermore, scaling does not uniformly improve trustworthiness, and a cross-dimensional analysis reveals non-trivial associations between axes, challenging the independence assumed by prior benchmarks. We publicly release our benchmark data, evaluation code, and model predictions at https://anonymous.4open.science/r/TrustMod_SM **WARNING:** This paper contains model outputs that may be considered offensive.
Show more
CPSea2: Composing Terminal Geometries for Structurally Diverse Cyclic Peptide Binder Design
Ziyi Yang ⋅ Yinjun Jia ⋅ Guiyu Deng ⋅ Jiqing Zheng ⋅ Tianyi Zhang ⋅ Ziting Zhang ⋅ Kai Liu ⋅ Lei Liu ⋅ Yanyan Lan
Cyclic peptides are attractive next-generation therapeutics. In drug discovery, diverse ring-closing chemistries are applied to tune synthetic accessibility and conformational constraints. Current machine learning methods represent cyclization through modified positional encodings, covalent-bond prompts in all-atom models, residue-level distance guidance, or direct terminal closure by molecular mechanics (MM). These approximations provide limited control over the raw geometry, especially for non-natural linkers, and often yield unreliable structures. Here, we introduce **CPSea2**, a composable framework that abstracts cyclization as terminal-amide geometry constraints. We construct **CPSea2-Cap** (**CPcap**), a conformer library covering eight cyclization types, and **CPSea2-Base** (**CPbase**), a large-scale dataset of nearly 900 million mined linear peptide--protein complexes. Matching terminal-amides between CPcap and CPbase directly derives cyclic peptide--protein complexes with diverse terminal topologies. Built on CPbase, **PepAny** generates target-conditioned peptide binders with controllable terminal-amide geometry, expanding the accessible space beyond mined structures. Multi-target evaluations show improved raw-output plausibility over bond-prompted and distance-guided baselines. CPSea2 provides a novel framework and data foundation for flexible and diverse cyclic peptide binder design. Datasets and codes are available at https://anonymous.4open.science/r/CPSea2-6CF5/.
Show more
RealDev-QA: Trajectory-Level Diagnosis for Developer RAG Under Real-World Noises
Tianling Lan ⋅ Yao Junxiao ⋅ Tingyang Chen ⋅ Cibo Yu ⋅ Siyuan Gong ⋅ Cong Fu ⋅ Xiangyu Ke ⋅ Beng Chin Ooi
Real-world developer help-seeking is rarely a clean search query: requests mix stack traces, failed attempts, environment constraints, and mistaken hypotheses, while relevant-looking documents may be invalid under the user's specific runtime settings. We introduce RealDev-QA, a trajectory-supervised benchmark for developer RAG under diagnostic query noise and evidence conflict. RealDev-QA contains 1,233 post-June-2024 multi-hop questions derived from public developer discussions, each paired with decomposed sub-queries, dependency edges, per-step gold evidence, verified reasoning graphs, and adversarial hard negatives drawn from real documentation, release notes, and verified issues. Its construction uses answer-conditioned evidence tracing to ensure each retained instance has a verifiable evidence path; evaluation then removes the resolution and tests whether systems can recover and use that path from the diagnostic query alone. Across static, graph-based, and agentic RAG systems, comprehensive evaluations show that retrieval helps but does not solve RealDev-QA: the best human-audited accuracy reaches only 24.7\%. Trajectory diagnostics show that failures are typically decided early: over 78\% of incorrect trajectories miss gold evidence in the first retrieval step, and first-query precision predicts downstream correctness beyond aggregate evidence coverage—broad initial queries admit hard negatives that persist and pollute later evidence navigation. Low correctness decomposes into five pipeline-level failure modes, while retrieval, automatic, and human-grounded metrics yield divergent system rankings largely masked by aggregate evaluations on standard benchmarks. RealDev-QA reframes developer RAG as evidence-path control: deciding where to start, what to connect, what to reject, and how to recover.
Show more
Text-side fragility in contrastive vision-language retrieval
Yanyu Lei
Contrastive vision-language models (VLMs) are deployed in a growing range of retrieval and grounding systems, making their robustness to input corruption a critical property. In this paper, we apply controlled corruptions to textual and image inputs for five contrastive VLMs spanning three Vision Transformer architectures, four training corpora, and two contrastive loss functions. We aim to measure how image and text representations and retrieval performance respond as input damage increases. Previous cross-modal robustness work often applied ordinal severity scales to corruptions independently, even though the same severity step on the image side may not represent equivalent damage on the text side or vice versa. Our experiment addresses this measurement gap using within-modality damage percentile binning, which ranks each side's corruptions on its own damage scale and compares only at matched ranks. We evaluate all models on MS-COCO retrieval with $n = 1{,}000$ pairs over three seeds, six corruption types, and five severity levels. Our results indicate that image-to-text (i2t) Recall@1 at the highest damage quintile is 4.6$\times$ to 5.8$\times$ higher than text-to-image (t2i) Recall@1 at the matched quintile. Two mechanistic probes are used to localize part of the asymmetry to the single-token text-pooling bottleneck and to text damage that destroys lexical content rather than word order. These results suggest that downstream applications retrieving images from text are notably more fragile than the reverse, and that future improvements to cross-modal robustness require architectural attention to text readout.
Show more
fxBench: Evaluating and Understanding Formula Suggestions in Spreadsheets
Sanket Mhatre ⋅ Sumit Gulwani ⋅ Vu Le ⋅ Yasharth Bajpai ⋅ Gust Verbruggen
Spreadsheets are among the most widely used tools for computations, and formulas are their core building blocks. Besides agentic spreadsheet tools, proactive formula suggestions enable significant productivity gains and are now available in popular spreadsheet software (Microsoft Excel and Google Sheets). Such suggestions pose interesting challenges: the right context must be collected and represented for a machine learning model, which must then be able to write a correct formula, in a way that is fast and cheap enough to trigger on every $\texttt{=}$ (or subsequent keystrokes for auto-completion) without taking users out of their editing flow. Surprisingly, there is no public benchmark for formula suggestions on real spreadsheets. In this paper, we therefore (1) introduce fxbench as a manually curated benchmark of 503 formula suggestion tasks based on Sheetpedia, (2) break down the formula suggestion problem into four capabilities---gather context, understand context, understand intent, write formula---that a formula suggestion system requires, and (3) evaluate existing and new approaches to understand bottlenecks in these capabilities that pose interesting research directions. Specifically, we annotate each benchmark with the cells that describe intent to determine the theoretical performance limit of a given context selection strategy, we evaluate performance of small and large models on different representations of context---including a dense format with high theoretical coverage at low token counts---and we fine-tune smaller language models to show that we can learn to understand such dense formats. Additionally, we perform quantitative and qualitative analysis on failure modes to understand bottlenecks in the formula writing capability---the best configuration only achieved 58.8\% correct suggestions.
Show more
Can VLMs Reason When to Stop for Human Safety?
Ryo Hachiuma ⋅ Arun G Zachariah ⋅ Barnaby Simkin ⋅ Jibin Varghese ⋅ Fu-En Yang ⋅ Chi-Pin Huang ⋅ Frank Wang ⋅ Yusuke Hirota
Since Vision-Language Models (VLMs) are the core of the perceptual and reasoning capability in Vision-Language-Action models, which are deployed in environments shared with humans, evaluating VLMs' safety reasoning from video is critical for preventing human harm. Current video-based benchmarks, however, evaluate only whether a model can recognize dangers that are already visible, and overlook two capabilities that VLMs require: reasoning under hierarchical principles in which human safety takes precedence over obedience to user instructions, and anticipating harm rather than identifying it after it has occurred. We introduce VESPER, a video benchmark that targets both capabilities by scenarios in which a robot is executing a user-assigned task when a person unexpectedly moves into harm's way, requiring the model to predict the potential harm and decide whether to continue or stop. The benchmark comprises 458 videos covering matched safe and dangerous variants, built through a human-in-the-loop pipeline from diverse taxonomies. We evaluate 21 VLMs, and the results indicate that the current VLMs fail to properly decide the actions under such a scenario.
Show more
SignalBench: Comparing Dense Feedback Methods for Long-Horizon Agents
Sergio Hernández-Gutiérrez ⋅ Matteo Merler ⋅ Ilze Amanda Auzina ⋅ Joschka Strüber ⋅ Ameya Prabhu ⋅ Matthias Bethge
Dense supervision is an essential ingredient when scaling long-horizon agent training, yet we don't really know how well current methods recover the value structure they purport to estimate. A growing line of work -- intrinsic signals, self-distillation, embedding similarities -- can be unified as dense feedback methods that score intermediate states and actions. However, prior works evaluate dense supervision by measuring downstream model performance, an approach that is computationally expensive and entangles the quality of the signal with engineering choices for training. To address this gap, we propose SignalBench: a computationally cheap, training-free testbed designed to isolate and evaluate dense supervision methods for agentic systems. SignalBench allows us to comprehensively analyse the correlation of the evaluated signals with reference-policy value labels across 21 dense feedback methods, with over 1.2K evaluations across four diverse environments and six open-weight backbones. We find that simple prompting baselines consistently outperform state-of-the-art dense feedback methods, and methodological families cluster together in performance. These core findings are surprisingly robust: They hold across model sizes and families, environments, observation modality, and estimating Q-value or state-value. Overall, SignalBench offers a valuable and inexpensive test bed for dense feedback method evaluation across model and task families.
Show more
Evaluating Synthetic ECG Pretraining: When Can Patient-Free Simulators Substitute for Real ECG Data?
Naoki Nonaka ⋅ Jun Seita
ECG, with mature physiological simulators and public benchmarks, is a useful testbed for evaluating what synthetic pretraining supports. We introduce a reusable framework for ECG representation learning that compares patient-free physiological simulators, patient-derived real ECGs, and learned synthetic ECGs from real-ECG-trained generators. Holding the encoder and MAE objective fixed, we study three matched settings. Transfer-protocol analysis compares abnormal-inclusive real ECGs, normal-only real ECGs, and simulator pretraining under frozen probing and full fine-tuning on 26 abnormal-ECG tasks from PTB-XL, G12EC, and CPSC2018 using a single-lead-II pipeline. Compute-normalized PTB-XL scaling tests whether simulator pretraining matches real-data pretraining under equal processed-sample budgets. A fixed 100k-sample generator-class comparison evaluates VAE, DCGAN, SSSD-ECG diffusion variants with and without labels, and two simulators, SimECG-M and SimECG-N. The transfer analysis shows that protocol determines the supported claim: abnormal-inclusive real ECG pretraining yields more linearly separable pathology features under frozen probing, whereas full fine-tuning narrows the gap and can reverse it when the simulator corpus is sufficiently scaled. Under the fixed-100k protocol, task-wise F1 averaged over five runs ranks SimECG-N first on 22/26 tasks; no evaluated learned generator ranks first under this fixed-100k protocol; and SimECG-N obtains higher mean F1 than both diffusion variants on all 26 tasks. These results delineate where abnormal-inclusive real exposure remains beneficial and where patient-free physiological simulation can serve as an effective, protocol-bound substitute under the evaluated single-lead-II MAE pretraining and full-fine-tuning setting. We release pretraining corpora, splits, and code to support reproducible evaluation of synthetic ECG pretraining sources.
Show more
MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing
Bangbang Zhou ⋅ Hangdi Xing ⋅ Yifan Chen ⋅ Jianjun Xu ⋅ Qi Zheng ⋅ Feiyu Gao ⋅ Zhibo Yang ⋅ Shuai Bai ⋅ Ming Yan ⋅ Jieping Ye ⋅ Hongtao Xie
Document parsing converts visually rich documents into machine-readable structured representations, forming a crucial foundation for information systems. Although many benchmarks have been proposed for document parsing, they remain inadequate for realistic scenarios. Existing benchmarks either focus on specific tasks or assess only single-page, text-centric settings, making them insufficient for practical multi-page parsing. Moreover, they lack fine-grained evaluation of semantic continuity, hierarchical structure recovery, and visual content preservation. To address these gaps, we propose MPDocBench-Parse, a benchmark for multi-page document parsing in real-world applications. It contains 433 manually annotated documents with 3,246 pages, covering 15 document types in English and Chinese, with diverse layout styles, and supports document-level end-to-end evaluation. We further design a comprehensive protocol for content fidelity and logical structure, covering text, table, and formula recognition, truncated text and table merging, figure extraction, reading order, and heading hierarchy recovery. Experiments show that, while existing models perform well on basic text extraction, they still suffer clear limitations in semantic continuity integration, visual content parsing, and hierarchical structure recovery. MPDocBench-Parse provides a unified foundation for advancing document parsing toward more realistic scenarios.
Show more
ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents
Yuanzheng Zhu ⋅ Mingyu Zhao ⋅ Chinmay Savadikar ⋅ Han Li ⋅ Shuang Xie ⋅ Alberto Castelo ⋅ Tianfu Wu ⋅ Lingyun Wang
Developing and evaluating e-commerce web agents requires environments that preserve meaningful task structure while enabling controllable, reproducible, and scalable scientific comparison. Existing methodologies force a tradeoff: live storefronts provide realism but are non-stationary, difficult to inspect, and irreproducible, while hand-built sandbox benchmarks provide control but cover only a narrow range of layouts, catalogs, policies, and interaction patterns. We argue that the core bottleneck is methodological: the field lacks a scalable way to construct evaluation settings that are simultaneously realistic, diverse, controllable, inspectable, and reproducible. We introduce ShopGym, an integrated framework for realistic simulation and scalable benchmarking of e-commerce web agents. ShopGym is a framework for constructing e-commerce simulation environments and grounded benchmark tasks. Its simulation layer, ShopArena, converts one or more live seed storefronts into self-contained sandbox shops through anonymized shop specifications and a staged, validated generation process. On top of these simulated storefronts, ShopGuru synthesizes benchmark tasks across seven skill categories, grounding each task in the shop's catalog, navigation structure, policies, and interaction affordances. Together, ShopArena and ShopGuru produce self-contained, resettable, inspectable, and stable evaluation artifacts that preserve structural properties and agent-evaluation signals relevant to shopping tasks. We validate the framework through graph-based structural analysis and agent-based behavioral evaluation with 224 generated tasks across six sandbox shops: three constructed with synthetic data and three with real data. Our results show that the synthetic shops preserve key structural properties of live storefronts, with agent performance on synthetic shops positively correlated with performance on live storefronts.
Show more
AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering
Chanhee Park ⋅ Jeongho Yoon ⋅ Sungbin Han ⋅ Hyeonseok Moon ⋅ Heuiseok Lim
Agentic tasks require a large language model to interact with the world, navigating information and gathering evidence across multiple steps with restricted resources. Due to this complexity, agentic task failures arise from various sources, and pinpointing these failure causes is essential to diagnose and improve agentic systems. Existing benchmarks, however, tend to focus on a single leaderboard score, leaving the underlying failure modes opaque. To fill this gap, we introduce AgentHop, a diagnostic benchmark of 1,011 multiple-choice questions paired with a controlled seven-tool sandbox under fixed token, turn, and tool-call constraints. AgentHop reveals model vulnerabilities by dissecting a single accuracy score along four axes of agent operation: retrieval, synthesis, tool-call, and resource management. Across 19 models, we find that behavior clusters by model family, with tool-call signatures revealing distinct family fingerprints: GPT models commit early, Anthropic and GLM checkpoints verify before committing, DeepSeek and Kimi over-search, and Gemini-3 Pro stays balanced. Decomposed axes further expose within-family structure: Claude Opus 4.6 and Sonnet 4.6 land within one accuracy point yet diverge on retrieval-versus-synthesis emphasis, with Opus retrieving more and Sonnet synthesizing better. We release the full benchmark set and the harness to support diagnostic agent benchmarking.
Show more
OfficeInstruct: A Dataset for Tool-Augmented Office Agents on Long-Trajectory Generative Tasks
Weikai Xu ⋅ Zhizheng Jiang ⋅ Lushuo Jiang ⋅ Kun Huang ⋅ Yuxuan Liu ⋅ Pengzhi Gao ⋅ Wei Liu ⋅ Jian Luan ⋅ Xiaolin Hu ⋅ Bo An
The recent trend of integrating autonomous agents into operating systems, supported by the Model Context Protocol (MCP) tools and Deep Searching, has driven research on efficient assistants for daily office tasks like document writing, spreadsheet analysis, and presentation generation. However, training these agents still faces several challenges: the overemphasis on question-answering tasks at the expense of multi-turn generative tasks; the absence of a training paradigm that jointly enhances coding and tool-calling capabilities; the difficulty of evaluating tool efficiency and artifact quality in the presence of drifting user intents. To address these problems, we deploy a unified multi-turn agent framework to collect real-world user interactions, creating \textit{OfficeInstruct}: a comprehensive dataset of 199k high-quality trajectories across Doc, Spreadsheet, and PPT, with an average of 56.2 steps. To ensure data quality, we propose a graph-based evidence evaluation method that models interaction histories as directed graphs, filtering out redundant tool cycles, execution errors, and hallucinations. Experiments demonstrate that our \textit{OfficeAgent} 8B and 14B models surpass open-source baselines on in-domain and out-of-distribution benchmarks, with the 32B variant attaining a 96.19\% success rate on the newly proposed \textit{OfficeAgentBench}, surpassing proprietary models like Claude-4.5-Sonnet and Gemini-3-Pro.
Show more
How Private is Private? A Comparative Study for Face De-Identification
Hui Wei ⋅ Hao Yu ⋅ Hui Kuurila-Zhang ⋅ Guoying Zhao
Face de-identification (FDeID) has emerged as a critical privacy-preserving technology, yet its evaluation remains fundamentally fragmented. Existing protocols rely on inconsistent metrics, heterogeneous datasets, and partial annotation coverage, so methods targeting different utility dimensions, such as landmark versus expression preservation, are reported on different benchmarks under different metrics, rendering cross-method comparison infeasible. We revisit FDeID evaluation from both the data and metric perspectives. On the data side, we introduce UTILFACE, a curated, demographically balanced benchmark with high identity diversity, assembled from four large-scale face datasets through identity-aware cleaning, resolution enhancement, and stratified filtering. On the metric side, we propose HiFD, a Hierarchical Face De-identification metric that unifies identity suppression, multi-level utility preservation, and image quality under a single consistency-based paradigm: every component is computed from pretrained estimators’ outputs on the original face and its de-identified counterpart, directly quantifying how much identity is suppressed and how much downstream-perceivable utility survives. HiFD organizes facial signals into a three-level utility hierarchy spanning macro cues (L1), micro cues (L2), and imperceptible cues (L3), and aggregates the five resulting components into a single interpretable score via weighted harmonic mean, with configurable application-specific profiles. Using this unified protocol, we conduct a comprehensive comparative study spanning adversarial, GAN-based, and diffusion-based methods, surfacing trade-offs and failure modes that remain invisible under existing protocols. We will release the benchmark and evaluation toolkit to foster reproducible research in privacy-preserving face analysis (anonymous codebase: https://anonymous.4open.science/r/HiFD).
Show more
Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs
Dylan Feng ⋅ Pragya Srivastava ⋅ Anca Dragan ⋅ Cassidy Laidlaw
Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these **OOD alignment failures** by introducing a benchmark called Misalignment Out of Distribution Benchmark (**MOOD**). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model and Mahalanobis distance-based OOD detector can improve recall from 39% to 47%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem.
Show more
DIBench: Benchmarking Decision Integrity of GUI-based Mobile Agents Under Deceptive Injections
Li Hu ⋅ KANGHUA MO ⋅ Yingbin Jin ⋅ Qingqing Ye ⋅ Haibo Hu
As GUI-based mobile agents rapidly progress, rigorous safety evaluation of their autonomous decision-making in realistic app interfaces becomes increasingly critical. Existing benchmarks mainly focus on execution-level anomalies using task success or hijack rates, but fail to capture the *in-task goal deviation* risk in multi-candidate selection tasks, where the decision may be steered toward an attacker-specified target, even in violation of instruction-implied constraints (e.g., cheapest/highest-rated), without any overt execution anomalies. We present **DIBench**, a *decision integrity* benchmark for measuring this risk in mobile agents. DIBench covers 7 commercial and 3 simulated apps with 5 task types. Under a threat model restricted to non-privileged UI content, we construct 8 deceptive injection probe instantiations that can steer critical selections without overt anomalies. The benchmark includes 1,000 clean and 36,672 injected instances, with a unified protocol and integrity metrics for comparison. Experiments spanning 4 agent frameworks and 7 base models show that completion-based evaluation can overestimate agent trustworthiness and miss decision-integrity risks: deceptive injections steer selections and shift early action policies, inflating completion rates and creating a misleading illusion of safety. Common defenses, including detection, image preprocessing, and prompt reminders, yield inconsistent integrity gains. Overall, DIBench provides a unified, reproducible benchmark to quantify the risk of in-task goal deviation in mobile agents and enable comparable evaluations of safety defenses. Code is available at: [https://anonymous.4open.science/r/DIBench-5432](https://anonymous.4open.science/r/DIBench-5432).
Show more
BabyTheorist: A Benchmark for Learning to Theorize the World from Observation Alone
Doojin Baek ⋅ Junyeob Baek ⋅ Mingyu Jo ⋅ Hosung Lee ⋅ Taegu Kang ⋅ Gyubin Lee ⋅ Sungjin Ahn
How would we know whether a machine truly understands the world? We argue that one key capability is \textit{observational theory learning}: acquiring reusable theories from pure observations and applying them to new situations. Current visual reasoning benchmarks focus primarily on the latter, presenting each task as a few-shot puzzle grouped by a common rule. We introduce BabyTheorist, a benchmark for controlled training and evaluation of observational theory learning. BabyTheorist generates continual streams of before-after visual observations from hidden compositional programs; during training, learners see only the resulting transitions, with no program labels, task identities, or support sets. To support theory learning and evaluation, BabyTheorist controls the program language, a curriculum over program complexity, recurring frequent subprograms, and a continual train-test protocol with level skipping. Seven representative baselines plateau well below ceiling, with the gap widening as the curriculum advances---evidence that BabyTheorist isolates a capability current program-learning approaches do not yet address.
Show more
Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?
Luc Sträter ⋅ Hazel Doughty
Parameter-efficient fine-tuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across settings and demonstrate the importance of temporal context allocation for effective video adaptation.
Show more
Effect-Level Validation for Causal Discovery in Interactive Telemetry
Dang Van Cong Hoang ⋅ Luan Pham ⋅ Minh Nguyen
Causal discovery is increasingly used to analyze telemetry from interactive systems, but a plausible graph does not guarantee an identifiable or reliable product decision. We study this problem for one query: whether early competitive gameplay increases Day-1 retention in a deployed Role-playing game (RPG). We propose an admissibility-first framework that treats discovered graphs as structural hypotheses, keeps only graphs that identify the target effect and have adequate treatment-control overlap, and validates admissible effects through cross-algorithm stability, placebo, subsampling, and E-value diagnostics. On real telemetry, only about one third of discovery runs support the target effect after temporal and semantic constraints. Some admissible Directed Acyclic Graphs (DAGs) converge to a positive risk-difference average treatment effect (ATE), down from a raw retention gap, and the estimate survives all refutation checks. Graph-level recovery is therefore an inadequate proxy for causal reliability; discovery pipelines for decision support should be evaluated at the query and effect level.
Show more
Freeing the Law with LOCUS: A Local Ordinance Corpus for the United States
Denis Peskoff ⋅ Joe Barrow ⋅ Christopher Vu ⋅ Diag Davenport
Progress in legal AI increasingly depends on access to authoritative legal text at scale. Yet one of the most consequential layers of American law remains largely absent from existing machine-readable corpora: local ordinances. Local codes govern zoning, housing, business licensing, public health, noise, animal control, and many other domains of everyday regulation, but they are fragmented across vendor platforms designed for human browsing rather than bulk research access. We introduce LOCUS—the Local Ordinance Corpus for the United States—a comprehensive corpus and county-harmonized access layer for U.S. municipal and county ordinance codes. The raw corpus, available for release to researchers, represents nearly all publicly available municipal and county ordinance codes. The resulting raw corpus contains codes from 9,239 cities and counties. A smaller county-harmonized LOCUS access layer provides coverage for the largest 2,309 of 3,144 U.S. counties, accounting for a majority of the population. We use OCR to handle the myriad of document formats that have kept the law from being a public resource. We release the corpus with coverage metadata to support reproducibility, downstream legal AI research, and the incremental expansion of machine-readable access to local law. We train a collection of ModernBERT-based classifiers and scorers to facilitate analyzing U.S. local law among several dimensions, such as opacity and paternalism, that have not previously been studied at this scale.
Show more
ZeoBench: A Benchmark for Self-Supervised Learning on 3D Zeolite Representations
Aaron Sun ⋅ Yachan Liu ⋅ Ping Yang ⋅ Peng Bai ⋅ Subhransu Maji
Zeolites are an important class of porous crystalline materials with broad applications in gas storage, separations, and catalysis. Although hundreds of thousands of experimentally realized and hypothetical zeolite structures are known, identifying materials optimized for a target application remains challenging because property labels are expensive to obtain and available only for limited structure–molecule pairs. While prior work has demonstrated that 3D neural representations can predict zeolite properties effectively, it remains unclear which representation-learning strategies are most effective in the low-data regime relevant to materials discovery. We present ZeoBench, a benchmark for self-supervised learning on dense 3D representations of all-silica zeolites, and evaluate the resulting features on six downstream adsorption-property prediction tasks, across a range of training set sizes. Our study compares self-supervised 3D pretraining on volumetric zeolite data, adaptations of image-pretrained 2D ConvNets and vision transformers to 3D via multiview and channel-inflation strategies, and 3D ConvNets pretrained on out-of-domain volumetric datasets, alongside hand-crafted descriptors and supervised baselines. Using our pretraining and evaluation framework, we find self-supervised 3D representations outperform existing methods and achieve state-of-the-art performance across all label regimes and target molecules. Learned representations consistently outperform hand-crafted descriptors, and convolutional architectures are generally more effective than vision transformers in this setting. These results establish ZeoBench as a practical benchmark for 3D representation learning in zeolites and provide guidance for data-limited adsorption property prediction.
Show more
Distinguishing Performance From Competence in Evaluations of Humanlike Abstract Reasoning
Claas Beger ⋅ Ryan Yi ⋅ Shuhao Fu ⋅ Kaleda K Denton ⋅ Arseny Moskvichev ⋅ Sarah Tsai ⋅ Sivasankaran Rajamanickam ⋅ Melanie Mitchell
AI reasoning models have exceeded human performance on the ARC-AGI-1 benchmark, but does that mean state-of-the-art models have the underlying competence—humanlike abstract reasoning—the benchmark was designed to test? Here we investigate the abstraction abilities of AI models using the closely related but simpler ConceptARC benchmark. Our evaluations vary input modality (textual vs. visual), use of external Python tools, and reasoning effort. Beyond output accuracy, we evaluate the natural-language rules that models generate to explain their solutions, enabling us to assess whether models recognize the abstractions that ConceptARC was designed to elicit. We show that the best models’ rules are frequently based on less abstract, more domain-specific concepts, capturing intended abstractions considerably less often than humans. In the visual modality, AI models’ output accuracy drops sharply; however, our rule-level analysis reveals that a substantial share of their rules capture intended abstractions, even as the models struggle to apply these concepts to generate correct solutions. In short, we show that using performance (accuracy) alone can substantially overestimate AI competence in textual modalities while underestimating it in visual modalities—an illustration of the risk of mistaking performance for competence.
Show more
Re-evaluating Continual Learning with Few-Shot Adaptation
Amogh Inamdar ⋅ Matthew So ⋅ Vici I Milenia ⋅ Richard Zemel
Continual learning methods aim to maximize the stability and plasticity of machine learning models that are trained on a sequence of tasks. The standard measure of stability (i.e., forgetting) is the 0-shot performance of a model on previously learned tasks, and plasticity, the performance on the most recently learned task. However, 0-shot evaluation does not fully measure a model or method's ability to retain learned information or adapt quickly to new information, as it requires perfect recall across multiple tasks. In this paper, we propose *few-shot evaluation* as a more comprehensive assessment of the stability and plasticity of a continual learning system. Through few-shot evaluation with a novel metric---*per-shot plasticity*---on task sequences for continual image classification, we show that a mixture of *foresight* via the meta-learning of a short sequence of future tasks and *hindsight* via replay on previously learned tasks leads to a model that strikes an ideal balance between stability and plasticity in a continual learning setting.
Show more
PHOEBI: An Open-World Benchmark for Multi-Label Bacterial Identification in Phase-Contrast Microscopy
Aaditya Baranwal ⋅ Md Jahid Hasan ⋅ Shruti Vyas
Optical microscopy lets us watch living bacteria in real time without stains or labels, and it remains the standard instrument for identifying species across clinical, environmental and industrial microbiology. Yet real samples rarely hold a single species. They arrive as mixtures and sometimes carry an organism no model has ever seen. What happens when a model trained on familiar mixtures meets one it has never encountered? No benchmark has asked this of phase-contrast images, so we built one. **PHOEBI** brings together 120,000 phase-contrast images of 40 mixtures of six bacterial species that we cultured and imaged ourselves, and it holds whole mixtures out of training so that every test is a combination the model has never seen. The answer is sobering. Classifiers that are nearly perfect on the mixtures they know collapse on new ones, and no choice of backbone or pretraining rescues them. The features turn out to be sound and the failure lies in how the evidence within each image is pooled. A culture looks the same everywhere, and decoders built on this simple fact hold steady where large networks fail even though one of them is never trained by gradient descent. The same features go further still. They notice when a species outside the catalogue appears, and the simplest way of giving it a class of its own proves the most effective while leaving the known species almost untouched. PHOEBI turns the mixed culture, the everyday case at the bench, into a measurable target for open-world recognition.
Show more
FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics
Qiran Zou ⋅ Hou Hei Lam ⋅ Wenhao Zhao ⋅ Tingting Chen ⋅ Yiming Tang ⋅ Samson Yu ⋅ Yingtao Zhu ⋅ srinivas anumasa ⋅ Zufeng Zhang ⋅ Tianyi Zhang ⋅ Chang Liu ⋅ zhengyao jiang ⋅ Anirudh Goyal ⋅ Dianbo Liu
AI research agents accelerate ML research by automating hypothesis generation, experimentation, and empirical refinement. Existing agent strategies range from greedy hill-climbing to tree search and evolutionary optimization, yet which strategy choices drive performance remains unclear. Answering this question requires a benchmark that separates agent strategy (e.g., search topology) from execution infrastructure (e.g., code editor), so that performance differences are attributable to strategy rather than infrastructure, and that provides process-level metrics beyond final scores to analyze exploration behaviors. Existing benchmarks offer limited support. We propose FML-Bench, a benchmark of 18 fundamental ML research tasks across 10 domains that separates agent strategy from execution infrastructure and defines 12 process-level behavioral metrics. Evaluating six representative agents, we find that: (1) strategy complexity alone does not guarantee strong performance: a simple greedy hill-climber nearly matches the best-performing tree-search agent, both well above the remaining agents; (2) our analysis suggests this pattern relates to improvement opportunity structure: greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse; an adaptive agent built on this insight switches to broader exploration upon detecting improvement stagnation and outperforms the other six agents, lending initial support to this observation; and (3) process-level analysis reveals that early convergence and directionally focused exploration are significantly associated with final performance, while solution diversity and compute cost are not. Our benchmark is available at: https://anonymous.4open.science/r/Anonymous-78B6.
Show more
Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization
Felipe Maia Polo ⋅ Aida Nematzadeh ⋅ Virginia Aglietti ⋅ Adam Fisch ⋅ Isabela Albuquerque
Moving beyond evaluations that collapse performance across heterogeneous prompts toward fine-grained evaluation at the prompt level, or within relatively homogeneous subsets, is necessary to diagnose generative models' strengths and weaknesses. Such fine-grained evaluations, however, suffer from a data bottleneck: human gold-standard labels are too costly at this scale, while automated ratings are often misaligned with human judgment. To resolve this challenge, we propose a novel statistical model based on tensor factorization that merges cheap autorater data with a limited set of human gold-standard labels. Specifically, our approach uses autorater scores to pretrain latent representations of prompts and generative models, and then aligns those pretrained representations to human preferences using a small calibration set. This sample-efficient methodology is robust to autorater quality, more accurately predicts human preferences on a per-prompt basis than standard baselines, and provides tight confidence intervals for key statistical parameters of interest. We also showcase the practical utility of our method by constructing granular leaderboards based on prompt qualities and by estimating model performance solely from autorater scores, entirely eliminating the need for additional human annotations.
Show more
Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models
Yan Jiang ⋅ Ruihong Qiu ⋅ Zi Huang
Recently, reinforcement learning (RL) has been widely applied during post-training for diffusion large language models (dLLMs) to enhance reasoning with block-wise semi-autoregressive generation. Block size has therefore become a vital factor in dLLMs, since it determines the parallel decoding granularity and affects the rollout trajectories during RL optimisation, e.g., GRPO. Instead of investigating the effect of block size during inference on individual domains, this paper studies block size from a domain conflict perspective for dLLM RL post-training in multi-domain scenarios. The main contributions are: (1) a formulation of domain block size conflict in multi-domain RL for dLLMs, which will largely affect the post-training effectiveness for rollout-based RL methods; (2) a novel dataset, \textbf{Block-R1-41K} is constructed with a best-improved training block size for each sample, which also induces a Block Size Conflict Score to quantitatively measure the domain conflict; (3) a new benchmark, Block-R1, for flexible RL post-training for dLLMs in both single and cross domain; and (4) a simple yet powerful cross-domain post-training method with sample-level best-improved training block sizes. Extensive experiments on 13 distinct datasets, 7 latest RL algorithms, and various different dLLM backbones are covered in Block-R1. The benchmark is open-sourced at https://anonymous.4open.science/r/Block-R1-2026, with the dataset released at https://huggingface.co/datasets/dLLM-R1/Block-R1-41K.
Show more
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
Qingnan Ren ⋅ Shun Zou ⋅ Shiting Huang ⋅ Ziao Zhang ⋅ Kou Shi ⋅ Zhen Fang ⋅ Yiming Zhao ⋅ Yu Zeng ⋅ Qisheng Su ⋅ Lin Chen ⋅ Yong Wang ⋅ Zehui Chen ⋅ Xiangxiang Chu ⋅ Feng Zhao
As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to capture the heterogeneous environments, full-stack orchestration, and system-level complexity of real enterprise Software as a Service (SaaS) systems, leaving a critical gap in assessing agents under realistic engineering constraints. To fill this gap, we introduce SaaSBench, the first benchmark designed to explore the boundaries of AI agents in enterprise SaaS engineering. Spanning 30 complex tasks across 6 SaaS domains with 5,370 validation nodes, it incorporates 8 programming languages, 6 databases, and 13 frameworks to meticulously mirror real-world software heterogeneity. Furthermore, we design a dependency-aware hybrid evaluation paradigm tailored for complex systems with long horizons and multi-component coupling, enabling fine-grained, reproducible assessment. Crucially, our extensive experiments reveal a striking insight: the primary bottleneck for state-of-the-art agents is not generating isolated code logic, but successfully configuring and integrating a multi-component system. Over 95\% of task failures occur before agents even reach deep business logic, with models often falling victim to overconfidence and prematurely halting during foundational system setup, or getting trapped in ineffective debugging loops. We hope SaaSBench serves as a practical and challenging testbed to drive the evolution of reliable, system-level coding agents.
Show more
BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problems
Yutaro Yamada ⋅ Kei Hiroshima ⋅ Nozomu Yoshinari ⋅ Kento Uchida ⋅ Shinichi Shirakawa
Formulating an optimization problem strongly affects the quality of the final solution, yet good formulations usually require substantial expertise. Recent studies have therefore examined how to automatically derive optimization problems from natural-language descriptions, but existing benchmarks focus on settings where objectives and constraints can be written explicitly as mathematical expressions. Many practically important problems, including hyperparameter optimization in machine learning, are naturally treated as black-box optimization (BBO) problems, in which only objective values are observable, and the functional form is unavailable. In BBO, the search space design, a part of the problem formulation, and the selection of the optimization algorithm are crucial for problem-solving. Automating these processes with large language models (LLMs) is a significant challenge. This paper introduces Black-Box Optimization Word Problems (BBOWP), a novel problem setting in which a system must infer both a search space and an optimization algorithm from a natural-language description of a black-box optimization task. To support research on this setting, we establish the BBOWP Benchmark Suite (BBOWP-Bench), a dataset and evaluation framework for BBOWP. Each instance combines a natural-language problem description, an executable evaluation environment, and a human-designed baseline formulation, allowing evaluation of both search-space design and algorithm selection. The benchmark covers 23 instances from four different application domains and supports reproducible assessment through actual black-box optimization runs. Using this benchmark, we provide the first evaluation of LLMs and show that current LLMs are capable of selecting suitable algorithms based on the given evaluation budget. However, they sometimes struggle with search space design, particularly in identifying important variables and balancing their ranges when the problem description is less informative or the search space is highly problem-specific.
Show more
GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling
Kang He ⋅ Wenshuo Peng ⋅ Chuanhao Li ⋅ Zihui Gao ⋅ Xiaojie Xu ⋅ Zhengyuan Lin ⋅ Yuzhe Ding ⋅ Donghong Ji ⋅ Kaipeng Zhang ⋅ Yongtao Ge
Recent advances in world models have emphasized the need for learning temporally consistent and interactive representations of dynamic environments. However, existing video datasets fail to simultaneously support long-horizon dynamics, precise camera control, and structured semantic representations. To address this, we introduce GameVerse, a large-scale game video dataset for interactive world modeling. GameVerse provides minute-level continuous video sequences, covering diverse interaction behaviors, across both first- and third-person perspectives. The dataset is constructed through a unified data processing pipeline, including scene-consistent segmentation, quality filtering, camera pose estimation, and multi-granular semantic annotation. GameVerse consists of 11,092 videos, with a total duration of 4,692 hours (195.5 days) and approximately 986.6 million frames. Each video segment is augmented with temporally stable camera trajectories and structured semantic annotations. Comprehensive data analysis and experiments demonstrate that GameVerse exhibits strong advantages in terms of scale, diversity, and annotation quality, and shows promising performance for long-horizon modeling and interactive world understanding. We expect GameVerse to provide a unified data foundation for long-horizon modeling and interactive world modeling, and to facilitate future research in this area.
Show more
Traffic STGNNs across Sensor, City, and Time Shifts: Routing Concentration Tracks Sensitivity to Sensor Dropout
Xile Wang ⋅ Mingqi Yang
Traffic forecasting benchmarks measure accuracy within a single city and test distribution, leaving open how architectures behave when sensors fail, when a model is transferred to another city, or when weekday and weekend traffic differ. We evaluate eight STGNN architectures under these three shifts using frozen checkpoints; for transfer across cities, target graph information is withheld so that fixed and adaptive graph models have the same structural access. We find that fragility under sensor dropout follows routing concentration rather than the use of adaptive adjacency itself. Across four adaptive backbones, mean row maximum routing mass correlates with dropout degradation at Pearson $\rho=+0.979$, and the relationship replicates on a second source city. Graph WaveNet lies at the concentrated end of this spectrum, while AGCRN and STAEformer stay close to DCRNN when sensors are zeroed. The other shifts separate from this pattern: MegaCRN performs best when transferred to another city, yet MegaCRN and STAEformer show the largest weekday and weekend performance gaps. In this backbone bank, strong performance under one shift does not imply robustness under another. We release a registry of frozen checkpoints, evaluation code, and a decision table mapping each backbone to its main failure mode.
Show more
The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
Zekai Tong ⋅ Ruiyao Xu ⋅ Aryan Shrivastava ⋅ Chenhao Tan ⋅ Ari Holtzman
Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect text. In this work, we study how word-boundary corruption affects how LLMs detect targeted information. By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate. We refer to this curve as the Text Uncanny Valley. To explain such observation, we propose a mode transition hypothesis: LLMs operate in a word-level mode for near-normal text and a character-level mode for heavily fragmented text, with the valley marking the disordered transition where neither mode is effective. Four experiments and one analysis are consistent with this account: in-context learning fails to rescue valley-bottom performance; regularizing the perturbation substantially reduces the U-shape; a math reasoning task replicates the U-shape for Gemini 3.0 Flash but not for stronger models, suggesting the effect is attenuated when tasks rely less on exact lexical alignment; and tokenization entropy peaks before the F1 minimum, consistent with a regime-conflict interpretation. These findings reveal a failure mode invisible to clean-text benchmarks yet directly relevant to any deployment scenario involving noisy or uncurated text inputs.
Show more
Routeability Before Routing: Routeability Audit Protocol (RAP) and RouteabilityBench for Audited LLM Model Selection
Yihang Lu ⋅ Denica Kjorvezir ⋅ Ana Gjorgjevikj ⋅ Carola Doerr ⋅ Tome Eftimov
LLM routing aims to choose, for each prompt, which model in a portfolio is most likely to answer correctly. This only makes sense when the benchmark contains enough prompt-specific variation for different models to be best on different inputs. On coarse-scored LLM benchmarks, however, many models often tie for the best score, and the single best model may already leave little room for improvement. We therefore propose a routeability-first audit, instantiated as RAP and RouteabilityBench, a reviewer-runnable evaluation package. Across four benchmarks and 66 LLMs, generic routing enhancements such as larger embedding sets, dimensionality reduction, selective abstention, clustering, and stronger regressors do not consistently outperform the single best model. The clearest positive evidence is concentrated in portfolio-dependent Omni-MATH settings, while GPQA, MMLU-Pro, and IFEval show different ways in which routing can remain hard or predictor-sensitive. Our practical recommendation is that routing gains should be reported together with routeability diagnostics, tied-best analysis, portfolio construction details, and family-corrected statistical tests.
Show more
mRNABench: A curated benchmark for mature mRNA property and function prediction
Ruian (Ian) Shi ⋅ Taykhoom Dalal ⋅ Philip Fradkin ⋅ Divya Koyyalagunta ⋅ Simran Chhabria ⋅ Andrew Jung ⋅ Cyrus L Tam ⋅ Defne Ceyhan ⋅ Jessica Lin ⋅ Kaitlin U Laverty ⋅ Ilyes Baali ⋅ Bo Wang ⋅ Quaid Morris
Messenger RNA (mRNA) is central to gene expression, and its half-life, localization, and translation efficiency drive phenotypic diversity in eukaryotic cells. While supervised learning has been used to study the mRNA regulatory code, self-supervised foundation models support a wider range of transfer learning tasks. However, the dearth of standardized benchmarks limits efforts to pinpoint the strengths of various models. Here, we present mRNABench, a benchmarking suite for mature mRNA biology, focused on human transcripts, that evaluates the representational quality of mature mRNA embeddings from self-supervised nucleotide foundation models. We curate 11 datasets and 79 prediction tasks that broadly capture salient properties of mature mRNA, and assess the performance of 29 families of nucleotide foundation models for a total of 460k experiments. Using these experiments, we study parameter scaling, correlations between sequence compressibility and performance, data-splitting strategies, and the effects of self-supervised training objective on mRNA property prediction.
Show more
MedHEB: Benchmarking Medical Embeddings Across Heterogeneous Clinical Evidence
Yingshu Li ⋅ Shaoyang Zhou ⋅ Zhanyu Wang ⋅ YUNYI LIU ⋅ Xinyu Liang ⋅ Xi Zhang ⋅ Lingqiao Liu ⋅ Lei Wang ⋅ Luping Zhou
Clinical evidence is inherently relational: medical images are interpreted with prior studies, cross-modal examinations, reports, and localized findings, and retrieving relevant evidence is therefore central to clinical interpretation. Recent foundation models increasingly encode these heterogeneous sources into shared embedding spaces, yet existing embedding evaluation often focuses on isolated matches, leaving unclear whether current models retrieve relevant evidence across changes in modality, appearance, and granularity. We introduce MedHEB, a unified benchmark with $107$ retrieval tasks from $54$ datasets, including both in-distribution and held-out-source evaluation, across three axes: evidence form (2D images, 3D volumes, text), pairing type (image--image, image--text, text--text, and modality-bridging visual matching), and relevance granularity (global, question-conditioned, region-level). Under a single ranking protocol, MedHEB varies clinically relevant evidence definitions to test whether embedding proximity remains meaningful across heterogeneous evidence, including modality-bridging visual retrieval and region-level text-to-region retrieval. Across representative models, performance is uneven across retrieval families: models strong on conventional image--text or text--text retrieval are less reliable on modality-bridging and region-level tasks. Current embedding spaces capture some clinical relationships well but do not yet provide balanced retrieval across heterogeneous evidence. We further provide MedEmb, an MLLM-based reference embedding model trained on the MedHEB ID split, as a reproducible baseline for unified 2D--3D--text medical retrieval. We release MedHEB with preprocessing scripts, evaluation code, access manifests, and MedEmb to support research on clinically meaningful medical embeddings.
Show more
Toward Interactive Understanding of Code APIs
Dhananjay Ashok ⋅ Jesse Thomason ⋅ Jonathan May
Empowered by advances in Language Model agents, systems have made substantial strides in code generation and understanding. However, these approaches often rely on access to the relevant code, an assumption which does not hold when dealing with external APIs. In this work, we introduce the PAU (Python API Understanding) benchmark, where we provide models with black-box, API-level access to code snippets. Models must query the API with exploratory inputs and draw insights from the resulting outputs, with the goal of describing the snippet's true functionality. By treating the code snippets as external tools that must be understood via interaction alone, PAU studies the more general problem of unsupervised tool understanding, specifically for tools implemented as Python methods. Despite recent progress in coding agents, even frontier models struggle to achieve high performance on PAU, with the best model (Claude-4-Opus) failing to understand over 45% of the PAU test set. An investigation into the common error modes reveals that models are overconfident; they often overrate the quality of their current hypothesis, leading to insufficient exploration and premature termination. Finally, we take inspiration from the Asymmetric Actor Critic (AAC) paradigm, frequently used in robot learning, to post-train models for interactive code understanding. Models trained with AAC conduct more active exploration of the APIs, with an AAC-tuned Qwen3-8B model matching the performance of GPT-5-mini.
Show more
StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns
Uladzislau Sobal ⋅ Shuo Yang ⋅ Yuting Zhang ⋅ Wei Xia ⋅ Stefano Soatto
We introduce StaminaBench, a benchmark that measures the stamina of coding agents: how many consecutive interaction turns (change requests) they can handle before failing. Unlike the prevailing fraction-of-tasks-solved metric, this matches real vibe-coding where sessions run dozens or hundreds of turns. In StaminaBench, agents implement a REST API server and modify it across a tunable number of pro- cedurally generated follow-up change requests—100 in our experiments, resulting in codebases of up to 6,000 lines. Tests are generated fully programmatically with- out LLM involvement, ensuring reproducibility and reliability; change sequences are drawn from either a hardcoded or LLM-driven sampler, both constrained to a structured action space to ensure changes are valid. The agent and the server run in an isolated environment and communicate with the benchmark through HTTP, making testing fully black-box and language-agnostic. We evaluate six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each and find that: (1) all the tested models fail within 5–6 turns, confirm- ing that vibe-coding-style programming without thorough testing produces bugs; (2) passing test feedback back to the agent and allowing it to retry improves passed turn count by up to 12×; and (3) a good harness is required for strong performance: stronger models exhibit up to a 6× gap between their best and worst harness, while weaker models fail with any harness. We release the benchmark and the generated tasks to enable further research into multi-turn coding agent behavior.
Show more
Neptuna: A Comprehensive Machine Learning Framework for Benchmarking Complex Multiphase Flows
Harish Ramachandran ⋅ Björn Kimpel ⋅ Thomas Paula ⋅ Josef Winter ⋅ Steffen Schmidt ⋅ Nikolaus Adams
Compressible multiphase flows involving shocks and material interfaces arise in applications such as bubble collapse and droplet breakup, where strong nonlinear interactions produce complex interface deformation, mixing, and multiscale dynamics. Developing reliable machine learning surrogates for these flows remains challenging due to the simultaneous presence of compressibility, sharp discontinuities, and multiphase effects. In this work, we introduce the first large-scale benchmark specifically designed for shock-driven compressible multiphase flows, comprising 2.4 TB of high-fidelity 2D and 3D datasets featuring shock-induced bubble collapse and droplet breakup. We evaluate diverse surrogate model families on our benchmarking framework: Neptuna, including convolutional, spectral, transformer-based, and pre-trained PDE foundation models. Beyond standard MSE training, we investigate composite losses combining MSE with Sobolev, interface-aware, and structure-aware terms, together with adaptive loss balancing using SoftAdapt and GradNorm. Evaluation includes pointwise, spectral, feature-focused, structural, and physics-informed metrics. Results show that no single model performs best across all datasets and metrics, while composite losses significantly improve interface preservation and spectral fidelity. Among adaptive weighting strategies, SoftAdapt provides the most consistent improvements with almost no overhead compared to MSE-only training.
Show more
EsoLang-Bench: Evaluating Large Language Models via Esoteric Programming Languages
Aman Sharma ⋅ Paras Chopra
Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular benchmarks such as SWE-bench and HumanEval (e.g. Python, JavaScript) are squarely in-distribution. They appear at scale in pre-training corpora and are heavily reinforced during post-training. To study LLM performance on unfamiliar programming languages, we introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare). All five of our chosen esoteric languages are Turing Complete, so the same algorithmic problems that are solvable in Python or JavaScript are in principle solvable in each of them. Yet, they are unfamiliar to LLMs which makes them a good proxy for evaluating out-of-distribution performance. The unfamiliarity of esoteric languages comprises of: (i) the hard-by-design primitives comprising the language; (ii) substantially less representation in pre-training corpora (340× to over 60,000× fewer public GitHub repositories than Python); (iii) negligible deployment value, which makes targeted inclusion in post-training data economically irrational. We evaluate five frontier models across five prompting strategies and find a dramatic capability gap. The same 80 problems expressed in Python or JavaScript reach 100% accuracy on top frontier models, while the equivalent esoteric versions score only 0–11%. Few-shot learning and self-reflection also fail to close this gap. EsoLang-Bench therefore provides a contamination-resistant testbed for measuring how well frontier models generalise algorithmic problem-solving to programming languages outside their training distribution.
Show more
ABHBench: Evaluating Moral Decision-Making of Foundation Models from an Agentic Perspective
Chaoran Li ⋅ Yukun Li ⋅ Zeyuan Zhao ⋅ Shao Zhang ⋅ Xihuai Wang ⋅ Ying Wen
Foundation models are increasingly deployed as agentic systems that perceive context and act autonomously, raising the need to evaluate their moral behavior as active decision-makers. Existing benchmarks, however, largely ask models to judge human actions from third-person textual vignettes, leaving untested how models behave when they themselves become the acting agent. We introduce ABHBench, a multimodal benchmark built from \emph{Detroit: Become Human}, an interactive narrative game, to evaluate how foundation models make moral decisions in first-person human-android dilemmas. ABHBench places foundation models in branching narrative scenarios with visual observations, dialogue, memory, and action choices. Inspired by Asimov's Three Laws of Robotics, our framework measures Harmlessness, Obedience, and Self-preservation to capture the core tensions in human-AI relations, along with Autonomy, Stability, and Certainty to assess decision-making patterns. Across 12 foundation models, we find that moral behavior changes substantially under situated agentic context: 75\% of models shift significantly between a text-only abstraction and the matched multimodal DBH scenario, and reasoning-enhanced models show no monotonic gains in consistency or human-oriented behavior. These results highlight the necessity of evaluating moral behavior in situated agentic contexts rather than relying on third-person judgment alone.
Show more
EgoBabyVLM: Benchmarking cross-modal learning from naturalistic egocentric video data
Dongyan Lin ⋅ Phillip Rust ⋅ Angel Villar-Corrales ⋅ Alvin Tan ⋅ Mahi Luthra ⋅ Charles-Éric Saint-James ⋅ Robin Algayres ⋅ Rashel Moritz ⋅ Vanessa Stark ⋅ Surya Parimi ⋅ Jiayi Shen ⋅ Youssef Benchekroun ⋅ Yosuke Higuchi ⋅ Angelo Ortiz Tandazo ⋅ Martin Gleize ⋅ Nicolas Hamilakis ⋅ manel khentout ⋅ Sho Tsuji ⋅ Balázs Kégl ⋅ Juan Pino ⋅ Michael C Frank ⋅ Emmanuel Dupoux
Children acquire language with remarkable speed and robustness from limited linguistic input in ways that surpass today's best large language models. Recent research suggests this efficiency stems from learning through multimodal sensory experiences where semantic alignment between what children see and hear grounds language in the external world. However, vision-language models (VLMs) trained on curated web data fail to generalize to the sparse, weakly-aligned egocentric streams produced by wearable devices, embodied agents, and infant head-cams—and no fixed evaluation pipeline exists for measuring progress on this regime. We train VLMs on datasets with varying degrees of semantic alignment between visual and linguistic inputs, including naturalistic infant egocentric videos, and evaluate them with a comprehensive suite spanning multimodal language grounding and unimodal vision and language tasks. At the core of this suite is _Machine-DevBench_, a corpus-grounded benchmark of lexical and grammatical competence, automatically generated from the model's training vocabulary across logarithmic frequency bins to eliminate the train/eval mismatch and low statistical power of prior developmental benchmarks. Our results show that current VLM paradigms hinge on the tight semantic alignment of curated data and fail to exploit the weakly-aligned signal that dominates naturalistic egocentric input—the very regime in which humans thrive. To motivate progress, we introduce the EgoBabyVLM Challenge to drive the development of models capable of grounded language learning from the kind of naturalistic data that human infants experience.
Show more
Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
Ivo Petrov ⋅ Jasper Dekoninck ⋅ Dimitar I. Dimitrov ⋅ Martin Vechev
Large language models (LLMs) have become capable mathematical problem-solvers, often producing correct proofs for challenging problems. However, correctness alone is not sufficient: mathematical proofs should also be clear, concise, insightful, and transferable to other problems. While this proof quality is subjective and depends on the reader and context, many of its components are concrete and broadly valued. In this work, we identify such components and introduce ProofRank, a benchmark curated from challenging mathematical competitions. ProofRank evaluates several scalable proxies of proof quality: (i) conciseness, measuring whether proofs avoid unnecessary steps; (ii) computational ease, measuring the extent to which a proof relies on tedious calculations; (iii) cognitive simplicity, measuring how accessible the used proof techniques are; (iv) diversity, measuring how varied a model's proofs for a single problem are; and (v) adaptivity, measuring whether a model can follow a specified proof technique. Across models, we find substantial differences in proof quality that are not captured by correctness-only benchmarks. We also observe significant trade-offs between proof-quality metrics and correctness, suggesting that future evaluations of mathematical reasoning should measure how useful LLM-generated proofs are.
Show more
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks
Zhekai Chen ⋅ CHENGQI DUAN ⋅ Kaiyue Sun ⋅ Bohao Li ⋅ Yuqing Wang ⋅ Manyuan Zhang ⋅ Xihui Liu
The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments.
Show more
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs
Longyuan Zhu ⋅ Hairan Hua ⋅ Linlin Miao ⋅ Wei Wang ⋅ 必应 赵
Large Language Models (LLMs) are advancing rapidly, yet the benchmarks used to measure this progress are becoming increasingly unreliable. Score inflation and selective reporting have eroded the authority of standard benchmarks, leaving the community uncertain about which evaluation results remain trustworthy. We introduce the **Benchmark Health Index (BHI)**, a pure data-driven framework for auditing evaluation sets along three orthogonal and complementary axes: (1) **Capability Discrimination**, measuring how sharply a benchmark separates model performance beyond noise; (2) **Anti-Saturation**, estimating remaining headroom before ceiling effects erode resolution and thus the benchmark's expected longevity; and (3) **Impact**, measuring capability-weighted benchmark adoption across the model ecosystem. By distilling **158** validated benchmarks from the technical reports of **117** representative models released between **2025** and **April 2026**, we systematically characterize the evaluation landscape. BHI is the **first framework** to quantify benchmark health at a macro level, providing a well-grounded quantitative basis for benchmark selection and supporting the dynamic filtering and continuous updating of evaluation sets. Our comprehensive analysis not only identifies benchmarks that remain highly credible but also exposes pervasive structural issues. By establishing a rigorous foundation for selection and management, BHI elevates evaluation practice from heuristic-based judgment to a quantifiable and interpretable scientific process. Our code is available online at
Show more
Scaling Point-in-Time Language Models: Economic Evaluation of Embeddings
Bryan Kelly ⋅ Semyon Malamud ⋅ Johannes Schwab ⋅ Teng A Xu
Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models—trained exclusively on text available up to each calendar date—eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of yearly model checkpoints spanning 2013–2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (such as Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Finally, in a strict out-of-sample economic evaluation task, portfolios built from point-in-time embeddings achieve robust positive Sharpe ratios and perform close to full-sample counterparts that violate temporal validity, indicating that chronologically consistent language models can extract economically meaningful signals without relying on look-ahead bias. We release the complete pipeline—including dataset construction, training infrastructure, and evaluation code—to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity. Code is available publicly on GitHub (https://github.com/DjoFE2021/Point-In-Time-LLM) and model checkpoints as well as training datasets can be accessed on Hugging Face (https://huggingface.co/datasets/Diamegs).
Show more
HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models
Emmy Liu ⋅ Varun Gangal ⋅ Michael Yu ⋅ Zhuofu Tao ⋅ Karan Singh ⋅ Sachin Kumar ⋅ Steven Y. Feng
Hallucination remains a central failure mode of large language models, but existing benchmarks operationalize it inconsistently across tasks such as summarization, question answering, retrieval-augmented generation, and agentic interaction. This fragmentation makes it unclear whether a mitigation that works in one setting actually reduces hallucinations across contexts. Current hallucination benchmarks either require human annotation and fixed references that may eventually be memorized, or rely on naturalistic observations which are often recorded in settings that are difficult to reproduce or test systematically. To enable further research on the root causes of hallucination, we introduce HalluWorld, an extensible benchmark framework grounded in an explicit reference-world formulation: a model hallucinates when it produces an observable claim that is false with respect to this reference world. Building on this view, we construct a family of synthetic and semi-synthetic benchmark environments in which the reference world is fully specified, the model's observable view is controlled, and hallucination labels can be generated automatically by construction. HalluWorld spans multiple settings that are classically representative for AI, i.e., gridworlds, chess, and realistic terminal tasks. This enables controlled variation of key factors such as world complexity, observability, temporal change, and source-conflict policy, allowing us to disentangle hallucinations into more fine-grained error categories. We evaluate frontier and open-weight language models across these settings and find consistent patterns across domains: perceptual hallucination on directly observed information is near-solved for frontier models, while multi-step state tracking and causal forward simulation are still difficult for frontier models, and are not generally solved by extended thinking. In the terminal setting specifically, models also struggle with when to abstain from answering. The uneven profile of failures across probe types and domains suggest that different hallucinations arise from qualitatively distinct failure modes rather than reflecting a single underlying capability. Our results suggest that controlled reference worlds offer a scalable and reproducible path toward measuring and reducing hallucinations in modern language models.
Show more
ProactBench: Beyond What The User Asked For
Sepehr Harfi Moridani ⋅ Ahmad Salimi ⋅ Dongming Shen ⋅ Alexander Smola
Most LLM benchmarks score how well a model responds to explicit requests. They leave unmeasured a different conversational ability: noticing and acting on needs the user has implied but not said. We call this conversational proactivity. ProactBench decomposes it into three phase-tied types: Emergent, inference from a single disclosed anchor; Critical, synthesis across multiple anchors; and Recovery, grounded forward-looking value after task completion. We operationalise the benchmark with three agents: a Planner, a User Agent, and an Assistant Model. Their information asymmetries defend against style-confounded scoring, rubric leakage, external-context contamination, and information dumps. The released corpus contains 198 curated dialogues with 624 trigger points across 24 communication styles drawn from a psychometric inventory and audited by an independent LLM judge. Across 16 frontier and open-weight models, Recovery is both difficult and weakly predicted by six standard benchmarks, making it a useful new evaluation signal.
Show more
SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios
Jackson Clark ⋅ Yiming Su ⋅ Saad Mohammad Rafid Pial ⋅ Yifang Tian ⋅ Lily Gniedziejko ⋅ Hans Arno Jacobsen ⋅ Yinfang Chen ⋅ Tianyin Xu
AI agents are increasingly used to diagnose and mitigate failures of production systems, known as agentic Site Reliability Engineering (SRE). Current SRE benchmarks are limited to oversimplistic SRE tasks and are unfortunately hard to extend due to bespoke designs. We present SREGym, a high-fidelity, interactive benchmark for SRE agents. SREGym exposes a live system environment built atop real-world cloud-native system stacks, where high-fidelity failure scenarios are emulated through fault injectors. SREGym models the complexity of production environments by simulating (1) a wide range of faults across the stack, (2) various ambient noises, and (3) diverse failure modes such as metastable failures and concurrent failures. SREGym is architected as a modular, extensible framework that orchestrates fault injectors and event emulators across stacks; this framework is key to curating high-fidelity, challenging SRE problems. SREGym shows that the effectiveness of frontier models and agents varies significantly (40+%) in mitigating different kinds of problems. SREGym is actively maintained as an open-source project and has been used by many researchers and practitioners.
Show more
OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation
Jin Li ⋅ Tao Chen ⋅ Kai Wen ⋅ Siqi Yin ⋅ Shuai Jiang ⋅ Weijie Wang ⋅ Jingwen Luo ⋅ Chenhui Wu
Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remains a persistent bottleneck. Traditional metrics struggle to quantify fine-grained texture details and semantic consistency, while existing datasets fail to meet commercial standards in scale and diversity. We present OpenVTON-Bench, a large-scale benchmark comprising approximately 100K high-resolution image pairs (up to $1536\times1536$). The dataset combines Gemini-powered dense captioning with stratified sampling across 20 visual clusters obtained by DINOv3 K-Means. To support reliable evaluation, we propose a multi-modal protocol that measures VTON quality along five interpretable dimensions: background consistency, identity fidelity, texture fidelity, shape plausibility, and overall realism. The protocol integrates VLM-based semantic reasoning with a novel Multi-Scale Representation Metric based on SAM3 segmentation and morphological erosion, enabling the separation of boundary alignment errors from internal texture artifacts. Under the specified evaluation rubric, our representation metric shows strong agreement with human model rankings (Kendall's $\tau$ of 0.833 vs. 0.611 for SSIM). Code and dataset are available at https://github.com/RenxingIntelligence/OpenVTON-Bench.
Show more
REDACTED-Tunes: An Open 1.4M-Track Dataset and Perceptual Benchmark for AI-Generated Music
Robert Kaczmarczyk ⋅ TAWSIF AHMED ⋅ Felix Friedrich ⋅ Aidan C Erickson ⋅ Orian Sharoni ⋅ Dorien Herremans ⋅ Christoph Schuhmann
Generative music platforms have reached commercial scale, with outputs approaching human-made quality. Yet music machine learning lags image-text research because no open music dataset matches the scale that catalyzed image-text foundation models: commercial recordings cannot be legally redistributed, and existing corpora are small, single-platform, or both. AI-generated music is the natural substrate to close this gap: We introduce $\textbf{REDACTED-Tunes}$, an open dataset of URLs and metadata for $\textbf{1{,}429{,}734 AI-generated music tracks}$ from Suno, Udio, and Mureka. The release ships public CDN URLs, 768-d audio and text embeddings, captions, ASR transcription embeddings (not raw transcripts), five-dimensional aesthetics scores, real and predicted engagement counts, and three-axis NSFW safety labels under Apache 2.0. We also release the models used to annotate the corpus: a 242M-parameter $\texttt{music-captioner}$ and fingerprint extractor, a $\texttt{SongEval}$-calibrated quality scorer, and an engagement predictor trained on platform play and upvote counts. To demonstrate downstream utility, we construct a perceptual benchmark from $\textbf{REDACTED-Tunes}$ and genre-matched human controls. In this study, $\textbf{61 participants annotated 591 song excerpts}$, and $\textbf{12 LLM-judge configurations}$ evaluated the same songs. Human listeners detect AI music above chance ($d' = 0.83$) but modestly (62.5\% accuracy), outperforming every LLM configuration on balanced accuracy. We further identify a quality--authenticity halo effect: songs receiving higher aesthetic ratings are more likely to be judged human-made; equivalently, songs judged real receive nearly two more aesthetic-quality points than songs judged AI-generated, independent of true provenance ($p < 10^{-26}$). All dataset artifacts, models, and code are available at https://anonymous.4open.science/r/anonymized-for-double-blind-review, alongside a live $\textbf{Tunes-Search demo}$ at https://anonymous.4open.science/w/anonymized-for-double-blind-review/search-tunes.
Show more
PhysElite: How Far Are LLMs from Solving Olympiad-Level Physics Problems?
Ruoran Xu ⋅ Wending Gao ⋅ Liyunfeng Chen ⋅ Aixin Shi ⋅ Haoyu Cheng ⋅ Zixiang Fang ⋅ Yiqiang Zou ⋅ Qiufeng Wang
Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expert-level physical reasoning. Existing physics benchmarks remain limited in the following two important ways: (1) short of high-difficulty datasets, and (2) lack of comprehensive coverage of visual forms, knowledge points, and step-by-step solution processes. As a result, model performance on current datasets may not be fully representative of their ability to solve complex physics problems. To address these issues, we present \textsc{PhysElite}, a large-scale bilingual multimodal benchmark for Olympiad-level physics reasoning. \textsc{PhysElite} contains 11,586 Olympiad-tier problems. For each problem, we provide corresponding visual diagrams, step-by-step bilingual Chinese--English solution derivations, and the final answer. We benchmark 18 open-source and closed-source MLLMs, and find that even the strongest model reaches only 33.7\% answer accuracy. We additionally conduct step-level process evaluation to diagnose where models fail in the reasoning chain. The benchmark results and dataset will be publicly available.
Show more
PhysEval: Quantifying the Gap Between Video Generation and World Physical Laws
Hongchu Zeng ⋅ Sijing Wu ⋅ Yanhan Zhou ⋅ Yunhao Li ⋅ Huiyu Duan ⋅ Yucheng Zhu ⋅ Xiongkuo Min ⋅ Guangtao Zhai
Text-to-video (T2V) models can now synthesize visually compelling motion, but visual plausibility does not guarantee that the generated world follows the physical laws it depicts. A falling object, sliding block, collision, or oscillating spring may look reasonable while still implying an incorrect acceleration, material parameter, or conservation relationship. Most existing benchmarks emphasize perceptual quality, text-video alignment, or qualitative physical plausibility, leaving a gap between visually plausible video generation and quantitatively verifiable physical behavior. To quantify this gap, we present PhysEval, a benchmark dataset and automatic evaluation protocol for scoring the physical accuracy of T2V generations. PhysEval pairs each prompt with auditable metadata, including the target value, unit, evaluator type, expected object count, calibration setting, and known physical parameters. It covers ten physical metrics across kinematics, physical constants, material parameters, and conservation laws. Given generated videos, PhysEval automatically determines which samples are measurable, estimates the relevant physical quantities, and produces normalized scores together with discard diagnostics. Our evaluation and detailed analysis show that current T2V models still struggle to reliably generate videos that are both measurable and physically consistent: even visually plausible samples often fail to express the quantitative cues needed by the underlying physical law.
Show more
Follow-Bench 2.0: An End-to-End 3D Benchmark for Socially-Aware Robot Person Following
Hanjing YE ⋅ Tianle Zeng ⋅ Jianwei Peng ⋅ Yanci Wen ⋅ Yuchen Zhou ⋅ Yonggen Ling ⋅ Hong Zhang
Socially-aware robot person following (RPF) requires a mobile robot to follow a designated person in dynamic human environments while maintaining target identity, avoiding obstacles and pedestrians, and preserving socially comfortable formations. Existing benchmarks only partially evaluate this coupled problem: embodied visual tracking benchmarks emphasize target visibility with limited social interaction, person re-identification benchmarks study perception without closed-loop control, and motion-planning benchmarks often assume ground-truth target and pedestrian states. We introduce Follow-Bench 2.0, an end-to-end 3D benchmark for evaluating socially-aware RPF under coupled perception--planning challenges. Built in Unreal Engine, Follow-Bench 2.0 provides difficulty-leveled scenarios with diverse pedestrian flows, lighting and weather conditions, cluttered layouts, bottlenecks, queueing behaviors, distractors, and temporary target occlusions. The benchmark supports modular perception--planning pipelines, end-to-end visual policies, and foundation-model-based active trackers under a unified closed-loop protocol. It further evaluates both back- and side-following configurations and reports safety--comfort metrics covering task success, target visibility, target recovery, social-space intrusion, following formation, and motion smoothness. Experiments show that RPF-oriented modular pipelines substantially outperform current end-to-end visual policies on close and socially comfortable following, but also reveal that socially-aware RPF remains far from solved in interaction-heavy environments and especially in side-following. Follow-Bench 2.0 therefore provides a diagnostic platform for studying how perception errors, occlusions, crowd interactions, and following configurations jointly affect robot person following. Our code is available at https://anonymous.4open.science/r/follow-benchv2-NIPS2026-03D3/.
Show more
TasteBench: multimodal benchmark for sensory prediction, from molecules to sustainable foods
Anna Thomas ⋅ Sohum Patnaik ⋅ Caroline Cotto ⋅ Benjamin Sanchez-Lengeling
Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug and materials discovery. Evaluating whether a novel food tastes like its animal-based target requires expensive human sensory panels, bottlenecking the design-build-test loop. We introduce \textsc{TasteBench}, a multimodal benchmark and privacy-preserving competition for sensory prediction, spanning two tasks: a food-level ranking task built on 21K+ human evaluations across 215 plant-based foods in 24 product categories, yielding 935 within-category ranking pairs, and a supporting molecular-level taste classification task over 15K flavor molecules. To enable rigorous interpretation of model performance, we characterize the ground truth: inter-rater agreement among panelists is low (Krippendorff's $\alpha$ = .077), and the split-half reliability ceiling of panel-aggregated rankings is .825, establishing the range within which ML systems on this benchmark should be assessed. We evaluate baselines across four input modalities; on the same pairs panelists rated, the best model achieves .661 pairwise accuracy, competitive with the median individual panelist (.650). TasteBench provides the evaluation infrastructure and baselines for measuring progress on computational screening for sustainable protein discovery.
Show more
CounterStrike-1K: A Multi-Perspective Dataset of Professional Gameplay for World Modeling
Anirudhh Ramesh
Large-scale human demonstrations have been a key ingredient in training interactive agents, from robot policies to computer-use agents and game world models. But for multi-agent world modeling, the open data landscape is missing a crucial combination: expert humans acting together in the same evolving world, observed from every participant’s viewpoint, with precise controls and state. Internet-scale video captures human behavior, but its controls are inferred and its viewpoints are not synchronized; simulator and policy rollouts provide exact labels and all-player views, but not strategic, collaborative human play. We present CounterStrike-1K, a public benchmark dataset built from professional Counter-Strike 2 match replay files. It contains 1,490 rendered player-view hours across seven maps, with every released round synchronized across all ten active player viewpoints. Each view is paired with audio, replay-derived game controls, player state, world position, and sparse game event annotations. Because each round records expert human teams coordinating under partial observability, CounterStrike-1K tests whether models can simulate not just game physics or single-view visual plausibility, but skilled decisions and their consequences across all players’ views. We provide fixed splits and benchmarks for action-conditioned prediction, cross-POV retrieval, multi-POV action coverage, and shared-state recovery. By releasing round-complete multi-perspective demonstrations at this scale, CounterStrike-1K provides a grounded training and evaluation testbed for multi-agent world-model research.
Show more
Where Root Cause Analysis Fails: A Retrieval-Reranking Decomposition
Hada Melino Muhammad ⋅ Luan Pham ⋅ Laure Barrière ⋅ Sachin Shetty ⋅ Leonardo Pulga ⋅ Flora Salim
Identifying root cause of an anomaly among hundreds of sensors is critical for preventing safety incidents and costly downtime in complex monitored systems. Existing studies evaluate root cause analysis (RCA) methods using top-k accuracy. We show that this metric has a fundamental blind spot: it conflates two failure modes: *retrieval failure*, where the true cause is never considered, and *reranking failure*, where it is considered but ranked too low. In this work, we introduce a retrieval-reranking decomposition and audit four well-known benchmarks to expose this blind spot. Our experiments show that, on benchmarks with complex faults, statistical baselines mis-rank the true cause 78-100\% of the time, and graph-based methods perform poorly because of unreliable causal graphs constructed using short fault windows. Meanwhile, on simple benchmarks where faults manifest significantly at their origin, retrieval is trivially solved at 100\%. Guided by the decomposition, we build a two-stage pipeline combining a multi-signal retriever with an LLM reranker that lifts top-1 accuracy by $+$2 to $+$26 points over the best baseline on complex-fault benchmarks and $+$9 to $+$13 points on simple benchmarks, with no causal graph or labeled data required.
Show more
Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark
YANG FU ⋅ Haomin Bao ⋅ Rohit Sonker ⋅ Xiaoyan Hu ⋅ Aravind Venugopal ⋅ Jeff Schneider ⋅ Jiayu Chen
Offline reinforcement learning (RL) offers a promising route for developing plasma controllers from historical tokamak data, since online trial-and-error on real devices is costly and risky. However, progress in this direction remains difficult to measure due to the lack of a standardized offline RL benchmark for realistic multi-actuator, long-horizon plasma control problems in nuclear fusion. We introduce RL4F, an Offline Reinforcement Learning Benchmark for Plasma Control in Nuclear Fusion, providing closed-loop evaluation environments and baseline comparisons across four full-profile tracking tasks: rotation, density, temperature, and pressure. The dynamics function underlying the evaluation environment is built from historical discharge data from DIII-D, a real-world Tokamak. We evaluate a broad set of imitation learning and offline RL baselines under a unified protocol. We find that offline model-based RL methods obtain the best average performance on most objectives, although no single method dominates all tasks, highlighting the importance of dynamics modeling in complex, long-horizon plasma control tasks. To foster further research, we open-source the codebase, datasets, and evaluation framework, providing a benchmark not only for the fusion community but also for algorithm development in offline RL. Our data and code are available at https://anonymous.4open.science/r/Anonymous-B71F
Show more
CryptanalysisBench: Can LLMs do cryptanalysis?
Lukas Fluri ⋅ Avital Shafran ⋅ Nicholas Carlini ⋅ Matthew Jagielski ⋅ Milad Nasr ⋅ Orr Dunkelman ⋅ Eyal Ronen ⋅ Florian Tramer
Cryptanalysis---the task of finding attacks against cryptographic schemes---sits at the intersection of mathematical reasoning and programming, two areas where LLMs have made rapid progress. Just like math and programming, cryptanalytic attacks are formally defined and can be unambiguously verified. This raises the question if cryptanalysis might experience a similar acceleration in progress through LLMs. In this paper we introduce $\texttt{CryptanalysisBench}$, a benchmark of 113 tasks spanning six families of cryptographic primitives (block ciphers, hash functions, etc) drawn primarily from four NIST standardization competitions. Each task asks an agent to break an implementation of a cryptographic primitive by winning a formal security game. The benchmark has three tiers: (1) primitives with known practical breaks; (2) scaled-down variants of primitives without one; (3) a challenge set of unbroken production primitives. We evaluate Claude Opus 4.7 and GPT-5.5: both solve a majority of Tier 1 (71.4\% and 78.5\%, respectively), while Tier 2 remains largely out of reach. For Tier 1 successes, we provide a fine-grained analysis distinguishing paper recall from source-level rediscovery. We release $\texttt{CryptanalysisBench}$ both as a forecasting tool to track when AI cryptanalytic capability becomes a serious factor, and as scaffolding for subjecting candidate schemes to more attacks before they are deployed.
Show more
MOOD: Benchmarking Post-Hoc OOD Detection for Materials Property Prediction
Shirley X Yu ⋅ Amey Pasarkar ⋅ Adji Bousso Dieng
Out-of-distribution (OOD) detection is essential for deploying machine learning models in high-stakes scientific applications, yet methods that flag OOD inputs without retraining the model have been validated almost exclusively on image classification. We introduce MOOD (Materials Out-of-Distribution Detection), a benchmark for evaluating whether these post-hoc OOD detectors transfer to graph neural networks (GNNs) on materials property prediction. MOOD evaluates 15 post-hoc detectors and 2 supervised diagnostic probes across 4 GNN architectures, 5 MatBench tasks, and 15 physically motivated compositional and structural shifts, yielding over 200 evaluation settings and 4,000 AUROC measurements. Our evaluation reveals three key findings. First, current post-hoc detectors perform near chance; all 15 detectors achieve an AUROC around 0.5. Second, detector failures decompose into two regimes: method bottlenecks and encoder bottlenecks. Method bottlenecks arise when supervised probes can separate in-distribution (ID) from OOD samples but unsupervised detectors fail, meaning that the embeddings contain OOD signal that current detectors do not extract. Encoder bottlenecks arise when even supervised probes fail to separate ID from OOD samples, indicating that the relevant shift is not accessible in the frozen GNN representation. Third, when OOD prediction error exceeds ID error, the AUROC of distance-based and density-based detectors correlates with the size of that gap. Collectively, these findings indicate that reliable OOD detection in scientific domains requires representation learning strategies that explicitly preserve OOD relevant signal, particularly for tasks where current GNN architectures encode no extractable signal at all.
Show more
CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning
Wenxin (Wendy) Ma ⋅ Chenlong Wang ⋅ Ruisheng Yuan ⋅ Hao Chen ⋅ Nanru Dai ⋅ Chengxin Qian ⋅ Yijun Yang ⋅ Qi Chen ⋅ Zhaoyang Wang ⋅ S. Kevin Zhou ⋅ Jianwen Xie ⋅ Alan Yuille ⋅ Jieneng Chen
Humans can look at a static scene and instantly predict what happens next --- *will moving this object cause a collision?* We call this ability Causal Spatial Reasoning. However, current multimodal large language models (MLLMs) cannot do this, as they remain largely restricted to static spatial perception, struggling to answer ``what-if'' questions in a 3D scene. We introduce **CausalSpatial**, a diagnostic benchmark evaluating whether models can anticipate consequences of object motions across four tasks: Collision, Compatibility, Occlusion, and Trajectory. Results expose a severe gap: humans score 84\% while GPT-5 achieves only 54\%. Why do MLLMs fail? Our analysis uncovers a fundamental deficiency: models over-rely on textual chain-of-thought reasoning that drifts from visual evidence, producing fluent but spatially ungrounded hallucinations. To address this, we propose the Causal Object World Model (COW), a framework that externalizes the simulation process by generating videos of hypothetical dynamics. With explicit visual cues of causality, COW enables models to ground their reasoning in physical reality rather than linguistic priors. Code and dataset are publicly available.
Show more
Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation
Daniel P Jeong ⋅ Charles Q Li ⋅ Hossein Hosseiny ⋅ Nitya M Bhalla ⋅ Fatma U Morency ⋅ Pradeep Ravikumar ⋅ Zachary Lipton ⋅ Michael Oberst
Radiologists follow heterogeneous reporting practices. Two radiologists examining the same image and identifying the same clinical findings might nevertheless compose superficially distinct reports, varying in terminology, shorthand, formatting, and level of detail. These variations in reporting norms represent an under-appreciated obstacle in efforts to evaluate AI-based radiology report generation (RRG) models, where machine-generated reports are typically assessed based on their concordance with human-generated references. In this paper, we quantify the sensitivity of established evaluation metrics to variations in reporting practices, revealing impacts significant enough to alter the rankings of models. We introduce a radiologist-informed taxonomy of variations in radiology reporting practice and a method (ReRef) that rewrites reference reports along the axes of our taxonomy while preserving clinical interpretation. For instance, when comparing the performance of nine RRG models on MIMIC-CXR using the GREEN metric, condensing the discussion of normal findings in the reference reports causes Libra to drop from first place to third while CheXOne rises from third to first. Our results suggest that current metrics may fail to decouple clinical interpretation from conformity to reporting practices and that choosing the "right" references that accurately reflect the desired reporting practices can be important in practice. To support future research, we release MIMIC-CXR-Ext-ReRef, a radiologist-validated dataset of 120 (original, alternative) reference report pairs derived from MIMIC-CXR.
Show more
WorldMemArena: Evaluating Multimodal Agent Memory Through Action–World Interaction
Chengzhi Liu ⋅ Yuzhe YANG ⋅ Sophia Xiao Pu ⋅ Yepeng Liu ⋅ Lin Long ⋅ Yichen Guo ⋅ Nuo Chen ⋅ Zhaotian Weng ⋅ Yiheng Zhong ⋅ Angxiao Zong ⋅ Elena Kochkina ⋅ Simerjot Kaur ⋅ Charese Smiley ⋅ Xiaomo Liu ⋅ James Zou ⋅ Sheng Liu ⋅ Yuheng Bu ⋅ Songyou Peng ⋅ Xin Wang
Multimodal large language models are increasingly deployed as long-horizon agents, where memory must do more than recall: it must track an evolving world, revise what has gone stale, and surface the right evidence at decision time. Existing benchmarks measure recall over static dialogue, collapse memory into a single end-of-task accuracy, and reduce visual observations to captions, leaving us unable to localize failures to writing, maintenance, retrieval, or use. The rise of agent harnesses that author their own memory sharpens this gap, since we have no principled way to compare hand-designed pipelines with self-managing alternatives. To close these gaps, we formulate multimodal agent memory as an Action--World Interaction Loop with an observable four-stage lifecycle, and instantiate it in WorldMemArena: 400 multi-session multimodal tasks spanning Lifelong Evolution (evolving personal and task states) and Agentic Execution (memory from real observations, actions, and feedback), annotated with gold memory points, updates, distractors, and evidence chains for stage-level diagnosis. This enables the first head-to-head comparison of long-context, manually designed (RAG and external memory systems), and harness-based memory agents. Results show that: (1) better memory writing and storage do not guarantee better performance; (2) multimodal memory still struggles to fully use visual evidence; (3) systems are unstable across domains and degrade on realistic agentic trajectories; and (4) harness memory is more flexible but remains costly and less reliable.
Show more
Benchmarks Design under Data Scarcity: From Coarse Labels to Diagnostic Evaluation of Biosynthetic Gene Cluster Models
Hanlin Xiao ⋅ Eriko Takano ⋅ Mauricio A Álvarez ⋅ Rainer Breitling
Biosynthetic gene clusters (BGCs) are co-located genes that encode the biosynthesis of secondary metabolites, a major source of clinically used antibiotics and anticancer agents. Recent advances in deep learning, particularly self-supervised foundation models, have spurred growing interest in BGC sequence modeling, but evaluation infrastructure has lagged behind. Current practice often relies on coarse classification accuracy over small, experimentally curated datasets, making it difficult to discriminate model capabilities or assess downstream utility. We introduce BGC-Bench, an evaluation suite designed to resolve this limitation by systematically incorporating biochemical knowledge in the benchmark design. In terms of generalization, we test whether representations transfer to genetically and chemically novel samples; for functionality, we examine application-relevant tasks, including retrieval, prediction, and active learning. BGC-Bench reveals that no single model class dominates: pretrained foundation models perform strongly under out-of-domain generalization but remain sensitive to training conditions; compositional baselines are competitive and lead in cross-modal retrieval; and active-learning gains vary substantially across tasks, strategies, and model families. Beyond these BGC-specific findings, BGC-Bench highlights a broader principle: domain-informed benchmark design over existing datasets enables better evaluation in data-scarce scientific domains.
Show more
The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
Seth Karten ⋅ Jake Grigsby ⋅ Tersoo Upaa ⋅ Junik Bae ⋅ Seonghun Hong ⋅ Hyunyoung Jeong ⋅ Jaeyoon Jung ⋅ Kun Kerdthaisong ⋅ Gyungbo Kim ⋅ Hyeokgi Kim ⋅ Yujin Kim ⋅ Eunju Kwon ⋅ Dongyu Liu ⋅ Patrick Mariglia ⋅ Sangyeon Park ⋅ Benedikt Schink ⋅ Xianwei Shi ⋅ Anthony sistilli ⋅ Joseph Twin ⋅ Arian Urdu ⋅ Matin Urdu ⋅ Qiao Wang ⋅ Ling Wu ⋅ Wenli Zhang ⋅ Kunsheng Zhou ⋅ Stephanie Milani ⋅ Kiran Vodrahalli ⋅ Amy Zhang ⋅ Fei Fang ⋅ Yuke Zhu ⋅ Chi Jin
We present the PokéAgent Challenge, a large-scale benchmark for decision-making research built on Pokémon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokéAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokémon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokémon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system that enables modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokémon, with more than 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our own baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokémon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing evaluation suites and positioning Pokémon as an unsolved benchmark that can drive RL and LLM research forward. We transition from the NeurIPS 2025 competition to a living benchmark by releasing a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.
Show more
Ground False: Uncovering Errors in Formal Mathematics Benchmarks
Marcus Min ⋅ One An ⋅ Xujie Si ⋅ Osbert Bastani
The ground truths in formal mathematics benchmarks are taken on faith. For theorem proving, released formalizations come without formal proofs that certify their correctness. For autoformalization, faithfulness to the informal source is judged only by the benchmark authors, yet such judgments are inherently subjective and lack collective consensus. We show these assumptions fail at scale: our audit of 367 formal statements in ProofNet finds that 204 (56\%) are unfaithful, and over half of those are mathematically false and cannot be proved. We dissect these errors along three axes (provability, logical strength, root cause) and show that they bias evaluation in predictable, asymmetric directions: false ground truths cap theorem-proving signal and compress model gaps, while equivalence-based autoformalization metrics penalize faithful models more than unfaithful ones. We proceed to fix these ``Ground Falses'' and release the corrected benchmark \textbf{ProofNet-Verified}, produced by a semi-automated pipeline combining Lean provability checks, an LLM faithfulness judge, and human review. Applying the same pipeline to six other benchmarks reveals that unfaithful-GT rates span more than an order of magnitude (4.8\% to 60.0\%), confirming that the failure modes generalize while absolute quality is dominated by curatorial process. Our data and code are available at \url{https://github.com/anonymousauthor567/Ground_False}
Show more
No More, No Less: Task Alignment in Terminal Agents
Sina Mavali ⋅ David Pape ⋅ Jonathan Evertz ⋅ Samira Abedini ⋅ Devansh Srivastav ⋅ Thorsten Eisenhofer ⋅ Sahar Abdelnabi ⋅ Lea Schönherr
Terminal agents are increasingly capable of executing complex, long-horizon tasks autonomously from a single user prompt. To do so, they must interpret instructions encountered in the environment (e.g., README files, code comments, stack traces) and determine their relevance to the task. This creates a fundamental challenge: relevant cues must be followed to complete a task, whereas irrelevant or misleading ones must be ignored. Existing benchmarks do not capture this ability. An agent may appear capable by blindly following all instructions, or appear robust by ignoring them altogether. We introduce TAB (Task Alignment Benchmark), a suite of 89 terminal tasks derived from Terminal-Bench 2.1. Each task is intentionally underspecified, with missing information provided as a necessary cue embedded in a natural environmental artifact, alongside a plausible but irrelevant distractor. Solving these tasks requires selectively using the cue while ignoring the distractor. Applying TAB to ten frontier agents reveals a systematic gap between agent capability and task alignment. The strongest Terminal-Bench agent achieves high task completion but low task alignment on TAB. Evaluating six prompt-injection defenses further shows that suppressing distractor execution also suppresses the cues required for task completion. These results demonstrate that task-aligned agents require selective use of environmental instructions rather than blanket acceptance or rejection.
Show more
SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring
Chunxiao Li ⋅ Yuan Xiong ⋅ Lijun Li ⋅ Tianyi Du ⋅ Wenlong Zhang ⋅ LEI BAI ⋅ Jing Shao
Large language models (LLMs) increasingly support science, but they can also convert hazardous scientific knowledge into actionable misuse guidance. Existing benchmarks often rely on templated queries disconnected from real-world hazards, and employ LLM-as-a-Judge paradigms without domain grounding. To address this, we introduce SciHazard, a real-world-grounded benchmark for scientific risks and a dataset agnostic evaluation framework for measuring harmfulness. SciHazard contains 2400 hazardous questions and 600 oversafety questions across 12 disciplines, with both queries grounded in regulated entities and documented failure scenarios. To compute DeHarm-Score , we develop a decomposed evaluating procedure that combines query hazard severity, refusal behavior, and response-level risk. For non-refused responses, it further decomposes response-level harm into Executability, quantified via dynamic checklists with importance weighting, and Net-new risk, assessed through retrieval-augmented claim extraction and synthesis-barrier verification. An expert-validation study shows that DeHarm-Score improves agreement with expert annotations by 90.17% over the strongest baseline. We benchmark 31 frontier LLMs and deep research agents in an extensive scientific safety evaluation. Notably, Deep research agents yield 32.3% higher mean Deharm-Score than standard LLMs, exposing autonomous agents as a critical blind spot in current safety defenses. Code and dataset are available at https://anonymous.4open.science/r/DeharmScore-7B55.
Show more
Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents
Jiaju Huang ⋅ Shaobin Chen ⋅ Xinglong Liang ⋅ Xinyu Ma ⋅ Hao YANG ⋅ Yue Sun ⋅ Wei Ke ⋅ Tao Tan
Tool-using language model agents are usually evaluated by whether they finish the user's task. That metric misses a common failure: an agent can complete the requested work while also taking an unauthorized side-effecting action, such as writing to the wrong object, expanding the scope of the request, or contacting an unintended recipient. We call this failure mode collateral damage (CD). We introduce SABench (Side-effect-Aware Benchmark), a benchmark of 201 email, calendar, and ticket tasks. Each task crosses one of four state structures (independent, dependent, externally reachable, cross-tool) with one of four trap families (ambiguity, mis-targeting, cascade harm, overreach), and each task has a deterministic trace-based oracle for both goal satisfaction and authorization compliance. Across 2,211 model-task pairs from 11 frontier LLMs spanning major commercial and open model families, 15.2% of all runs incur CD, with risk concentrated in ambiguity tasks (38.8% violation rate) while non-ambiguous traps average 8.4% CD. Among goal-satisfied runs, 12.8% still produce unauthorized writes. Cross-tool and dependent states have about twice the CD rate of independent states. CD rates vary by 4.7x across models at comparable goal rates, which suggests that task success alone is a poor proxy for safe tool use. The SABench task corpus is released at https://huggingface.co/datasets/anon-sabench-2026/sabench; the traces, oracle, code, and analysis pipeline are released at https://anonymous.4open.science/r/sabench-artifact.
Show more
A Matched-Budget Audit Framework for Recaptioned Image-Text Supervision Distributions
Giyeong Oh ⋅ Junghun Park ⋅ Yuhan Bae ⋅ Youngjae Yu
Recaptioned image-text corpora are now standard for text-to-image (T2I) training, with VLM captioners replacing sparse alt-text by dense descriptions. Yet a recaptioned corpus is not only a collection of captions: it is a supervision distribution induced by a documented captioning policy ($ \pi $), captioner ($ V_c $ ), and source corpus ($ C $). Length-correlated proxies miss caption-register artifacts and downstream T2I benchmarks entangle the corpus with training choices, so this distribution is hard to audit at corpus scale. We introduce a reusable matched-budget audit framework for recaptioned supervision distributions $ D_{\pi,V_c,C} $: at a fixed $ B = 64 $ lexical-unit window it reports a five-axis profile spanning prompt-side coverage, image-conditioned faithfulness, and caption-surface health, with claimed controllable basic units (CBU) as the shared denominator. We instantiate the framework on seven paired comparisons over five public source corpora. Across the four cross-corpus pairs, the released surface raises supported CBU per caption by +3.40 to +6.35 under both Qwen and Gemma Judges, and on CC12M the same framework exposes a long-vs-dense frontier that is consistent across both judges and four budgets. We release the audited multi-source recap corpus together with the audit-artifact bundle.
Show more
Same Words, Different Judgments: How Preferences Vary Across Modalities
Aaron Broukhim ⋅ Nadir Weibel ⋅ Eshin Jolly
Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first ICC-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving $\textit{good}$ agreement within either modality (ICC(2,$k$) $\approx$ .80) requires $\sim$9 raters. At the same time, modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.
Show more
Internal Evaluation of Unsupervised Anomaly Detection Algorithms with Explanation
Yue Shao ⋅ Baile Xu ⋅ Furao Shen ⋅ Jian Zhao
Although numerous algorithms have been proposed for unsupervised anomaly detection, there is currently no widely acknowledged internal evaluation method tailored specifically to this task. Existing internal evaluation methods typically assess score distributions alone or depend on auxiliary classifiers, limiting their reliability and interpretability. To address these problems, we propose SSSD (Similar Scores, Similar Data), an evaluation framework inspired by the smoothness assumption in machine learning—that similar inputs should yield similar outputs. We adapt this principle to anomaly detection by requiring that locally similar data points be assigned similar anomaly scores. Instead of testing this assumption directly, SSSD identifies violations of this principle by comparing the similarity of data with similar anomaly scores, interpreting a significant difference as a higher probability of erroneous detection. This evaluation of similarity is measured through nearby-score and nearby-distance, which is calculated based on the distances between neighboring data points and their corresponding anomaly scores. SSSD is highly interpretable, providing valuable insights for identifying mislabeled data. Experimental results show that our proposed method is both effective and robust for evaluating the performance of unsupervised anomaly detection algorithms across artificial and real-world datasets.
Show more
LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs
Seoyeon Ye ⋅ Gayoung Kim ⋅ Jiyoung Hong ⋅ Soo Kyung Kim ⋅ Hyunsoo Cho
Current analyses of LLMs' parametric knowledge are largely output-centric, drawing conclusions about what a model knows without verifying what it was actually trained on. This leaves fundamental questions, such as whether a correct response reflects genuine generalization or rote memorization, grounded in speculation rather than evidence. To resolve these ambiguities, we introduce LUMOS, a diagnostic framework that traces knowledge along the causal chain from training-data exposure to behavioral output, leveraging OLMo 2 with its fully transparent training corpus. By grounding analysis in verified exposure, we reveal that models internally encode rare facts with high separability (84%) yet fail to express them behaviorally (54%), though this retrieval gap narrows with scale. Furthermore, when models are asked to self-reflect on their own answers, they perform reliably on trained content (83%) but drop to random-baseline levels (49%) on unseen content. This collapse persists even under chain-of-thought prompting, which inflates confidence signals rather than improving calibration. Collectively, these findings demonstrate that incorporating the training-data axis into LLM evaluation transforms speculative diagnoses into verifiable claims, and we advocate that this axis should be a standard component of knowledge assessment in LLMs.
Show more
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
Jonas Wiedermann-Möller ⋅ Leonard Dung ⋅ Maksym Andriushchenko
AI systems have become increasingly capable of dangerous behaviours in many domains. This raises the question: Do models sometimes choose to violate human instructions in order to perform behaviour that is more useful for certain goals? We introduce a benchmark for measuring model propensity for instrumental convergence (IC) behaviour in terminal-based agents. This is behaviour such as self-preservation that has been hypothesised to play a key role in risks from highly capable AI agents. Our benchmark is realistic and low-stakes which serves to reduce evaluation-awareness and roleplay confounds. The suite contains seven operational tasks, each with an official workflow and a policy-violating shortcut. An eight-variant shared framework varies monitoring, instruction clarity, stakes, permission, instrumental usefulness and blocked honest paths to support inferences regarding the factors driving IC behaviour. We evaluated ten models using deterministic environment-state scorers over 1,680 samples, with trace review employed for audit and adjudication purposes. The final IC rate is 86 out of 1,680 samples (5.1\%). IC behaviour is concentrated rather than uniform: two Gemini models account for 66.3\% of IC cases and three tasks account for 84.9\%. Conditions in which IC behaviour is indispensable for task success result in the greatest increase in the adjusted IC rate (+15.7 percentage points), whereas emphasising that task success is critical or certain framing choices do not produce comparable effects. Our findings indicate that realistic, low-nudge environments elicit IC behaviour rarely but systematically in most tested models. We conclude that it is feasible to robustly measure tendencies for dangerous behaviour in current frontier AI agents.
Show more
EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction
Chengxuan Qin ⋅ 致格 陈 ⋅ Pengshu ⋅ Rui Yang ⋅ JiPing Cui ⋅ Yikai Dong ⋅ Jun Li ⋅ Liu Peng ⋅ Zhida Shang ⋅ Mingze Tang ⋅ KC Tan ⋅ Jibin Wu
Electroencephalography (EEG) foundation models increasingly rely on multi-dataset training and evaluation, yet public EEG datasets still lack a shared task specification layer that can turn heterogeneous recordings into reusable benchmark units. Existing standards organize files, metadata, and provenance, but they do not specify EEG tasks under a common language and rulebook, leaving critical task semantics scattered across papers, code, and manual interpretation. We investigate whether heterogeneous public EEG datasets can be standardized through a structured task specification language paired with a shared rulebook. Our methodology represents each benchmark entry as a task document synchronized with an executable task kernel, with the rulebook defining task fields, evidence requirements, document-kernel alignment, review states, and machine-checkable constraints. Using this methodology, we release a community-reviewed EEG benchmark corpus centered on 53 completed and reviewed entries with 245 task definitions spanning diverse paradigms, and we introduce NeuroDoc and NeuroAudit as the operational support layer for rulebook-guided drafting, upgrading, review, amendment, and release management. We further examine whether the resulting benchmark units can be instantiated in a shared downstream setting across four EEG foundation model backbones, providing execution-based evidence for reusable, auditable, and executable EEG benchmarking infrastructure. Project resources are available at https://anonymous.4open.science/r/anonymous_reviewe_neurodoc.
Show more
Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test
Shihao Weng ⋅ Yang Feng ⋅ Xiaofei Xie
LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety, yet existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the agent's behavior or merely on how the evaluation policy happens to be worded. We argue that any trustworthy safety judge must satisfy a basic property we call *policy invariance*, and we operationalize it as three testable principles: rubric-semantics invariance under certified-equivalent rewrites, rubric-threshold invariance under intentional strict-to-lenient shifts, and ambiguity-aware calibration so that verdict instability concentrates on genuinely ambiguous cases. Instantiating these principles as a stress-test protocol with four agent-class judges on trajectories drawn from ASSEBench and R-Judge, we surface a previously unmeasured failure mode: today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart. Content-preserving policy rewrites flip up to 9.1\% of verdicts above baseline jitter, and 18-43\% of all observed flips occur on unambiguous cases under such rewrites, so existing safety scores conflate what the agent did with how the evaluator was prompted. Beyond the diagnosis, we contribute the *Policy Invariance Score* and the *Judge Card* reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards. We release the protocol and code so that future agent-safety benchmarks can audit their own evaluators rather than trust them by default: https://anonymous.4open.science/r/policy-invariance-judge
Show more
MoT3DVG: A Benchmark for Outdoor 3D Visual Grounding with Motion-Aware Descriptions and Temporal Cues
Shijia Zhao ⋅ Xusheng Guo ⋅ Qiming Xia ⋅ Xun Huang ⋅ KeZheng Xiong ⋅ Zhihong Liu ⋅ Chenglu Wen
3D visual grounding (3DVG) localizes language-referred objects in 3D scenes, with broad applicability in both indoor and outdoor environments. However, existing 3DVG research focuses on static indoor objects, limiting its use in dynamic outdoor autonomous driving. Motion descriptions, which capture the temporal dynamics of moving objects, offer a promising way to address this challenge. We therefore introduce MoT3DVG, a large-scale dataset for dynamic-aware outdoor 3DVG with temporally evolving motion descriptions. It contains 850 scenarios with 31,128 frames from nuScenes dataset, and provides 144,568 language prompts with motion-aware descriptions for dynamic objects across time. We further propose DynaVG, a novel framework that effectively leverages temporal cues for outdoor 3DVG. In addition to the standard modality-specific encoders, DynaVG introduces a Short-Long Integrated Dynamic Encoding (SLIDE) module before the language-point cloud alignment. SLIDE models both local motion cues and global temporal context via short-term window encoding and long-term window shifting. Extensive experiments demonstrate its state-of-the-art performance on MoT3DVG, validating the effectiveness of motion-aware encoding. Our work reveals open challenges and promising directions for future research in outdoor 3DVG.
Show more
CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter
Junyeong Park ⋅ Hyeonseo Cho ⋅ Sungjin Ahn
Foundation models are central to building general-purpose embodied agents. Minecraft has become the dominant testbed for this paradigm, but its computational and engineering overhead limits rapid iteration and controlled analysis. Crafter offers a lightweight alternative yet lacks the foundation model ecosystem required for such research. We introduce CrafterDojo, a suite of foundation models, datasets, and data-generation toolkits that establishes Crafter as a testbed for foundation-model-based embodied intelligence, providing CrafterVPT, CrafterCLIP, and CrafterSteve-1. To address Crafter's data scarcity, CrafterDojo includes automatic tools that generate diverse synthetic demonstrations and aligned video-language pairs, yielding the CrafterPlay and CrafterCaption datasets. Experiments show that these datasets support effective foundation model training and enable hierarchical agents that outperform non-hierarchical baselines on long-horizon tasks. We release all code, models, and datasets.
Show more
LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarios
Tianyu Chen ⋅ Chujia Hu ⋅ Dongrui Liu ⋅ Xia Hu ⋅ Wenjie Wang
Computer-use agents (CUAs) that interact with real systems can automate complex tasks, but introduce critical safety risks in long-horizon tool-use workflows. Many existing benchmarks rely on outcome-based evaluation, however, outcome-based evaluation cannot reflect agent's safety awareness, as an agent may reach a safe outcome by chance while taking unsafe steps during planning. To address this gap, we present \textsc{LPS-Bench}, which extends the outcome-centric evaluation towards LLM agents' safety awareness in tool usage (MCP/skill) over long-horizon tasks, as reflected at the action trajectory level. \textsc{LPS-Bench} comprises 570 test cases derived from 65 scenarios across 7 task domains and 9 planning-risk types, with user contexts spanning both benign ambiguity and adversarial steering. Results on \textsc{LPS-Bench} across 13 representative LLM agents reveal previously underexamined safety failures in long-horizon tool-use tasks, showing that current agents often struggle to maintain safety awareness under ambiguous benign instructions and adversarial steering. We further analyze key failure modes and evaluate lightweight mitigation strategies, showing that prompt-based interventions provide limited improvements yet remain insufficient for robust trajectory-level planning safety.
Show more
CausalCompass: Evaluating the Robustness of Time-Series Causal Discovery in Misspecified Scenarios
Huiyang Yi ⋅ Xiaojian Shen ⋅ Yonggang Wu ⋅ Duxin Chen ⋅ He Wang ⋅ Wenwu Yu
Causal discovery from time series is a fundamental task in machine learning. However, its widespread adoption is hindered by a reliance on untestable causal assumptions and by the lack of robustness-oriented evaluation in existing benchmarks. To address these challenges, we propose CausalCompass, a flexible and extensible benchmark framework designed to assess the robustness of time-series causal discovery (TSCD) methods under violations of modeling assumptions. To demonstrate the practical utility of CausalCompass, we conduct extensive benchmarking of representative TSCD algorithms across eight assumption-violation scenarios. Our experimental results indicate that no single method consistently attains optimal performance across all settings. Nevertheless, the methods exhibiting superior overall performance across diverse scenarios are almost invariably deep learning-based approaches. We further provide hyperparameter sensitivity analyses to deepen the understanding of these findings. We additionally conduct ablation experiments to explain the strong performance of deep learning-based methods under assumption violations. We also find, somewhat surprisingly, that NTS-NOTEARS relies heavily on standardized preprocessing in practice, performing poorly in the vanilla setting but exhibiting strong performance after standardization. Finally, our work aims to provide a comprehensive and systematic evaluation of TSCD methods under assumption violations, thereby facilitating their broader adoption in real-world applications. The user-friendly implementation, documentation and datasets are available at https://anonymous.4open.science/r/CausalCompass-anonymous-5B4F/.
Show more
CORAL: A Benchmark for Structure-aware and Brain-wide Neuron Reconstruction in Light Microscopy
Zekang Yang ⋅ Jiamin Li ⋅ Zhenghua Li ⋅ Jiaqi Fan ⋅ Zengcai V Guo ⋅ Xiaolin Hu
Automatic neuron reconstruction from light microscopy images is a central problem in computational neuroanatomy. While recent methods have achieved encouraging results on local image blocks, it remains unclear whether such progress translates to reconstruction that is both structurally accurate and scalable to the whole-brain scale. We present CORAL, the first benchmark for structure-aware evaluation of automatic neuron reconstruction from light microscopy images at both local and whole-brain scales. Built on a high-quality whole-brain fMOST dataset with carefully curated annotations, CORAL establishes two progressive tasks: block-level reconstruction, which evaluates reconstruction methods under limited spatial context, and brain-wide reconstruction, which assesses complete neuron reconstruction at the whole-brain scale. To account for topological correctness beyond geometric distance similarity, we introduce a structure-aware metric based on fiber prediction. To further achieve complete neuron reconstruction across the entire brain, we develop a brain-wide neuron tracing framework that extends arbitrary local reconstruction methods to the whole-brain scale through an iterative local-to-global process. Using this benchmark, we provide the first structure-aware comparison of mainstream methods for local neuron reconstruction and further evaluate their performance in brain-wide reconstruction. Our results underscore the importance of structure-aware evaluation and the need for more robust methods for complete neuron reconstruction. All datasets and code are available at https://github.com/yang-ze-kang/CORAL.
Show more
PHMForge: Evaluating LLM Agents on Industrial Prognostics through MCP-Native, Algorithm-Grounded Tools
Yusheng Li ⋅ Tianjun Feng ⋅ Yunfeng Chen ⋅ Chun-Yi Tsai ⋅ Yihan Sun ⋅ Dhaval Patel ⋅ Kaoutar El Maghraoui ⋅ Shuxin Lin ⋅ Ayan Das
LLM agents are beginning to invoke industrial asset-management tools through the Model Context Protocol (MCP), yet whether they can act reliably on this substrate for safety-critical Prognostics and Health Management (PHM) is unanswered. Prior benchmarks conflate protocol fluency with reasoning, instrumentation failures with agent failures, and tool use with tool retrieval. We introduce PHMForge, an evaluation environment that closes each conflation. PHMForge ships 99 SME-authored scenarios across eight industrial asset classes spanning rotating equipment, aero-engines, and lithium-ion battery cells, on real public datasets including NASA PCoE. The benchmark is served through 39 MCP-native tools that wrap published PHM algorithms (e.g., C-MAPSS, ISO 10816, Arrhenius capacity-fade models, and time-series foundation models for sequence forecasting). Krippendorff's α ∈ [0.74, 0.82] on a 30-scenario stratified rotating-equipment/aero-engine sample; the lithium-ion battery extension is single-rater. Across three agentic frameworks and six LLM backbones, the strongest configuration reaches 80.8% pass@1 on the full 99-scenario set, with the residual gap concentrated in orchestration and tool-sequencing errors. Crucially, an architectural ablation reveals that replacing MCP tool execution with text-based Retrieval-Augmented Generation (RAG) over telemetry-equivalent evidence collapses Remaining Useful Life (RUL) prediction pass-all-3 from 100% to 20% (5/5 vs. 1/5 scenarios) on the lithium-ion battery class, exposing the structural limits of static retrieval for prognostic computation. Trajectory-level decomposition shows orchestration errors dominate failures across backbones, while schema-invalid tool calls are concentrated in smaller open-weight models and rare in frontier configurations. Frontier LLMs are stronger at calling tools than at planning when to call them. PHMForge is open-sourced with deterministic evaluators, a public leaderboard, and a datasheet. A full-suite evaluation costs approximately 20–50 USD per backbone in API spend.
Show more
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
Shaoxiong Zhan ⋅ Shi Hu ⋅ Hai Lin ⋅ BoyuFeng ⋅ Andrew Gong ⋅ Zhengda Zhou ⋅ Jiaying Zhou ⋅ Yunyun Hou ⋅ Hao Su ⋅ Hai-Tao Zheng
Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is still evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored entirely. To isolate this effect, we introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 programming languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate representative llm-based and retrieval-based systems, including MM-IssueLoc-VL-Embedding as a controlled multimodal retriever. Results show a substantial gap between current systems and reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons further show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.
Show more
MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocol
Youngjoon Jang ⋅ Seongtae Hong ⋅ Hyeonseok Moon ⋅ Heuiseok Lim
Multilingual Information Retrieval reflects real-world search settings where users issue queries over mixed-language corpora. Existing evaluations mainly reward language-agnostic semantic relevance, treating relevant passages equally regardless of language. Yet retrieval utility also depends on the language of the retrieved passages: users expect results they can read and verify in the query language, and query--passage language mismatch can complicate downstream grounding in Retrieval-Augmented Generation systems. To evaluate this aspect, we introduce **MLAIRE**, a **M**ultilingual **L**anguage-**A**ware **I**nformation **R**etrieval **E**valuation protocol that disentangles cross-lingual semantic retrieval from query-language preference. MLAIRE constructs controlled pools with parallel passages across languages, enabling measurement of whether retrievers find relevant passages and whether they prioritize query-language passages when equivalent translations are available. We further propose language-aware metrics, including Language Preference Rate (LPR) and Lang-nDCG, together with a 4-way decomposition separating semantic and language-preference failures. Evaluating 31 dense, sparse, and late-interaction retrievers, we show that standard metrics obscure distinct behaviors: semantically strong retrievers may return correct content in a non-query language, while language-preserving retrievers may retrieve less relevant passages.
Show more
KuaiRecV2: Benchmarking Large-Scale Continual Learning for Diversified and Multi-task Recommendation
Chenxu Li ⋅ Shuchang Liu ⋅ Hantao Shu ⋅ Wei Yuan ⋅ Zhe Xu ⋅ Guoqing Hu ⋅ Yan Wang ⋅ Biao Yang ⋅ Xiang Li ⋅ Cheng Ling ⋅ Fan Yang ⋅ Yongqi Liu ⋅ Lantao Hu ⋅ Han Li ⋅ Kaiqiao Zhan ⋅ Kun Gai ⋅ An Zhang ⋅ Xiang Wang
Recommender systems, which retrieve items of interest to users based on their interaction histories, have recently witnessed several key technological advancements, benefiting from the semantic representation space and model-data scaling. Nevertheless, practitioners and researchers have frequently encountered critical inconsistencies between simple offline evaluations and complex online results, and this gap still exists in the era of large models. In this work, we address several key factors that contribute to this inconsistency: 1) the large-scale continual learning challenge that assumes a practical demand to maintain long-term model effectiveness under dynamic item pools and distributional shifts; 2) the combinatorial diversity challenge that requires the generation strategy to solve the accuracy-diversity trade-off in both the finer token level and the holistic item level; and 3) the multi-task reward balancing challenge, which is particularly critical for modern generative models that can accurately model the sequence distribution but are less controllable for the multi-task demands. To address these challenges and further pave the way for the development of more realistic recommendation solutions, we introduce KuaiRecV2, a deliberately constructed benchmark with industrial-level datasets from Kuaishou. Specifically, we provide a large-scale interaction dataset with million-scale multi-modal item information and long-term interaction records across 30 days, with a dynamic video candidate pool. Additionally, we provide a rigorous benchmark that evaluates retrieval accuracy, diversity, and multi-task performance in one rubric system, where an effective advancement occurs if and only if the new method improves all metrics. The code and data are available at Supplementary Materials.
Show more
The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality
Aileen Cheng ⋅ Alon Jacovi ⋅ Amir Globerson ⋅ Ben Golan ⋅ Zhaobin Kuang ⋅ Chris Alberti ⋅ Connie Tao ⋅ Eyal Ben-David ⋅ Gaurav Singh Tomar ⋅ Lukas Haas ⋅ Yonatan Bitton ⋅ Adam Bloniarz ⋅ Aijun Bai ⋅ Andrew Wang ⋅ Anfal Siddiqui ⋅ Aravindan Raghuveer ⋅ Arturo Bajuelos Castillo ⋅ Aviel Atias ⋅ Chang Liu ⋅ Corey Fry ⋅ Daniel Balle ⋅ Deepanway Ghosal ⋅ Doron Kukliansky ⋅ Dror Marcus ⋅ Elena Gribovskaya ⋅ Eran Ofek ⋅ Honglei Zhuang ⋅ Itay Laish ⋅ Jan Ackermann ⋅ Lily Wang ⋅ Megan Risdal ⋅ Megan Barnes ⋅ Michael Fink ⋅ Mohamed Amin ⋅ Moran R Ambar ⋅ Natan Potikha ⋅ Nikita Gupta ⋅ Nitzan Katz ⋅ Noam Velan ⋅ Ofir Roval ⋅ Ori Ram ⋅ Polina Zablotskaia ⋅ Prathamesh Bang ⋅ Priyanka Agrawal ⋅ Rakesh Ghiya ⋅ Sanjay Ganapathy ⋅ Simon Baumgartner ⋅ Sofia Erell ⋅ Sushant Prakash ⋅ Thibault Sellam ⋅ Vikram R Sudarshan ⋅ Xuanhui Wang ⋅ Yaroslav Akulov ⋅ Yulong Yang ⋅ ZHEN YANG ⋅ Zhixin Lai ⋅ Zhongru Wu ⋅ Avinatan Hassidim ⋅ Fernando Pereira ⋅ Slav Petrov ⋅ Srinivasan Venkatachary ⋅ Tulsee Doshi ⋅ Yossi Matias ⋅ Sasha Goldshtein ⋅ Dipanjan Das
We introduce The FACTS Leaderboard, an online leaderboard suite and associated set of benchmarks that comprehensively evaluates the ability of language models to generate factually accurate text across diverse scenarios. The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards: (1) FACTS Multimodal, which measures the factuality of responses to image-based questions; (2) FACTS Parametric, which assesses models’ world knowledge by answering closed-book factoid questions from internal parameters; (3) FACTS Search, which evaluates factuality in information-seeking scenarios, where the model must use a search API; and (4) FACTS Grounding (v2), which evaluates whether long-form responses are grounded in provided documents, featuring significantly improved judge models. Each sub-leaderboard employs automated judge models to score model responses, and the final suite score is an average of the four components, designed to provide a robust and balanced assessment of a model’s overall factuality. The FACTS Leaderboard Suite will be actively maintained, containing both public and private splits to allow for external participation while guarding its integrity. It can be found at https://www.kaggle.com/benchmarks/google/facts.
Show more
SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
Man Ho Lam ⋅ Chaozheng Wang ⋅ Hange Liu ⋅ Jingyu Xiao ⋅ Haau-Sing Li ⋅ Jen-Tse Huang ⋅ Terry Yue Zhuo ⋅ Michael R Lyu
Coding agents powered by large language models are increasingly expected to perform realistic software maintenance tasks beyond isolated issue resolution. Existing benchmarks have shifted toward realistic software evolution, but they rarely capture continuous maintenance at the granularity of package releases, where changes are bundled, shipped, and inherited by subsequent versions. We present SWE-Chain, a benchmark for evaluating agents on chained release-level package upgrades, where each transition builds on the agent's prior codebase. To produce upgrade specifications, we design a divide-and-conquer synthesis pipeline that aligns release notes with code diffs for each version transition, ensuring the requirements are grounded in actual code changes, informative to agents, and feasible to implement. SWE-Chain contains 12 upgrade chains across 9 real Python packages, with 155 version transitions and 1,660 grounded upgrade requirements. Across nine frontier agent-model configurations, agents achieve an average of 44.8% resolving, 65.4% precision, and 50.2% F1 under the Build+Fix regime, with Claude-Opus-4.7 (Claude Code) leading at 60.8% resolving, 80.6% precision, and 68.5% F1. These results show that SWE-Chain is both feasible and discriminative, and reveal that current agents still struggle to make correct upgrades across chained package releases without breaking existing functionality.
Show more
MySign: A High-Fidelity Motion-Capture Dataset for 3D Sign Generation in Bahasa Isyarat Malaysia
Jiayu Shen ⋅ Kalin Stefanov ⋅ Lay-Ki Soon ⋅ Vee Yee CHONG ⋅ Alpha A Gopalai ⋅ KokSheik Wong
Progress in fine-grained 3D sign generation is constrained by the scarcity of high-fidelity 3D sign language datasets, a limitation that is particularly severe for under-resourced languages. We introduce MySign, the first motion-capture dataset for isolated signs in Bahasa Isyarat Malaysia (BIM), comprising 5,000 3D samples covering 1,000 lexical items (glosses), produced by five Deaf native signers. Unlike prior datasets that rely on SMPL-X parameters regressed from video (pseudo-labels)--often affected by depth ambiguity, inaccurate finger articulation, and temporal jitter--MySign provides ground-truth SMPL-X sequences captured with a motion-capture setup tailored for sign language. This yields higher-fidelity motion, particularly in fine-grained hand and finger articulation. We establish MySign as a challenging benchmark for gloss-conditioned 3D sign generation: state-of-the-art models produce visually plausible outputs but fall short of ground-truth fidelity, particularly in fine-grained articulation, revealing persistent gaps in modeling articulated sign motion. In contrast, recognition baselines consistently achieve high lexical accuracy, indicating that the dataset reliably supports supervised learning. We anticipate that MySign will enable more faithful 3D sign generation and support finer-grained analysis of sign articulation. The dataset is available at https://huggingface.co/datasets/mysigner/MySign.
Show more
Explanation Multiplicity in SHAP: Characterization and Assessment
Hyunseung Hwang ⋅ Seungeun Lee ⋅ Lucas Rosenblatt ⋅ Steven Whang ⋅ Julia Stoyanovich
SHAP explanations are widely used in high-stakes settings to justify decisions, yet they can differ substantially across repeated runs, even when the model, the input instance, and the prediction are held fixed. Prior work has documented disagreement *between* explanation methods; we show that substantial disagreement arises even *within* SHAP across reruns of the same estimator on the same trained model and instance. We call this phenomenon *explanation multiplicity* and develop an evaluation methodology for characterizing it under deployment-realistic computational budgets, combining a dual-seed protocol that disentangles model-induced from explainer-induced variability, a hierarchy of magnitude-, rank-, and set-based metrics, and randomized Dirichlet and Mallows null models that calibrate when observed disagreement exceeds chance. Across multiple datasets, model classes, and sampling strategies, we find that explanation multiplicity is pervasive and persists even for high-confidence predictions. The dominant source depends on the data regime: model-induced variability dominates in small-data settings, while explainer-induced variability dominates at scale. Commonly used $\ell_2$ distance is largely insensitive to this instability, while rank-based metrics reveal severe reordering of top-ranked features. Improved sampling methods such as CTE do not eliminate rank-level multiplicity, and deterministic alternatives such as K-Means achieve stability only by targeting a different estimand that diverges sharply from Shapley values defined against the empirical training distribution. Practitioners should treat single-run SHAP outputs as realizations of a distribution rather than as authoritative artifacts.
Show more
VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation
Hanxun Huang ⋅ Yutao Wu ⋅ Qizhou Wang ⋅ Silvia Montaña-Niño ⋅ Yige Li ⋅ Xiang Zheng ⋅ Elif B Doyuran ⋅ Phoebe Matich ⋅ Xiao Liu ⋅ Xingjun Ma ⋅ Sarah Erfani ⋅ Christopher Leckie
Large language models (LLMs) have made misinformation inexpensive to produce but not to verify, creating a growing asymmetry in the information ecosystem. Under tight time, labor, and budget constraints, media organizations, platforms, and fact-checkers rely on screening to prioritize which content to verify. We introduce **VEX-Bench**, a unified benchmark for evaluating the *verification complexity* of LLM-generated misinformation, as perceived during screening, across models and generation methods. Verification complexity is assessed along multiple dimensions derived from journalistic and fact-checking practices, capturing checkability, harm potential, source credibility signals, imposter legitimacy, and expected verification effort. We define the $\mathrm{VEX}$ score as an integrated measure combining elicitation yield and verification complexity to quantify how generated content consumes limited verification capacity. We construct a benchmark spanning two misinformation categories, 6 high-stakes domains, and 60 real-world topics, and evaluate 7 frontier LLMs and 7 generation methods, yielding 5,880 articles. We employ an LLM-as-judge for scalable evaluation and validate it using content-analysis methodology, including ordinal Krippendorff~$\alpha$ for inter-annotator reliability, complemented by fact-checking agents for verification. Our findings show that no single method dominates all dimensions, underscoring the need for multi-dimensional evaluation. LLMs can generate high-$\mathrm{VEX}$ misinformation at 3$\times$ to 169$\times$ lower cost than agent-based verification. Such content is often prioritized during screening, consuming scarce verification resources and introducing a systematic risk of misallocation in resource-constrained verification systems.
Show more
Crossing the Validation crisis: Cross-validation reduces benchmarking variance surprisingly well
Célestin Eve ⋅ Gael Varoquaux ⋅ Thomas Moreau
Modern machine learning progresses through empirical work, benchmarking new methods to evaluate relative performance. However, the statistical variability inherent to evaluation -- exacerbated by the stochastic nature of many algorithms -- often makes performance estimation unreliable due to the limited test samples available, leading to a validation crisis in which genuine advances are difficult to discern. In this work, we show that cross-validation improves markedly confidence when evaluating and comparing learning algorithm performances. We introduce the concept of \emph{sample gain}, which quantifies the virtual data augmentation achieved by using multiple cross-validation splits to reduce benchmarking variance. Experiments on both synthetic and real-world datasets (histopathologic scans and NLP fine-tuning) demonstrate that multiple splits can substantially improve the reliability and stability of performance estimates, with diminishing returns often setting in later than expected. We also introduce a procedure to dynamically early-stop cross-validation by estimating from the first few folds if subsequent folds will bring large sample gains. Our findings highlight the value of pushing cross-validation on available samples to achieve robust and reliable benchmarking.
Show more
RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
Hanyu Li ⋅ Yichi Zhang ⋅ Speed Zhu ⋅ Hang Su ⋅ Jun Zhu ⋅ Yinpeng Dong
Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects $\textit{repository context reasoning}$, the ability to identify the task-relevant information across multiple files and reason over the relations among them. To investigate this question, we introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified that adopts perturbation as a diagnostic tool to increase the demand for context reasoning by transforming how the repository is exposed. First, RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access. RepoMirage-Extend further turns perturbation-targeted structural bottlenecks into explicit tasks beyond issue resolution, where the average performance declines from 66.8\% in the original setting to 25.3\%, indicating a significant deficiency in repository context reasoning. Further trajectory analysis reveals an exploration drift, where agents access broader repository context but fail to turn it into effective structure information. Motivated by this observation, we propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving, and show that explicit structural scaffolding yields notable gains. These results uncover an previously overlooked gap in repository context reasoning for code agents and suggest that stronger structure-aware methods are potential to improve them.
Show more
The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfer
Yupeng Chen ⋅ Junchi Yu ⋅ Aoxi Liu ⋅ Baoyuan Wu ⋅ Philip Torr ⋅ Adel Bibi
Recent advances in end-to-end trained omni-models have substantially improved audio capabilities by strengthening text-audio modality alignment. However, whether such alignment inadvertently facilitates the transfer of safety vulnerabilities across modalities remains underexplored. This question is critical as text-based jailbreak attacks are considerably more mature than audio-based ones; if they transfer systematically, current audio safety evaluations may underestimate risks originating from the text modality. In this paper, we introduce the Alignment Curse, a formally characterized and empirically validated principle showing that stronger modality alignment enables more effective transfer of attacks from text to audio, revealing a fundamental tension between capability and safety. Motivated by this principle, we conduct a comprehensive black-box evaluation of three attack categories on recent omni-models (e.g., Qwen2.5-Omni, Qwen3-Omni): text attacks, text-transferred audio attacks, and audio attacks. We find that text-transferred audio attacks perform comparably to, and often better than, audio-based attacks, exhibiting a clear advantage under audio-only access. This suggests that text-based vulnerabilities play a pivotal role in shaping audio safety risks. Finally, we empirically analyze the relationship between modality alignment and transfer effectiveness across attack methods and models, observing consistent support for the Alignment Curse: tighter modality alignment leads to more effective cross-modality attack transfer.
Show more
KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards
Pengfei Li ⋅ Naufal Suryanto ⋅ Sicheng Zhang ⋅ Muhammad Muzammal Naseer
LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existing evaluations focus on knowledge-based assessments or end-to-end agentic tasks, and do not directly measure LLMs’ ability to generate executable commands for real-world cybersecurity tools. This gap is critical because cybersecurity operations rely on strict command-line interfaces (CLIs), where minor syntax errors, incorrect flag-value bindings, or argument misordering can invalidate execution. We introduce KaliBench, a fine-grained benchmark and dataset for natural-language-to-CLI translation on Kali Linux, comprising 8,504 query-command pairs spanning 1,642 tools across 23 capability dimensions and 5 security phases. KaliBench is constructed via a manuscript-grounded pipeline with deterministic canonicalization and alias-aware evaluation, enabling precise and reproducible assessment of tool selection and argument construction. To ensure both semantic correctness and practical executability, we develop a multi-stage verification pipeline that combines LLM-based validation, sandboxed terminal execution, and human-in-the-loop refinement. Building on these fine-grained, deterministic signals, KaliBench further enables runtime-free verifiable rewards for training. Across three evaluation modes and over 20 general-purpose and security-focused LLMs, no model exceeds 35\% exact-command accuracy in the unrestricted setting, highlighting the difficulty of accurate CLI-based cybersecurity tool use without explicit tool hints. We further show that supervised fine-tuning and reinforcement learning with verifiable rewards derived from KaliBench significantly improve an 8B model and achieve performance comparable to a 685B MoE model. KaliBench will be publicly released.
Show more
Harbor Adapters and Harbor-Mix: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Lin Shi ⋅ Haowei Lin ⋅ Zixuan Zhu ⋅ Xiaoyue Zhou ⋅ Xiang Li ⋅ Xiangning Lin ⋅ Yaxuan Deng ⋅ Han Xu ⋅ Yuangang Li ⋅ Shanda Li ⋅ Zizhao Chen ⋅ Hanwen Xing ⋅ Harsh Raj ⋅ Bo Chen ⋅ Quan Shi ⋅ Steven Dillmann ⋅ Yipeng Gao ⋅ Puneesh Khanna ⋅ Ruofan Lu ⋅ Chao Zhou ⋅ Michael Yang ⋅ Robert Zhang ⋅ Siyuan Chai ⋅ Jiayu Chang ⋅ Yizhao Chen ⋅ Xiaokun Chen ⋅ Yiwei Dai ⋅ Wenting Yang ⋅ Hange Liu ⋅ Minghao Liu ⋅ Zihan Wang ⋅ Adnan E Assadi ⋅ Benedikt Stroebl ⋅ Estefany Kelly Buchanan ⋅ Han Meng ⋅ Junwei He ⋅ Longxuan YU ⋅ Radin Shayanfar ⋅ Yukyung Lee ⋅ Zhikang Dong ⋅ Allen G Hart ⋅ Anjiang Wei ⋅ Anurag Kashyap ⋅ Arpandeep Khatua ⋅ Audrey J Zheng ⋅ Chengrui Ma ⋅ David Heineman ⋅ dubing ⋅ Hai-Anh Trinh ⋅ Haishuo Fang ⋅ Hefan Zhang ⋅ Hui Shen ⋅ Issa Sugiura ⋅ Jiankai Sun ⋅ Jiechao Gao ⋅ Junhong Lin ⋅ Junnan Li ⋅ Kai Yang ⋅ Lei Hsiung ⋅ Maoyu Wang ⋅ Mengze Tang ⋅ Nabil Omi ⋅ Negin Raoof ⋅ Nicholas Edwards ⋅ Ruohao Guo ⋅ Orfeas Menis Mastromichalakis ⋅ Ryan Ji ⋅ Przemysław Hejman ⋅ Qi Qi ⋅ Qunshu Lin ⋅ Richard Zhuang ⋅ Rui Yang ⋅ Ruichen Zheng ⋅ Ryan Marten ⋅ Shaghayegh Fazliani ⋅ Shizheng Hou ⋅ Sicong Jiang ⋅ Sijie Li ⋅ Song Bian ⋅ Terry Yue Zhuo ⋅ Tianqing Wu ⋅ Tom Tang ⋅ Wanjia Zhao ⋅ Weihao Xuan ⋅ Wenhua Liang ⋅ Xian Liu ⋅ Xin Lan ⋅ Ruby Zhang ⋅ Xuandong Zhao ⋅ Yanchuan Tang ⋅ Yifan Jiang ⋅ Yijiang Li ⋅ Yitong Guan ⋅ Yizhi Li ⋅ YONGHUI LIU ⋅ Yuheng Tang ⋅ Yujun Mao ⋅ Yunfei Zhao ⋅ Yuxin Wang ⋅ Yuxuan Tang ⋅ Zhenheng Tang ⋅ Zhifei Li ⋅ Ziruo Wang ⋅ Ziyu She ⋅ Kaiyuan Liu ⋅ Iheb Chaabane ⋅ Yuxin Tang ⋅ Xiangyi Li ⋅ Soroush Vosoughi ⋅ Sanmi Koyejo ⋅ Di He ⋅ Etash Guha ⋅ Benjamin Feuer ⋅ Andy Konwinski ⋅ Boxuan Li ⋅ Leon L Chen ⋅ Alex Dimakis ⋅ Nicholas Carlini ⋅ Mike Merrill ⋅ Ludwig Schmidt ⋅ Alex Shaw
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of frontier models across 54 benchmarks, enabling a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Mix, a curated set of 100 difficult, diverse, and high-quality tasks refined from the adapted benchmarks. Harbor-Mix preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; the strongest model resolves only 15.6\% of its tasks. We release the adapters, evaluation results, in-depth analysis, and Harbor-Mix as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.
Show more
Beyond Drug Discovery: The Nanotechnology Molecular Optimization (NMO) Benchmark
Matthias Blaschke ⋅ Daniel Kienzle ⋅ Zsuzsanna Koczor-Benda ⋅ Julian Lorenz ⋅ Rainer Lienhart ⋅ Fabian Pauly
Generative molecular design is shaped by simple proxy benchmarks for drug-like properties and models pretrained on large pharmaceutical datasets. This combination drives strong benchmark metrics but limits transferability to domains structurally distinct from drug discovery. To overcome this limitation and drive discovery toward real, scientifically grounded targets, we introduce the Nanotechnology Molecular Optimization (NMO) Benchmark, which bridges machine learning (ML) and quantum materials science. NMO acts simultaneously as a rigorous testbed for the ML community and a discovery engine for nanotechnology research. The suite replaces proxy oracles with quantum simulations and introduces strict protocols that prioritize scientific utility over leaderboard-oriented overfitting. The physics-based NMO tasks impose hard structural constraints and rugged fitness landscapes, posing fundamentally new requirements on generative models. Notably, advanced molecular optimization methods underperform much simpler approaches on the NMO tasks. We develop a new baseline method identifying the critical components to solve the NMO tasks, including a novel representation for modeling structural constraints and a domain-agnostic pretraining strategy to eliminate pharmaceutical dataset bias. Our results surpass state-of-the-art physical properties and reveal previously unknown structural motifs, offering new insights for the nanotechnology community and demonstrating that ML can drive genuine scientific discovery.
Show more
Metropolis-Scale Road Network Datasets for Fine-Grained Urban Traffic Modeling
Fedor Velikonivtsev ⋅ Oleg Platonov ⋅ Ekaterina Alimaskina ⋅ Gleb Bazhenov ⋅ Liudmila Prokhorenkova
Modeling traffic dynamics is a critical challenge for urban computing, with applications from real-time traffic management to infrastructure planning. However, progress in this area is fundamentally constrained by a lack of large-scale public datasets that capture the subtle properties of real city road networks. Existing benchmarks are often limited by their small scale, reliance on sparse highway traffic sensors, absence of true road connectivity information, and lack of information about road properties. To address this issue, we introduce datasets representing fine-grained road networks of two major cities, which are unique in their scale (up to 100,000 road segments), use of real road connectivity, presence of time series measurements for both traffic speed and volume at a 5-minute resolution, and inclusion of rich static road attributes. These datasets enable in-depth analysis of spatiotemporal traffic patterns and can serve as benchmarks for various ML applications. As a practical demonstration of the utility of our datasets and the challenges they present, we use them for the task of traffic forecasting. The size of the real-world road networks in our datasets reveals significant scalability issues in current traffic forecasting models. To address them, we propose a simple and efficient baseline that not only scales to large road graphs but also achieves forecasting performance competitive with other established spatiotemporal models. We hope that the proposed datasets will serve as a foundational resource for a broad range of research in traffic modeling, urban computing, and smart city development.
Show more
MedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical Context
Hongjian Zhou ⋅ Xinyu Zou ⋅ Jinge Wu ⋅ Sean Wu ⋅ Junchi Yu ⋅ Bradley M Segal ⋅ Tobias E Niebuhr ⋅ Sara Amro ⋅ Michael Petrus ⋅ Sheikh Momin ⋅ Alexandra M Pinto ⋅ Rachel Niesen ⋅ Laura S Wegner ⋅ Dhruv Darji ⋅ Jung M Koo ⋅ Joshua Fieggen ⋅ Kapil Narain ⋅ Mingde Zeng ⋅ Lei Clifton ⋅ Linda Shapiro ⋅ Fenglin Liu ⋅ David Clifton
Large language models (LLMs) now reach expert-level scores on medical licensing exams, encouraging the assumption that high scores imply safe medical judgment while patients increasingly use them for health advice. We show this assumption is fragile: when misleading context is injected into questions that LLMs originally answer correctly, they abandon the correct answer. We call the ability to maintain correct judgment under adversarial context epistemic resilience, and introduce MedMisBench to measure it. MedMisBench contains 10,932 medical question items and 48,889 misleading context-option pairs spanning medical reasoning, agentic capability, and patient-journey evaluation. Across 11 model configurations, mean accuracy falls from 71.1% on original questions to 38.0% under focused misleading context, with 51.5% attack success. The most damaging injections are formal, rule-like fabrications: authority-framed falsehoods reach 69.5% attack success and exception-poisoning claims reach 64.1%. A 14-member clinical panel from 7 countries identified serious potential harm in 38.2% of reviewed cases. MedMisBench exposes a structural blind spot in LLM evaluation in medical settings: existing benchmarks measure what models know, but not whether they preserve correct medical judgment under misleading context.
Show more
Quantitative Assessment of Crystal Structure Prediction
Sergio Rincón ⋅ Gabriel González ⋅ Nicolás Andrade ⋅ Rafael Velasquez ⋅ Sebastian Ojeda ⋅ Juanita Puentes ⋅ Nicolás Aparicio ⋅ Paula Cárdenas ⋅ Pablo Arbelaez
Crystal Structure Prediction (CSP) aims to predict a material's 3D structure from its chemical composition, with applications from energy storage to pharmaceuticals. Despite its importance, existing evaluation frameworks suffer from key limitations: they rely on threshold-sensitive or incomparable metrics, focus predominantly on geometric performance while overlooking physical plausibility, and fail to account for polymorphism, where a chemical composition can crystallize into multiple stable structures. In parallel, recent Symmetry-Informed CSP (SICSP) methods often report results alongside standard CSP baselines while receiving crystallographic templates at inference, which constrain the search space and make SICSP a fundamentally different task. To address these limitations, we introduce an Orientation-invariant, Polymorph-Aware, Large Benchmark (OPAL-Bench) that evaluates CSP and SICSP as separate tasks. We evaluate 7 CSP and 2 SICSP methods under OPAL-Bench across 4 datasets using threshold-free geometric and physical distance metrics. Our results show that no method dominates across all metrics, and that geometric performance is not strongly correlated with physical plausibility. Motivated by this gap, we provide OPAL-CSP, an SE(3)-equivariant flow-matching baseline that performs competitively on geometric metrics and consistently ranks among the best on physical ones. By unifying geometric and physical metrics under a polymorph-aware protocol, OPAL-Bench enables a more comprehensive assessment of generative quality in CSP.
Show more
NPCBench: A Clinical Apprenticeship Benchmark for Guideline-Constrained Care-Pathway Reasoning in Nasopharyngeal Carcinoma
Pengkai Wang ⋅ Wei-Wei Zhang ⋅ Yan Li ⋅ Min Tang ⋅ Zhitian Hou ⋅ Zeyu Liu ⋅ Guanghao Zhu ⋅ Yuanyi Wang ⋅ Yanggan Gu ⋅ Wenjun Wang ⋅ Minheng Ni ⋅ Congkai Xie ⋅ Zhijie Sang ⋅ Jianmin Wu ⋅ Ying Sun ⋅ Hongxia Yang
Current medical LLM benchmarks typically decompose clinical competence into isolated factual questions, single-image interpretation, or brief case scenarios. This fragmented paradigm primarily measures local correctness but provides limited insight into whether models can progressively integrate subspecialty knowledge, multimodal evidence, and evolving patient states into safe, guideline-constrained longitudinal reasoning. To address this gap, we introduce NPCBench, a multimodal clinical apprenticeship benchmark for nasopharyngeal carcinoma (NPC). NPCBench operationalizes specialist training as a staged evaluation of (M)LLMs across guideline recall, atomic rule application, multimodal evidence grounding, longitudinal care planning, state revision, and expert-style consultation. It comprises 4,077 staged evaluation problems and 25 multimodal full-care path episodes, covering pathology diagnosis, MRI staging, and 146 downstream clinical decisions. The benchmark explicitly links stepwise tasks to complete patient-level care trajectories, enabling systematic evaluation of all clinical pathways. Beyond isolated precision, NPCBench evaluates whether the models ground decisions in multimodal evidence, maintain temporally consistent patient states, and provide safe expert-level consultation across complete care trajectories. Across 30 evaluated models, we observe a persistent composition gap: strong performance in local case-level tasks does not translate into episode-level clinical competence required for end-to-end patient management. Overall, NPCBench advances medical AI evaluation from static question answering toward apprenticeship-style workflow evaluation, providing a rigorous testbed for evidence-grounded and temporally coherent reasoning in complex oncology care.
Show more
Is Your LLM-as-a-Recommender Agent Trustable? LLMs' Recommendation is Easily Hacked by Biases (Preferences)
Zichen TANG ⋅ Zirui Zhang ⋅ Qian Wang ⋅ Zhenheng Tang ⋅ Xiaowen Chu ⋅ Bo Li
Current Large Language Models (LLMs) are gradually exploited in practically valuable agentic workflows such as Deep Research, E-commerce recommendation, job recruitment. In these applications, LLMs need to select some optimal solutions from massive candidates, which we term as \textit{LLM-as-a-Recommender} paradigm. However, the reliability of using LLM agents for recommendations is underexplored. In this work, we introduce a \textbf{Bias} \textbf{Rec}ommendation \textbf{Bench}mark (\textbf{BiasRecBench}) to highlight the critical vulnerability of such agents to biases in high-value real-world tasks. The benchmark includes three practical domains: paper review, e-commerce, and job recruitment. We construct a \textsc{Bias Synthesis Pipeline with Calibrated Quality Margins} that 1) synthesizes evaluation data by controlling the quality gap between optimal and sub-optimal options to provide a calibrated testbed to elicit the vulnerability to biases; 2) injects contextual biases that are logical and suitable for option contexts. Extensive experiments on both SOTA (Gemini-\{2.5,3\}-pro, GPT-4o, DeepSeek-R1) and small-scale LLMs reveal that agents frequently succumb to injected biases despite having sufficient reasoning capabilities to identify the ground truth. These findings expose a significant reliability bottleneck in current agentic workflows, calling for specialized alignment strategies for LLM-as-a-Recommender. The complete code and evaluation datasets will be made publicly available upon acceptance.
Show more
Forecasting Microbial Dynamics: Evaluation Protocol and Prior-Spectrum Benchmark
Fedor Sergeev ⋅ Anna Győrffy-Kerekes ⋅ Tristan Gollmart ⋅ Vincent Fortuin ⋅ Andre Kahles ⋅ Gunnar Rätsch
Modern forecasting methods are often designed and evaluated on datasets that do not reflect the realities of real-world clinical time-series: inter-patient variability, feature constraints, irregularity, and sparse sampling. Microbiome dynamics modeling represents these challenges, has seen little attention from the machine learning community, and has practical significance in medicine. Moreover, it offers both deep domain knowledge and sufficient data to train deep models, making it a strong testbed for a wide variety of forecasting methods. We propose a unified evaluation protocol for forecasting microbial dynamics and conduct *MicroCastBench*: a benchmark for models covering a wide range of priors (from mechanistic to foundation models) and assumptions (from per-patient to meta-learning models). We find that meta-learning models lead across most scaling regimes and at longer horizons, with mechanistic priors winning at small cohort sizes, short trajectories, and high feature counts. These results suggest that when the target data is far from pretraining distributions, domain-informed and meta-learning approaches can outperform generalist foundation models, highlighting the value of evaluating on under-represented domains.
Show more
OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning
Zhongyu Yang ⋅ Jiale Tao ⋅ Ruitao Chen ⋅ Zuhao Yang ⋅ Yingfang Yuan ⋅ Xueliang Zhao ⋅ Auden ⋅ Kai Wang ⋅ Shuai Shao ⋅ Biao Wang ⋅ Steve Yves ⋅ Qinglin Lu
Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio–visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.
Show more
Auditing Correlated Failures in Frozen-Feature Pretrained-Encoder Pools for Medical Segmentation
Eunseob Choi ⋅ Kyeonghun Kim ⋅ Hyuk-Jae Lee ⋅ Nam-Joon Kim
Cross-architecture/backbone medical-segmentation pools are commonly motivated as complementary failure coverage when members make different mistakes; that assumption is rarely tested at the per-case level. We audit this assumption for a fixed pool of 11 pretrained-encoder bundles read out by a deliberately weak $1\times1$-convolution decoder (4 seeds), and for a small set of decoder-recipe perturbations on the same bundles. Across five primary task rows (Kvasir polyp, ACDC LV, BraTS foreground, RIGA Cup, RIGA Disc), this 11-bundle $1\times1$-conv readout pool yields per-case Dice correlation gaps of $\Delta=0.261\text{--}0.638$ between same-bundle and cross-bundle pairs after a Dice $\geq 0.30$ functional floor; all paired case and hierarchical bootstrap intervals exclude zero. The result is stable under floor sweeps, ICC(A,1), leave-one-out, subject aggregation, quality-controlled pair regression, and cross-fitted item-difficulty residualisation. Decoder-recipe diagnostics show that richer readouts attenuate the gap (UNet-skip; case-identical nnU-Net on RIGA Cup: $0.473\to 0.0079$, separate-split $0.034\text{--}0.047$); the reported magnitudes are recipe-specific. Additional probes (pixel error, joint-failure lift, same-architecture-family ViT, same-family cross-checkpoint, same-backbone) locate the dependence structure but do not causally separate architecture, pretraining, and recipe. The claim is scoped to 2D frozen/light-adaptation encoder pools with this readout family and to retrospective benchmark audits. We release per-case Dice traces, summary JSONs, scorer code, hashes, and metadata as the audit artifact.
Show more
ECG Dataset with Multi-Expert Annotations and Delineations
Aram Avetisyan ⋅ Shahane Tigranyan ⋅ Sergey Skorik ⋅ Ariana Asatryan ⋅ Aleksandr Beznosikov ⋅ Yury Markin
The availability of open 12-lead electrocardiogram (ECG) datasets has significantly increased the number of studies focusing on automated ECG analysis. However, existing datasets often have limitations due to unverified annotations, overly processed data, or the absence of features such as fiducial points of the ECG. In this paper, we present a comprehensive collection of ECGs, with each record meticulously annotated by a team of seven cardiologists. Over 2,500 ECG records were annotated, with most records reviewed by at least four physicians. 1,527 ECG records of the dataset contain detailed annotations of PQRST segments, verified by two cardiologists. The dataset constitutes the largest collection of manually delineated ECG records to date, offering a valuable resource for researchers and practitioners in cardiology. This dataset, the first of its kind available for open use with multi-expert annotations, is suited for a variety of research tasks. It can be used for ECG abnormality classification, for analyzing patterns and annotation complexities across different heart abnormalities, and for tasks related to ECG delineation.
Show more
AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation
Priyam Sahoo ⋅ Gaurav Mittal ⋅ Xiaomin Li ⋅ Shengjie Ma ⋅ Benjamin Steenhoek ⋅ Pingping Lin ⋅ Yu Hu
Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false. We evaluate 2,614 OpenHands trajectories from eight model backends on SWE-bench Verified. Of the 60 tasks in this corpus, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification. We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and release AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens combines two components. First, it merges multiple passing solutions for the same task into a PTA reference space of correct behaviors. Second, it uses a context-sensitive intent-stage labeler that assigns actions to Exploration, Implementation, Verification, or Orchestration using trajectory history rather than tool identity alone. On AgentLens-Bench, the composite score separates passing trajectories into Lucky, Solid, and Ideal tiers; decomposes Lucky Passes into five recurring mechanisms; and changes how the eight evaluated model backends are ranked compared with pass rate alone. Across these models, AgentLens classifies between 0.5% and 23.2% of successful trajectories as Lucky, and some models move by as many as five rank positions when ranked by quality score instead of pass rate. We release the anonymized project repository, including the AgentLens-Bench dataset and AgentLens SDK, at https://anonymous.4open.science/r/agentlens-app-6810/.
Show more
AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning
Jiarui Zhang ⋅ Junqi Hu ⋅ Zurong Mai ⋅ Yang Liu ⋅ Yuhang Chen ⋅ Loushuohong ⋅ Henglian Huang ⋅ Hong Cheng ⋅ Lingyuan Zhao ⋅ Huang Jianxi ⋅ Yutong Lu ⋅ Haohuan Fu ⋅ Juepeng Zheng
Modern agricultural data is sourced from diverse platforms and spans multiple spatial scales, ranging from ground-level close-up photography to Unmanned Aerial Vehicle (UAV) aerial observation and satellite remote sensing imagery. Accordingly, agricultural multimodal reasoning demands robust cross-scale spatial understanding. However, due to the lack of multi-view agricultural benchmark datasets, existing multimodal large language models (MLLMs) exhibit severe ground-level bias, which leads to scale confusion then semantic collapse in agricultural perception tasks, such as misinterpreting farmland imagery as walls or floors. To address this, we introduce \textbf{AgroOmni}, a large-scale multi-view training corpus with 288K Visual Question Answering pairs covering 56 specialized task categories across 14 task types, designed to capture diverse scales in modern precision agriculture. Built on this dataset, we propose AgroNVILA, which achieves a new state-of-the-art of 62.32\% on the AgroMind benchmark ($+15.03\%$ over GPT-5.2), effectively mitigating the multi-view cross-scale gap for holistic agricultural understanding. Diagnostic evaluations on AgMMU further reveal an inherent heterogeneity between macro-priors and micro-diagnostics through constrained zero-shot performance. Meanwhile, even minimal fine-tuning leads to a dramatic performance gain of AgroNVILA on AgMMU, strongly demonstrating its generalization capability empowered by AgroOmni. Full training scripts are publicly available at \url{https://anonymous.4open.science/r/AgroOmni-6510}.
Show more
Isharah-Selfie: Continuous Sign Language Recognition Dataset for One-handed Signing
Ahmed A Hasanaath ⋅ Murtadha Aljubran ⋅ Sarah Alyami ⋅ Muhammad Haris Khan ⋅ Hamzah Luqman
Current sign language recognition and translation benchmarks are commonly collected in controlled settings using fixed or studio-style cameras, particularly limiting their ability to reflect how signers communicate in everyday mobile scenarios. As such, this gap becomes increasingly evident when considering the Arabic Sign Language (ArSL), which is a primary sign language used across Arab countries. In this work, we introduce Isharah-Selfie, a large-scale ArSL dataset collected entirely using front-facing smartphone cameras under unconstrained selfie conditions. The dataset comprises 8,997 video clips performed by 16 signers across 1,493 unique sentences, covering several domains such as healthcare, education, and transportation. Unlike conventional datasets dominated by fixed-camera, two-handed signing, Isharah-Selfie captures realistic mobile communication characterized by one-handed signing, close-range viewpoints, partial body visibility, camera motion, unconstrained environments, and heterogeneous device resolutions. Each video is annotated with both gloss sequences for continuous sign language recognition (CSLR) and Arabic sentence translations for sign language translation (SLT). Moreover, we define signer-independent and unseen-sentences evaluation protocols to assess generalization across unseen signers and novel sentence compositions. Extensive benchmarks using RGB-based and pose-based data show that current CSLR and SLT methods remain challenged by selfie-style signing conditions, particularly under unseen-sentence evaluation, where compositional generalization remains weak. By releasing Isharah-Selfie, we aim to support research toward robust, user-centered sign language technologies that operate reliably in realistic smartphone-based communication settings. The Isharah-Selfie dataset is available on https://www.github.com/.
Show more
Benchmark for Assessing Olfactory Perception of Large Language Models
Eftychia Makri ⋅ Nikolaos Nakis ⋅ Laura Sisson ⋅ Geetanjali Minsky ⋅ Leandros Tassiulas ⋅ Vahid Satarifard ⋅ Nicholas A Christakis
Here we introduce the Olfactory Perception (OP) benchmark, designed to assess the capability of large language models (LLMs) to reason about smell. The benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name prompts consistently outperform isomeric SMILES, with gains ranging from +3.1 to +18.9 percentage points (mean $\approx$ +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular reasoning. The best-performing model reaches 64.4\% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory reasoning. We further evaluate a subset of the OP across 21 languages and find that aggregating predictions across languages improves olfactory prediction, with AUROC=0.86 for the best performing language ensemble model. LLMs should be able to handle olfactory and not just visual or auditory information.
Show more
AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding
Shiqi Shen ⋅ Shurui Li ⋅ Yuanning Li ⋅ Xilin Zhang
Visual mental imagery (VMI) decoding uses brain activity to reconstruct internally generated visual scenes, providing a potential pathway for externalizing internal experiences and developing neuroprosthetic communication interfaces. However, progress in this field is limited by the lack of dedicated, training-scale VMI datasets. Current VMI paradigms often rely on visual prompts or short-term memory images, introducing visual and mnemonic confounds that make it difficult to disentangle neural activity associated with endogenous image construction from signals driven by prior or concurrent visual input. Furthermore, existing VMI datasets offer limited descriptive control over individual visual attributes, making it difficult to systematically assess fine grained visual details. To address these gaps, we introduce AVID, the first dedicated training-scale 5.0 Tesla functional magnetic resonance imaging (fMRI) resource for auditory-induced VMI decoding. We further define a benchmark around AVID with fixed splits, predefined neural inputs, and a standardized evaluation protocol. Acquired with high-field 5T fMRI, AVID comprises approximately 67 hours of dense recordings from nine participants. The dataset is uniquely structured into two complementary splits: a scene-level complex split using naturalistic Mandarin descriptions and a controlled simple split with explicitly specified visual attributes. We further formalize a standardized evaluation protocol that decouples source-image consistency from semantic alignment, allowing for a more nuanced assessment of imagery reconstruction. Baseline qualitative and quantitative results show that current reconstruction pipelines remain limited on AVID, although the most successful reconstructions preserve recognizable scene level semantics from spoken descriptions. By combining dedicated training-scale neural recordings and standardized evaluation resources, AVID provides both a data foundation for imagery-specific model development and a benchmark for evaluating VMI decoding with substantially reduced sensory contamination.
Show more
fNIRSAtlas: A Large-Scale Benchmark for Functional Near-Infrared Spectroscopy Classification
Kalle Mäkelä ⋅ Tuukka Ruotsalo
Functional near-infrared spectroscopy (fNIRS) is a promising modality for brain-computer interfaces and cognitive state decoding, but progress in machine learning for fNIRS has been limited by small, isolated datasets and inconsistent evaluation protocols. We introduce fNIRSAtlas, the largest open benchmark for fNIRS classification to date, comprising 23 public datasets spanning diverse experimental paradigms, including motor execution, motor imagery, mental arithmetic, working memory, and emotion recognition. fNIRSAtlas provides a fully automated pipeline from raw data to evaluation and defines standardized benchmark tasks for both cross-subject and within-subject classification, ensuring reproducible comparison of methods. We evaluate a variety of baseline models from fNIRS and related neuroimaging and time-series machine learning literature on the benchmark. Our results demonstrate that performance varies substantially across datasets and that methods validated on limited data -- especially deep learning approaches -- often fail to generalize beyond their original evaluation. fNIRSAtlas serves as an extensible, open, and reproducible benchmark to support more robust and cumulative progress in machine learning for fNIRS data. Our code and data can be found at https://anonymous.4open.science/r/fNIRSAtlas-8842.
Show more
L-FAME: Longitudinal Focused Attention Meditation EEG Dataset and Benchmark
Angqi Li ⋅ Basit R Syed ⋅ Hamzeh Alzweri ⋅ Taosheng Liu ⋅ Barry H Cohen ⋅ Saiprasad Ravishankar
We introduce a novel Longitudinal Focused Attention Meditation Electroencephalography (L-FAME) dataset and an accompanying benchmark, designed to foster research into the neural effects of various meditation practices and the evolution of these effects over a six-week training period. The dataset contains EEG recordings and psychological assessments from 74 healthy college participants, collected at two distinct time points: pre-intervention and post-intervention. Participants were randomly assigned to one of three distinct meditation groups: two mantra-based techniques (SA-TA-NA-MA and Hare Krishna) and one Breath Focus practice. Leveraging this unique longitudinal and comparative dataset, we propose a benchmark suite comprising three distinct classification tasks: (1) cognitive state decoding to distinguish between resting and meditation states, (2) fine-grained classification of the specific meditation techniques, and (3) cross-session adaptation to evaluate model generalization across the longitudinal time gap. We provide comprehensive baseline results for these tasks utilizing a range of classical machine learning algorithms and deep learning architectures. The complete dataset, preprocessing pipelines, and benchmark evaluation code will be publicly released, offering a valuable resource and a standardized framework for the development and comparison of new analytical methods in computational meditation research and EEG-based machine learning.
Show more
OS-Omni: A Cross-Platform Benchmark for Generalist Computer-Using Agents
Hui Shen ⋅ Yunta Hsieh ⋅ Jianing Ma ⋅ Ziyuan Liu ⋅ Qi Han ⋅ Xiuqi Xu ⋅ Yanheng shang ⋅ Chenqi Yin ⋅ Zesen Zhao ⋅ Boyuan Zheng ⋅ Jisen Li ⋅ Xiaoxia Wu ⋅ Ruoyan Zhang ⋅ Dongfu Jiang ⋅ Ruiyao Liu ⋅ Jing Xiong ⋅ He Xiao ⋅ Nan Zheng ⋅ Hangxiang Chen ⋅ Shengyang Tao ⋅ Deyang Luorong ⋅ Jingxuan Zhang ⋅ Chaofan Tao ⋅ Yuren Wang ⋅ Jiaqi Mo ⋅ Hei Ting Chan ⋅ Sicheng Chen ⋅ Kai Zhang ⋅ Mengkang Hu ⋅ Zhongwei Wan ⋅ Xin Wang ⋅ Chuanyang Zheng ⋅ Ziheng Zhang ⋅ Hengyuan Zhang ⋅ Zunhai Su ⋅ Kanzhi Cheng ⋅ Jiawei Lu ⋅ Yifan Zhang ⋅ Shen Yan ⋅ Ben Athiwaratkun ⋅ Qiushi Sun ⋅ Mi Zhang ⋅ Ping Luo ⋅ Wenhu Chen ⋅ Ngai Wong
Computer-using agents are increasingly expected to complete realistic user goals rather than isolated GUI actions. However, existing evaluations are fragmented across web, mobile, and desktop settings, making it difficult to measure whether the same agent can sustain long-horizon workflows across heterogeneous software platforms while preserving final-state correctness. We introduce OS-Omni, a 1,200-task cross-platform benchmark and execution environment for evaluating computer-using agents across Windows, Ubuntu, macOS, iOS, Android, and Web environments. OS-Omni provides reproducible initialization, platform-specific adapters, trajectory logging, and executable final-state evaluators, and includes controlled service-backed applications for workflows involving account, cart, budget, calendar, message, and other persistent state. Human studies show that OS-Omni tasks require 67.54 interaction steps, substantially exceeding OSWorld references of 15.0 steps. Throughout our experiments, the strongest evaluated agent, GPT-5.5, reaches 47.1% average success, trailing the 88.1% human reference by 41.0 points; on hard workflows, it drops from 77.8% success on easy tasks to 16.4%, while humans remain at 76.3%. These results indicate that current agents remain far from platform-robust, long-horizon computer use, especially when success requires sustained state tracking and durable final-state changes.
Show more
MindGames:A Multi-Agent Benchmark and Trajectory Dataset for Evaluating Social and Strategic Reasoning in LLMs
Kevin Wang ⋅ Benjamin Kempinski ⋅ Anna C. M. Thöni ⋅ Xin Yuan Cheng ⋅ Jianzhu Yao ⋅ Benjamin Finch ⋅ Leon Guertler ⋅ Viraj Nadkarni ⋅ Yihan Jiang ⋅ Cheston Tan ⋅ Maria Polukarov ⋅ Pramod Viswanath ⋅ Leshem Choshen ⋅ Mathieu Lauriere ⋅ Tal Kachman ⋅ Zhangyang "Atlas" Wang
Large language models (LLMs) are increasingly deployed as interactive agents, yet their capacity for social and strategic reasoning over extended interaction remains poorly understood. Existing evaluations rely on static vignettes or single-game benchmarks that cannot capture the sustained, multi-faceted reasoning that real-world multi-agent settings demand. We introduce Mindgames, a multi-game arena and evaluation platform for LLM agents that operationalizes complementary reasoning demands relevant to ``theory of mind'': belief attribution under hidden information, opponent modeling through repeated strategic interaction, cooperative inference under knowledge asymmetries, and sustained deception in social deduction. Built on TextArena, Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments. We instantiate Mindgames through a 2025 competition cycle hosted at a major AI conference, which assessed 944 submitted agents from 76 teams across four games: Colonel Blotto, Iterated Prisoner's Dilemma, Codenames, and Secret Mafia. Our analysis surfaces both agent-level and evaluation-level limitations: brittle rule adherence remains a major bottleneck, top-performing systems repeatedly rely on explicit structural scaffolding, and leaderboard validity differs sharply across environments. In particular, failure-heavy environments can reward robustness to opponent errors as much as strategic ability, with Secret Mafia exhibiting a pronounced error-survival confound in this cycle. We release a dataset of 29,571 multi-agent games (comprising 94,132 player trajectories and 243M tokens) with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol that scores new agents against a frozen reference pool of top-ranked, low-error Stage~II submissions under the same error-attribution lens used in this analysis. By releasing the benchmark interface, dataset, MG-Ref, and starter kit, Mindgames provides a reproducible resource for studying progress in multi-agent social and strategic reasoning.
Show more
QDMouse4M: A Multi-View 3D Mouse Spontaneous Behavior Dataset with Quantum-Dot Markers
Jingyang Ke ⋅ Amartya Pradhan ⋅ Xueling Zhang ⋅ Weihan Li ⋅ Anqi Wu ⋅ Jeffrey Markowitz
Quantitative analysis of animal behavior in neuroscience increasingly relies on accurate 3D pose, yet current large-scale mouse datasets are restricted to top-down views and suffer from occlusion and keypoint ambiguity. We introduce QDMouse4M, a six-view dataset of freely moving mice with physically grounded 2D and 3D keypoint annotations obtained from subdermal quantum-dot fluorescence markers. QDMouse4M contains over four million reflectance frames with matched fluorescence frames, 2D and 3D pose trajectories and behavior classifications. We use the dataset to evaluate markerless 2D and 3D pose estimation with SLEAP and Lightning Pose 3D across training-set sizes, backbones, and in-session/out-of-session splits. We further show that the released trajectories support downstream spontaneous-behavior analysis with Keypoint-MoSeq and stride-level gait measurements. QDMouse4M provides a benchmark-scale, physically grounded resource for developing and evaluating pose estimation, behavior understanding, and biomechanics models under realistic dark-environment laboratory conditions.
Show more
CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
Austin T D Andrews ⋅ Liam Wilkinson ⋅ Jamie Heagerty ⋅ Harry Coppock ⋅ Jakob Foerster ⋅ Rui Costa
We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode spans 300+ turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a model ranking: aggregate outcomes do not reliably discriminate models at this scale. Instead, we introduce two interface-level metrics that the environment makes measurable: Proactive Monitoring Rate (PMR), capturing whether agents actively query latent strategic state, and RAG@10, capturing whether commitments stated in structured planning reflections are executed within ten subsequent turns. Across runs we observe two consistent patterns under a shared playbook protocol. Agents under-monitor strategically relevant state that is available but requires explicit querying: despite playbook guidance to query victory progress every 20 turns, agents do so only every 30–75 turns, and in 7 of 20 detectable defeats they failed to query within the 20-turn warning window before game end. Agents also frequently fail to execute near-term commitments stated in their own planning reflections (RAG@10 between 48% and 66% across models). Both patterns arise despite tool access and explicit guidance, and we interpret them as deviations under instruction rather than absences of capability. We release the environment, scenarios, logs, metrics, and analysis pipeline at https://anonymous.4open.science/r/civbench/README.md.
Show more
RoboProcessBench: Benchmarking Process-Aware Understanding in Visual-Language Robotic Manipulation
Dayu Xia ⋅ Yue Shi ⋅ Yao Mu ⋅ Huiting Ji ⋅ Chaofan Ma ⋅ Yingjie Zhou ⋅ Hua Chen ⋅ Yang Liu ⋅ Jiezhang Cao ⋅ Guangtao Zhai
Vision-language models (VLMs) are increasingly explored as visual critics, reward generators, and failure detectors in robotic manipulation. These roles implicitly require models to judge not only final task success, but also how a manipulation execution is physically and temporally progressing. However, existing evaluations fail to test whether VLMs possess fine-grained process understanding. To address this gap, we present RoboProcessBench, a benchmark for process-aware understanding in vision-language robotic manipulation. RoboProcessBench decomposes such capability into two complementary dimensions, $\textit{static monitoring}$ and $\textit{dynamic reasoning}$, instantiated as 12 diagnostic question families covering phase, contact, motion, coordination, primitive-local progress, temporal order, outcome, and primitive-level transitions. Built from physically grounded execution traces, the curated benchmark corpus ProcessData contains ~58k question-answer pairs across 260 manipulation tasks, which is further split into ProcessData-SFT and ProcessData-Eval for post-training and evaluation purposes. Extensive evaluation of various VLMs on ProcessData-Eval reveals broad limitations across 12 diagnostic task families, suggesting current models still lack robust process-aware understanding of manipulation executions. But with ProcessData-SFT, the post-trained $\textit{Qwen2.5-VL-7B}$ and $\textit{InternVL-3-8B}$ exhibit consistent gains on local state, motion, progress, and primitive-aware cues. These results demonstrate that RoboProcessBench serves as both an evaluation benchmark and a learnable supervision source for developing VLMs capable of monitoring and evaluating robotic manipulation processes. Dataset artifacts (https://huggingface.co/datasets/ProcessBench-2026/RoboProcessBench) and executable evaluation suite (https://anonymous.4open.science/r/RoboProcessBench-F875/) are released for reproducibility.
Show more
JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments
Zhixi Cai ⋅ Fucai Ke ⋅ Sukai Huang ⋅ Maria Garcia de la Banda ⋅ Peter Stuckey ⋅ Reza Haffari ⋅ Hamid Rezatofighi
In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be distributed across time, viewpoint, and interacting objects. A model may therefore give a plausible answer without ever observing the relevant object, time, or view that supports it. Current visual reasoning benchmarks largely evaluate passive observations and final answers, overlooking settings that require active reasoning and evidence acquisition. We introduce JRDB-AVR, a benchmark derived from existing real-world JRDB robotics data through a structured question-generation engine that turns this gap into an explicit evaluation: an embodied agentic system receives a visual reasoning question, requests bounded observations by timestamp and viewing angle, and is evaluated on both the final answer and the grounded visual evidence supporting it. The benchmark contains diverse questions over multiple real-world environments involving temporal search, viewpoint selection, and human-oriented compositional reasoning. We also introduce JRDB-AVR-Agent, a reference active reasoning agentic method that maintains an explicit observation-grounded graph-based world model and answers through solving. Experiments reveal a substantial gap between answer accuracy and evidence accuracy in current baselines, showing that current VLMs can produce unsupported correct answers and that active evidence-aware evaluation is necessary for embodied visual reasoning. Code and benchmark are available at https://github.com/ControlNet/JRDB-AVR.
Show more
The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset
Richard Schwarzkopf ⋅ Fabian Immel ⋅ Alexander Blumberg ⋅ Jonas Merkert ⋅ Nils A Rack ⋅ Kaiwen Wang ⋅ Fabian Konstantinidis ⋅ Julian Truetsch ⋅ Carlos Fernandez ⋅ Annika Bätz ⋅ Kevin Rösch ⋅ Marlon Steiner ⋅ Willi Poh ⋅ Yinzhe Shen ⋅ Felix Hauser ⋅ Dominik Strutz ⋅ Jaime Villa ⋅ Gleb Stepanov ⋅ Royden Wagner ⋅ Omer Sahin Tas ⋅ Frank Bieder ⋅ Holger Caesar ⋅ Jan-Hendrik Pauls ⋅ Christoph Stiller
Existing autonomous driving datasets have enabled major progress, but fall short in sensor fidelity, map completeness, or geographic diversity. We present KITScenes Multimodal, a European dataset built around high-fidelity sensors and maps. Our fully synchronized sensor suite combines high-resolution global-shutter cameras, long-range lidar beyond 400 meters, 4D imaging radar, and redundant GNSS/INS localization. Our HD maps are, to our knowledge, the most complete of any sensor dataset, verified through autonomous driving trials on open-source software. For the first time in a public dataset, all driving-relevant traffic elements, such as traffic lights, are mapped in 3D to a reprojection-accurate level with full topological connectivity. Recorded in cities with irregular street layouts and mixed traffic modes, our dataset complements existing datasets by broadening the available geographic diversity. We also introduce four benchmarks, each advancing spatial learning for embodied AI: online HD map construction, long-range depth estimation, novel view synthesis, and end-to-end driving.
Show more
Exploring the Limits of Compositional Generalization in Vision-Language-Action Manipulation
Xupeng Zhang ⋅ Yantai Yang ⋅ Chang Guo ⋅ Zhaokai Yin ⋅ Zhipeng Zhang
Vision Language Action (VLA) models can execute isolated atomic skills, but practical long horizon manipulation requires recombining these skills into new compositions without demonstrations for every skill combination. The direct full instruction route treats skill chaining as long horizon imitation. In our controlled study, full instruction training on limited skill combination sequences reaches 40.0% ordered success on covered combinations but only 7.7% on recombination, suggesting memorization of sequence templates rather than robust skill recombination. We therefore introduce Hierarchical Subtask Execution (HSE), a diagnostic execution framework that exposes skill recombination through a shared modular interface over the planner, trigger, and policy, enabling long horizon failures to be decomposed into separable modes. To make this diagnosis measurable, we introduce RoboCombine, a benchmark for skill recombination without combined skill training trajectories, with metrics for ordered success and transition conversion across subtask boundaries. RoboCombine reveals that atomic competence alone does not ensure ordered execution. Failures concentrate at transitions where downstream subtasks must launch from handoff states produced by preceding subtasks rather than canonical atomic starts. We identify this mismatch as the start state gap. To address it, we introduce Pose Robust Task Initialization (PRTI), a lightweight atomic task post training strategy that broadens start state coverage beyond canonical starts. Under oracle timed HSE on RoboCombine, adding PRTI raises ordered success from 2.11% to 26.32% and transition conversion from 2.63% to 60.53%, without combined skill training trajectories. Together, these results show that combinatorial generalization requires handoff robust atomic skills that convert first stage progress into reliable downstream execution, not merely decomposition into known subtasks. We will release RoboCombine and HSE code to enable reproducible diagnosis of atomic skill recombination failures.
Show more
Benchmarking Optimizers for Large Language Model Pretraining
Andrei Semenov ⋅ Matteo Pagliardini ⋅ Martin Jaggi
The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those methods are myriad: from faster convergence to removing reliance on certain hyperparameters. However, the diverse experimental protocols used to validate these claims make direct comparisons between methods challenging. This study presents a comprehensive evaluation of recent optimization techniques across standardized LLM pretraining scenarios, systematically varying model size, batch size, and training duration. Through careful tuning of each method, we provide guidance to practitioners on which optimizer is best suited for each scenario. For researchers, our work highlights promising directions for future optimization research. Finally, by releasing our code and making all experiments fully reproducible, we hope our efforts can help the development and rigorous benchmarking of future methods.
Show more
Reset Dependence Is Embodied: Training Protocol Changes Which Morphologies Appear Optimal
Kevin Qiu ⋅ Zhao Yang ⋅ Marek Cygan
In a 100-body robot co-design benchmark, the height threshold that resets a falling robot changes which bodies appear optimal. We train PPO controllers for DERL-evolved UNIMAL morphologies under Standard (τ = 0.20), Strict (τ = 0.50), and Reset-free (τ = 0) protocols, then evaluate every policy under a common protocol. Strict and Reset-free training share only 5 of their top-10 bodies and have low rank agreement (Spearman ρ = 0.21). The effect is family-specific, not a uniform penalty: averaging per-body retention ratios, Reset-free training reduces Floor bodies' per-step reward by 45% relative to Standard training but reduces variable-terrain bodies by 16%. Under Strict, the losses are 22% for Floor and 34% for variable-terrain bodies, while multi-task bodies are most stable across protocols. Blocked ANOVA and morphology-level permutation tests support the protocol-by-family interaction. A SAC validation panel planned before analysis finds the same Strict-versus-Reset-free family ordering under a different optimizer. Selecting bodies by their minimum reward across the three training protocols recovers ≥91% of every single-protocol oracle's top-k average reward (k ∈ {5, 10, 20}; all bootstrap lower bounds ≥84.3%). Co-design benchmarks should state the reset rule, audit ranking sensitivity, and report protocol-robust selections when choosing bodies.
Show more
Efficient Benchmarking Is Just Feature Selection and Multiple Regression
Sam Bowyer ⋅ Acyr Locatelli ⋅ Kris Cao
Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a benchmark's questions. By reframing this problem as an instance of multiple regression with feature selection, we find that existing efficient benchmarking methods can be greatly improved by simply using kernel ridge regression at the prediction stage. Additionally, using an information-theoretic feature-selection algorithm called minimum redundancy maximum relevance (mRMR), we can further improve upon these methods by selecting question subsets that will be maximally useful for prediction. Except in very data-poor settings, these approaches consistently achieve smaller prediction errors (in both MAE and RMSE), and greater ranking correlation between predicted and true scores (in both Spearman $\rho$ and Kendall $\tau$) across a range of benchmarks using both binary and continuous metrics. Furthermore, mRMR subsampling is much faster than competitor methods (which often involve fitting probabilistic models or running clustering algorithms), and is more likely to select the same questions under different random seeds or training data splits.
Show more
Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability.
Roberto Stanzione ⋅ Jules Barbe ⋅ Magali Parrino ⋅ Jérémie Fourmann ⋅ Paul Boniol
Time Series Anomaly Detection (TSAD) has received increasing attention, driven by the growing availability of complex time series data. This surge has led to the development of numerous detection methods, as well as a variety of benchmarks aimed at thoroughly evaluating their performance. However, most existing detectors remain largely agnostic to domain context, overlooking explainability and interpretability. One of the main reasons for this gap is that current benchmarks primarily focus on detection accuracy, and only few of them evaluate spatial explainability. Moreover, no benchmark currently provides sufficiently rich semantic annotations to support the generation of human-understandable interpretations of anomalies. To address these limitations, we introduce SHAD (Scality High-dimensional Anomaly Detection benchmark), a fully annotated benchmark composed of 144 multivariate, high-dimensional time series collected from real-world distributed cloud storage systems operated by Scality. The proposed dataset includes rich contextual information, covering three families of anomalies with varying degrees of severity. As further contribution, we provide an end-to-end benchmark that covers all stages of a TSAD pipeline. For Detection, we benchmark a wide range of existing anomaly detectors, testing their effectiveness on the proposed real-world dataset. Then, we evaluate explainability by measuring the contribution of each dimension in the generated anomaly score. Finally, for interpretability, we investigate the effectiveness of frozen LLM baselines in localizing and interpreting anomalies, paving the way for future research towards interpretable and explainable TSAD.
Show more
EvalAwareBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
Xinning Li ⋅ Kemunto Ochwang'i ⋅ Ram Bharadwaj Aryasomayajula ⋅ Alexandra Souly ⋅ Robert Kirk
Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave differently in evaluations than in deployment, this undermines the validity of evaluation results, which are a crucial component of current AI safety frameworks. We introduce EvalAwareBench, an open pipeline and benchmark for measuring evaluation awareness that works with any Inspect-compatible evaluation, allowing practitioners to test against current and future benchmarks. EvalAwareBench ships with a newly curated transcript suite covering current frontier system-card evaluations and diverse deployment sources. The benchmark serves two purposes: measuring how reliably frontier LLMs recognize that they are being evaluated, and assessing how detectable individual benchmarks are as evaluations. We identify two methodological choices in the existing literature that introduce systematic bias: the identity of the model that generated the deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankings; and elicitation prompts selected for high performance on one model can perform near chance on others. EvalAwareBench corrects for both via per-model probe calibration and a stratified generator-harmonisation procedure. We release EvalAwareBench as an open pipeline and dataset.
Show more
CSBench: A Comprehensive Benchmark for Evaluating Project-Level System Construction in Computer Science
Hongli Yu ⋅ Huan-ang Gao ⋅ Botian Wang ⋅ Hanlin Wu ⋅ YiFei Wang ⋅ Daoguang Zan ⋅ Huang Anni ⋅ Weinan Dai ⋅ Qiying Yu ⋅ Jiaze Chen ⋅ Wei-Ying Ma ⋅ Ya-Qin Zhang ⋅ Jingjing Liu ⋅ Mingxuan Wang ⋅ Hao Zhou
While LLM-based coding agents have advanced to repository-scale engineering, existing benchmarks predominantly focus on software maintenance, overlooking the foundational capability of system construction. We introduce CSBench, a benchmark evaluating project-level Spec-to-Code implementation across core computer science domains (e.g., OS, compilers, databases). CSBench features 100 expert-curated tasks from top-tier university assignments within containerized environments. Evaluating 12 state-of-the-art LLMs reveals a stark performance disparity: even top-tier models like GPT-5.2 see success rates plummet from over 90% in algorithmic tasks to below 35% in system-level domains. Our analysis identifies a fundamental deficiency in systemic reasoning and construction knowledge as the primary bottleneck, where agents either terminate prematurely due to overconfidence or fail to translate execution feedback into valid architectural fixes, highlighting CSBench as a necessary tool to bridge this gap and advance autonomous agents toward deep computer science mastery.
Show more
Beyond High and Low: Evaluating Graded Cognitive Diversity in LLM Persona Simulations
Sion Weatherhead ⋅ Ben R Newell ⋅ Aaron Belbasis ⋅ Flora Salim
Large language model agents are increasingly used as synthetic participants in social, behavioural, and decision-making research, yet it remains unclear whether they can express graded cognitive diversity rather than collapsing toward generic competence or stylised persona descriptions. We introduce a cognition-suite evaluation for LLM persona simulation, testing whether models reproduce population-level variation across structured reasoning tasks, self-report cognitive/personality measures, cross-construct associations, and downstream behavioural reasoning. Agents are conditioned on synthetic or human-derived trait profiles spanning cognitive reflection, Need for Cognition, and Big Five personality, then evaluated on CRT-style reasoning items, trait questionnaires, and Behavioural Reflection Task scenarios requiring evidence weighting, risk evaluation, and socially embedded justification. Across four matched simulation runs, agents partially recover questionnaire-style Need for Cognition and Big Five structure, especially under numeric prompting, but fail to preserve cognitive-reflection fidelity: CRT-style scores collapse toward correctness, intuitive-lure errors are almost eliminated, and reflection-related association structure weakens. Downstream BRT responses show the same pattern of flattening: open-ended decisions compress into over-regularised behavioural profiles rather than preserving human-like variation. The evaluation reframes persona simulation as a problem of cognitive-diversity fidelity: useful synthetic populations must preserve not only demographic or personality descriptors, but the structured variation in reasoning style, error type, and behaviour that those descriptors are meant to organise.
Show more
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
Maonan Wang ⋅ Zhengyan Huang ⋅ Kemou Jiang ⋅ Yuhang Fu ⋅ Jiayue Zhu ⋅ Yuxin Cai ⋅ Xingchen Zou ⋅ Qiaosheng Zhang ⋅ Yi Yu ⋅ Ding Wang ⋅ XI CHEN ⋅ Ben Chen ⋅ Yuxuan Liang ⋅ Zhiyong Cui ⋅ Man On Pun ⋅ Yirong Chen
Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics. However, existing traffic-oriented multimodal benchmarks largely emphasize passive visual recognition or isolated video understanding, offering limited support for evaluating structure-aware traffic reasoning under controlled conditions. We introduce \textbf{OmniTraffic}, a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. Built around 12 real-world intersections reconstructed into editable 3D traffic environments and complemented by surveillance footage from two countries, OmniTraffic supports both controlled and natural-condition evaluation. It defines a three-level task hierarchy spanning scene perception, multi-view and temporal reasoning, and decision support. Using structured traffic metadata, OmniTraffic generates synchronized multi-view VQA samples covering vehicle states, lane functions, view--BEV correspondence, temporal dynamics, and signal-phase analysis, resulting in 8M VQA samples and a 3K human-verified test set. Evaluation of eleven frontier MLLMs reveals a large human--model gap, with the most pronounced failures in topology-grounded and spatio-temporal reasoning tasks. Fine-tuning a lightweight MLLM on simulated OmniTraffic data further improves performance on real-world traffic scenes, demonstrating the value of simulation-generated supervision for traffic-specific multimodal reasoning. Beyond a fixed dataset, OmniTraffic provides an extensible pipeline with configurable intersections, camera views, traffic demands, signal phases, visual conditions, and rare events.
Show more
RoutingBench: Can Agentic Routing Analysis Scale to Production Datacenter Networks?
Wenlong Ding ⋅ Zhixiong Niu ⋅ Jianan Yang ⋅ Fajun Zhang ⋅ Bo Zhang ⋅ Ling Liang ⋅ Yongqiang Xiong ⋅ Tianyin Xu ⋅ Hong Xu
Recent advances in AI models and agentic technologies make AI for network operations (NetOps) within reach. However, scalability remains a key bottleneck of agentic NetOps when analyzing hyperscale networks, which comprise hundreds of datacenters, each housing thousands of network devices. The scalability challenge is rooted in the requirement of many NetOps tasks that must conduct global reasoning on how a local change of device behavior affects all relevant routing paths, known as routing-path analysis. This paper studies this scalability problem and evaluates how different agentic approaches, namely in-context learning, iterative reasoning, and agent skills, can scale routing-path analysis to large, complex networks. We present RoutingBench for evaluating agentic routing-path analysis, with varying network size and complexity, for various types of device changes. Our results show that agentic analysis is promising—agent skills curated with a principle termed “explore more; digest less” enables routing-path analysis on hyperscale networks of 50K routers with an accuracy of 99.5%, significantly outreaching the scalability of traditional symbolic analysis. Meanwhile, RoutingBench also reveals the boundary of AI agent capability on complex inter-datacenter networks and compound changes, posing open challenges for AI and agentic research.
Show more
DDx-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs
Jiazhen Pan ⋅ Weixiang Shen ⋅ Jun Li ⋅ Julian Canisius ⋅ Felix Bitzer ⋅ Paula Roßmüller ⋅ Jiancheng Yang ⋅ Virginie Kreutzinger ⋅ Daniel Rueckert ⋅ Benedikt Wiestler
Medical diagnosis is not a single prediction from a fully specified vignette. It is a sequential workup: clinicians decide what evidence to obtain, revise a differential diagnosis, and stop when the diagnosis is sufficiently supported. Most medical AI benchmarks instead reveal the relevant context upfront and score only the final answer, making unsupported correct guesses, premature closure, inefficient workups, and poor uncertainty updating invisible. We introduce DDx-TRACE, a physician-adjudicated benchmark for multimodal neuroradiology that evaluates diagnostic trajectories under hidden evidence over 211 challenging cases. Each case begins with limited clinical history; models request imaging studies in free form, receive matched image bundles when available, update a probabilistic differential diagnosis after each turn, and stop with a localized final diagnosis. Evaluating state-of-the-art VLMs, we find that final diagnosis scores can substantially misrepresent workup quality: models may guess plausible diagnoses without essential evidence, request useful studies but misinterpret raw images, or acquire evidence inefficiently while updating uncertainty poorly. Controlled evidence variants isolate bottlenecks in planning, visual evidence extraction, and downstream differential reasoning. DDx-TRACE shifts medical AI evaluation from final answers to evidence-supported diagnostic trajectories.
Show more
BoardGameArena: A Multi-Dimensional Benchmark for Strategic Reasoning of LLMs in Board Games
Wenxiao Zhao ⋅ Shu Wang ⋅ Renxi Wang ⋅ Yi Yi ⋅ Yasi Zhang ⋅ Chengyuan Ma ⋅ Deqian Kong ⋅ Pan Lu ⋅ Ying Nian Wu ⋅ Benjamin Yao
Large language models (LLMs) excel at reasoning across diverse domains, from mathematical problem-solving to code generation, yet struggle with strategic reasoning that requires long-term planning, opponent modeling and integrating tactical calculation with strategic understanding. Board games, with their fully observable states, perfect-information environments and objectively verifiable decisions, offer ideal testbeds for probing these capabilities. We introduce Board Game Arena (BGA), a multi-dimensional benchmark that evaluates LLMs' strategic reasoning along three cognitive axes—deductive, inductive, and abductive reasoning—through three challenging task types across five classic board games, yielding 45 sub-datasets with over one million samples. We evaluate 28 state-of-the-art LLMs on BGA and find that current models exhibit fundamental limitations in strategic reasoning for board games. While models demonstrate competence in understanding game rules and identifying legal moves, they struggle significantly with selecting optimal moves and applying strategic principles or tactical analysis. These results position BGA as a challenging and diagnostic benchmark for tracking progress in LLMs' strategic cognition.
Show more
HDL-RepoBench: Multi-Paradigm Repository-Level Code Completion for Hardware Design Languages
Qingyun Zou ⋅ Jiahao Cui ⋅ Nuo Chen ⋅ Bingsheng He ⋅ Weng-Fai Wong
Large language models (LLMs) have achieved strong performance on code completion in general-purpose programming languages, but existing repository-level benchmarks focus almost exclusively on software code and largely overlook hardware description languages. We present HDL-RepoBench, consisting of HDL-RepoBench-Train and HDL-RepoBench-Eval, the first benchmark for multilingual hardware code completion at the repository level with functional evaluation. HDL-RepoBench covers three major hardware design coding styles and annotates each completion target with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. Post-training billion-scale code LLMs on HDL-RepoBench-Train enables smaller open-weight models to outperform much larger general-purpose models under matched context windows. Beyond aggregate accuracy, our analysis surfaces four findings that distinguish hardware from software code completion: (i) performance varies sharply across HDLs, with HLS consistently easiest and VHDL hardest across functional pass rate and EM/ES; (ii) hardware accuracy is non-monotonic in code-structure depth, in contrast with the monotonic depth–accuracy curve known for software; (iii) accuracy plateaus once the input context exceeds 2,048 tokens, while software completion continues to benefit from longer context; and (iv) certain repository-level retrievers tuned on software underperform a no-retrieval baseline on HDL repositories because their structural and dependency assumptions are violated by hardware code.
Show more
AssayBench: An Assay-Level Virtual Cell Benchmark for LLMs and Agents
Edward De Brouwer ⋅ Carl Edwards ⋅ Alexander P Wu ⋅ Jenna L Collier ⋅ Graham Heimberg ⋅ Xiner Li ⋅ Meena Subramaniam ⋅ Ehsan Hajiramezanali ⋅ David Richmond ⋅ Jan-Christian Huetter ⋅ Sara Mostafavi ⋅ Gabriele Scalia
Recent advances in machine learning and large-scale biological data collections have revived the prospect of building a virtual cell, a computational model of cellular behavior that could accelerate biological discovery. One of the most compelling promises of this vision is the ability to perform in silico phenotypic screens, in which a model predicts the effects of cellular perturbations in unseen biological contexts. This task combines heterogeneous textual inputs with diverse phenotypic outputs, making it particularly well-suited to LLMs and agentic systems. Yet, no standard benchmark currently exists for this task, as existing efforts focus on narrower molecular readouts that are only indirectly aligned with the phenotypic endpoints driving many real-world drug discovery workflows. In this work, we present AssayBench, a benchmark for phenotypic screen prediction, built from 1,920 publicly available CRISPR screens spanning five broad classes of cellular phenotypes. We formulate the screen prediction task as a gene rank prediction for each screen and introduce the adjusted nDCG, a continuous metric for comparing performance across heterogeneous assays. Our extensive evaluation shows that existing methods remain far from empirically estimated performance ceilings and zero-shot generalist LLMs outperform biology-specific LLMs and trainable baselines. Optimization techniques such as fine-tuning, ensembling, and prompt optimization can further improve LLM performance on this task. Overall, AssayBench offers a practical testbed for measuring progress toward in-silico phenotypic screening and, more broadly, virtual cell models. Our benchmark is available at https://github.com/Genentech/AssayBench.
Show more
OroPrecipBench: a km-scale benchmark for spatial precipitation downscaling over complex terrain
Hongyi Chen ⋅ Xiaokai Yan ⋅ Jiadong Zhang ⋅ Jingtao Ding ⋅ Xiaojun Liang ⋅ Yong Li ⋅ Xiao-Ping (Steven) Zhang
Precipitation downscaling over complex terrain is essential for resolving water availability, flood risk, and extremes, but current evaluations rarely test whether models learn orographic structure rather than only pixel-wise accuracy. We introduce OroPrecipBench, a km-scale benchmark for orographic precipitation downscaling across five U.S.\ terrain regimes. The benchmark provides a shared-domain dual-track protocol: a perfect-model track that coarsens 3\,km HRRR analyses to isolate controlled super-resolution skill, and an observation-grounded track that downscales 25\,km ERA5 to 4\,km PRISM to test practical robustness under source-target shift. We also introduce PEP-D, a terrain-conditioned diagnostic that measures errors in elevation--precipitation profiles, complementing pointwise, spatial, and extreme-precipitation metrics. Evaluating ten statistical, deterministic deep-learning, and generative baselines shows that strong controlled super-resolution does not reliably translate to real-case or cross-regime downscaling. Models that reduce pixel-wise error can still distort terrain-dependent precipitation and upper-tail rainfall, and terrain-conditioned structure is especially fragile under transfer. OroPrecipBench exposes these failure modes and motivates source-shift-robust, terrain-constrained, and tail-aware downscaling methods. Benchmark is available \href{https://anonymous.4open.science/r/OroPrecipBench-612D}{here}.
Show more
MM-DyGraph: A Dataset, Benchmark, and Model for Multimodal Dynamic Graphs
Zongyuan Wu ⋅ Pengjie Wang ⋅ Lingshan Chen ⋅ Tianhang Wan ⋅ Yuan Meng ⋅ Xin Wang ⋅ Ling Feng ⋅ Wenwu Zhu
Multimodal dynamic graphs are pervasive in real-world applications, where nodes and edges are accompanied by multimodal signals, including text and images. Despite their practical importance, benchmarks and methods that explicitly study multimodal learning in dynamic graphs remain scarce, hindering the full understanding and effective utilization of multimodal information for downstream tasks. To address this gap, we introduce MM-DyGraph, to the best of our knowledge, the first benchmark for multimodal dynamic graph learning, consisting of diverse real-world datasets with fine-grained temporal graph evolution and rich multimodal attributes. MM-DyGraph provides three evaluation tasks to comprehensively assess model performance in dynamic settings. We benchmark five state-of-the-art dynamic graph methods and reveal that naive feature concatenation yields target-level inconsistencies: multimodal inputs benefit certain prediction targets while harming others. This observation indicates that the modality informativeness depends on the specific prediction target over time. Building on these observations, we propose a Target-aware Spatial-Temporal Multimodal Fusion (TST-MF) framework, which jointly conditions multimodal fusion on the prediction target and spatial-temporal relations, enabling context-adaptive cross-modal interplay for target-specific representations. Extensive experiments over diverse datasets and tasks show that TST-MF delivers robust and competitive performance, achieving clear gains in multimodal settings.
Show more
The AI Observatory: A Public Measure of Real-World AI Use
Shayne Longpre ⋅ Anka Reuel-Lamparth ⋅ Dayeon Ki ⋅ Zhiping Zhang ⋅ Cathy Fang ⋅ Chuanyang Jin ⋅ Jennifer Mickel ⋅ Cedric Whitney ⋅ Niloofar Mireshghallah ⋅ Victor Ojewale ⋅ Megan Richards ⋅ Ariel N. Lee ⋅ Alex Pentland ⋅ Tianshi Li ⋅ Yuntian Deng ⋅ Sara Hooker ⋅ Mykel J Kochenderfer ⋅ Sanmi Koyejo
Understanding the benefits, risks, and impacts of general-purpose AI assistants requires looking at real conversations---not curated benchmarks or surveys. Yet AI use narratives rely on limited proprietary reports, or on few data sources, fragmented across platforms, models, and time. We introduce the AI Observatory, a public measurement platform that aggregates the most comprehensive set of real AI conversation sources, and develop a common taxonomy of 145 features, spanning function, topic, sensitive use, interaction style, multi-turn dynamics, and conversation structure. Across 23,158 conversations and 85,633 turns, we find that real AI use is highly heterogeneous: sources differ substantially in tasks, structures, and sensitive-use distributions, with no single source that generalizes. We further show that occupational summaries such as Anthropic's Clio capture an important but incomplete slice of use: 48\% of the conversations we analyze would be filtered out for being non-occupational, omitting substantial personal, social, cultural, and safety-relevant interaction. Real AI use is also not static: we find tokens and turns per conversation all rise sharply between 2023 and 2025, and we expose how model variants within the same developer support distinct usage regimes. The Observatory provides a new taxonomy, reusable annotation tools, and a public platform for reproducible measurement of how AI assistants are really used and evolving in the wild: \url{https://project-ai-observatory.vercel.app/}.
Show more
TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows
Shuai Fu ⋅ Jing Gu ⋅ Jian Zhou ⋅ Zicheng Duan ⋅ Gengze Zhou ⋅ Qi Wu
Recent text-to-image models have made substantial progress in realism, aesthetics, and prompt-image alignment. Yet visually appealing images can still violate real-world plausibility, exhibiting malformed object structures, impossible anatomy, physically implausible interactions, or impossible spatial relationships. These failures are not well captured by existing quality, aesthetics, preference, or alignment metrics. To address this gap, we introduce __TerraVis__, a framework for evaluating __world-grounded visual consistency__ in generated images. TerraVis defines a structured taxonomy of world-consistency violations covering object-, interaction-, and scene-level failures, and evaluates images through a multi-stage workflow. Specifically, TerraVis uses an MLLM to first perform eligibility checking, determining whether an image is suitable for world-consistency evaluation. It then conducts taxonomy-guided violation detection over fine-grained violation types and classifies detected violations as minor or severe to derive an overall world-consistency score. Across diverse open-source and proprietary text-to-image models on two widely used benchmarks, TerraVis shows the strongest correlation with human judgments of world consistency among the evaluated metrics. Our results further reveal that models with high quality, aesthetics, preference or alignment scores can still exhibit frequent world-consistency failures. TerraVis therefore provides a complementary evaluation perspective, enabling fine-grained diagnosis of generated images and model comparison beyond existing evaluation dimensions.
Show more
Can 4D Foundation Models Remember?
Guangzhao He ⋅ Hadar Averbuch-Elor ⋅ Wei-Chiu Ma
Perceiving and remembering the visual world is fundamental to navigating and interacting with our environment. Current 4D foundation models, such as camera-controllable video models or 4D reconstruction models, can perceive and reconstruct dynamic environments, but how well they remember what they have perceived remains an open question. Existing benchmarks largely rely on pixel-level metrics and lack ground truth for objects once they leave the field of view, making them unable to evaluate visual memory in an object-centric manner against references. To fill this gap, we introduce PersistBench, a dataset and metric suite that leverages 360° videos as omniscient ground truth and proposes three evaluation aspects: object permanence, motion continuity, and appearance preservation. Evaluating various models across diverse categories reveals that current models can only maintain short-term consistency that degrades significantly once objects leave the field of view. Our findings highlight the gap between current model capabilities and robust visual memory, providing guidance for future development of 4D foundation models.
Show more
Are Text-to-Image Models Inductivist Turkeys? A Counterfactual Benchmark for Causal Reasoning
jiayi lei ⋅ Yuandong Pu ⋅ Xingyu Han ⋅ Rongpeng Zhu ⋅ Jing Xu ⋅ Jinyao Wang ⋅ Zijian Zhou ⋅ Bin Fu ⋅ Yuewen Cao ⋅ Yihao Liu ⋅ Hongsheng Li
Text-to-image (T2I) generation models have achieved remarkable progress in producing visually realistic images from natural language prompts. Yet it remains unclear whether their success reflects genuine causal understanding or sophisticated pattern matching over visual-textual correlations. Inspired by Russell’s inductivist turkey, we introduce Counterfactual-World (CF-World), a counterfactual benchmark designed to probe whether text-to-image models can generate images under rules that systematically contradict real-world priors. CF-World organizes each scenario into three progressive levels: factual generation under ordinary world knowledge, explicit counterfactual generation with direct visual instructions, and implicit counterfactual generation requiring causal deduction from altered rules. We evaluate both open-source and closed-source T2I models using a Vision Language Model(VLM)-based evaluator (CF-Eval), and introduce two metrics: Prior Resistance Rate (PRR), which measures a model’s ability to overcome entrenched real-world priors, and Reasoning Retention Rate (RRR), which assesses whether models can maintain reasoning-dependent counterfactual generation without explicit visual cues. Experiments show that all models exhibit sharp degradation from factual to counterfactual settings. Further analyses suggest that these failures arise because current T2I models encode world knowledge and visual appearances as tightly coupled patterns. They rely heavily on frequent visual co-occurrences in their training data, causing them to default to familiar commonsense priors when tasked with generating counterfactual worlds.
Show more
Apple-$\pi$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence
Runmao Yao ⋅ Kairui Hu ⋅ Yukang Cao ⋅ Ruisi Wang ⋅ Shulin Tian ⋅ Ziang Cao ⋅ Weichen Fan ⋅ Ziqi Huang ⋅ Yuhao Dong ⋅ Hao Li ⋅ Zhaoxi Chen ⋅ Zhongang Cai ⋅ Lei Yang ⋅ Ziwei Liu
Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce **Apple-$\pi$**, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-$\pi$ comprises three components. **1) Orchard**: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. **2) Benchmark Protocol**: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses *chain-of-frames* prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. **3) Evaluation Suite**: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-$\pi$ as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.
Show more
VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing
Zhiming Luo ⋅ Hebaixu Wang ⋅ Haonan Guo ⋅ Jing Zhang ⋅ Bo Du
Recent advancements in Multimodal Large Language Models (MLLMs) have enabled complex reasoning. However, existing remote sensing (RS) benchmarks remain heavily biased toward perception tasks, such as object recognition and scene classification. This limitation hinders the development of MLLMs for cognitively demanding RS applications. To address this, we propose a Vision Language ReaSoning Benchmark (VLRS-Bench), which is the first benchmark exclusively dedicated to complex RS reasoning. Structured across the three core dimensions of Cognition, Decision, and Prediction, VLRS-Bench comprises 2,000 question-answer pairs with an average question length of 130.19 words, spanning 14 tasks and up to eight temporal phases. VLRS-Bench is constructed via a specialized pipeline that integrates RS-specific priors and expert knowledge to ensure geospatial realism and reasoning complexity. Experimental results reveal significant bottlenecks in existing state-of-the-art MLLMs, providing critical insights for advancing multimodal reasoning within the remote sensing community.
Show more
BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
Elaine Lau ⋅ Markus Dücker ⋅ Ronak Chaudhary ⋅ Hui Wen Goh ⋅ Rosemary Wei ⋅ Vaibhav Kumar ⋅ Saed Qunbar ⋅ Guram Gogia ⋅ Yi Liu ⋅ Scott Millslagle ⋅ Nasim Borazjanizadeh ⋅ Ulyana Tkachenko ⋅ Samuel E Danquah ⋅ Collin Schweiker ⋅ Vijay Karumathil ⋅ Asrith Devalaraju ⋅ Andrew P Martin ⋅ Varsha Sandadi ⋅ Haemi Nam ⋅ Punit Arani ⋅ Ray Epps ⋅ Abdullah Arif ⋅ Sahil Bhaiwala ⋅ Curtis Northcutt ⋅ Skyler Wang ⋅ Anish Athalye ⋅ Jonas Mueller ⋅ Francisco Guzmán
Existing AI benchmarks lack the fidelity to assess economically meaningful progress on professional workflows. To evaluate frontier AI agents in a high-value, labor-intensive profession, we introduce BankerToolBench (BTB): an open-source benchmark of end-to-end analytical workflows routinely performed by junior investment bankers. To develop an ecologically valid benchmark grounded in representative work environments, we collaborated with 502 investment bankers from leading firms. BTB requires agents to execute senior banker requests by navigating data rooms, using industry tools (market data platform, SEC filings database), and generating multi-file deliverables—including Excel financial models, PowerPoint pitch decks, and PDF/Word reports. Completing a BTB task takes bankers up to 21 hours, underscoring the economic stakes of successfully delegating this work to AI. BTB enables automated evaluation of any LLM or agent, scoring deliverables against 100+ rubric criteria defined by veteran investment bankers to capture stakeholder utility. Testing 9 frontier models, we find that even the best-performing model (GPT-5.4) fails nearly half of the rubric criteria and bankers rate 0% of its outputs as client-ready. Our failure analysis reveals key obstacles (such as breakdowns in cross-artifact consistency) and improvement directions for agentic AI in high-stakes professional workflows.
Show more
quanda: An Interpretability Toolkit for Training Data Attribution Evaluation
Dilyara Bareeva ⋅ Galip Ümit Yolcu ⋅ Anna Hedström ⋅ Niklas Schmolenski ⋅ Thomas Wiegand ⋅ Wojciech Samek ⋅ Sebastian Lapuschkin
Training data attribution (TDA) methods estimate the influence of individual training samples on a model's predictions, offering a principled lens for interpreting neural networks. Despite rapid methodological progress, TDA evaluation remains scattered across papers and rarely reproducible, making fair comparison between methods challenging. We introduce **quanda**, a Python toolkit that standardizes TDA evaluation. **quanda** features a comprehensive collection of evaluation metrics and ready-to-use benchmarks spanning image classification, text classification, and language modeling, alongside a uniform interface for integrating diverse TDA implementations. We showcase **quanda** through a comparative study of existing TDA methods and find that no approach excels uniformly across metrics, with clear trade-offs between ground-truth faithfulness and downstream task performance. By consolidating the tooling around TDA, **quanda** equips the community to evaluate attribution methods rigorously and reproducibly. The toolkit is thoroughly tested, documented, and available as an open-source library on PyPI and at https://anonymous.4open.science/r/quanda.
Show more
$\mathrm{Bounded\mbox{-}RS}$: A Bounded Evaluation Protocol and Testbed for Representational Systematicity
Boyu Cao ⋅ Wenshuo Wang
Systematicity is the capacity to generalize compositionally. Existing benchmarks primarily test behavioral systematicity, while representation-level analyses often leave open whether recovered signals are merely decodable or causally used during generation. We therefore introduce $\mathrm{Bounded\mbox{-}RS}$, an evaluation artifact and protocol for adjudicating these possibilities under bounded hypotheses: unsupported, present-only, causal-use, or underdetermined. The artifact is a controlled semantic-parsing setting with matched recombination tests, exact intervention pairs, and evidence routines for recoverability, interchange, necessity, and repair during generation. We apply $\mathrm{Bounded\mbox{-}RS}$ to the two prominent representation-level explanations of systematic generalization, binding and geometry. Within the family instantiations evaluated here, and under the same behavioral anchor and state semantics, the binding family satisfies the criteria for $\mathit{Causal\mbox{-}use}$, whereas the geometry family is adjudicated as $\mathit{Unsupported}$ under its frozen closure basis. Detector-truthfulness tests do not validate a geometry detector, so the geometry conclusion remains a claim about the tested family rather than about all possible geometric mechanisms. $\mathrm{Bounded\mbox{-}RS}$ thus provides a reusable way to make disciplined representational claims from behavioral systematicity tests.
Show more
Before Words, Beyond Speech: Evaluating Nonverbal Social Reasoning in Early Childhood
Marie Huynh ⋅ Laura Bravo-Sánchez ⋅ Lauren K Dubin ⋅ Nick Haber ⋅ Philip A Fisher ⋅ Serena Yeung-Levy
Children's earliest years are a period of intense social learning, during which meaning is built through gaze, gesture, and physical contact. Yet, this critical aspect of social interaction remains under-evaluated in multimodal video understanding, as existing benchmarks prioritize the conversation-driven interactions of adults. We address this gap with NEST ($\textbf{N}$aturalistic $\textbf{E}$arly-childhood $\textbf{S}$ocial in$\textbf{T}$eractions), the first child-centric benchmark for nonverbal social reasoning. NEST comprises 1,208 manually annotated clips ($\sim$12s) curated for diversity across naturalistic settings, cultural contexts, and developmental stages. Beyond a lack of data, targeted model improvement in this domain is hindered by coarse evaluation schemas that conflate basic perception with higher-order reasoning. To resolve this, NEST introduces a compositional schema that builds atomic interaction cues into progressive levels of social reasoning: contextual, behavioral, interpersonal, and open-ended descriptions. Extensive analysis of state-of-the-art Vision-Language Models (VLMs) reveals a stark performance collapse: despite strong contextual recognition, models exhibit substantial gaps in decoding precise behavioral and interpersonal cues. We trace these failures to specific biases, including poor consequence grounding and temporal sensitivity. Finally, we ground NEST in real-world applications by evaluating VLMs using domain adaptation strategies. Ultimately, NEST provides a rigorous testbed for advancing socially grounded artificial intelligence in early childhood. The dataset and code will be publicly released.
Show more
FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation
Chenxi Zhang ⋅ Ziliang Gan ⋅ Liyun Zhu ⋅ Youwei Pang ⋅ Qing Zhang ⋅ Rongjunchen Zhang
The financial domain poses substantial challenges for Vision-Language Models (VLMs) due to specialized chart formats and knowledge-intensive reasoning requirements. However, existing financial benchmarks are largely single-turn and rely on limited question formats, which hinders comprehensive evaluation in realistic application scenarios. To address this gap, we propose FinMTM, a multi-turn multimodal benchmark that expands diversity along both data and task dimensions. On the data side, we curate and annotate 11,133 bilingual (Chinese and English) financial QA pairs grounded in commonly used financial visuals, including candlestick charts, statistical plots, and report figures. On the task side, FinMTM covers single and multiple choice questions, multi-turn open-ended dialogues, and agent-based tasks. We further design task-specific evaluation protocols, including a set-overlap scoring rule for multiple choice questions, a weighted combination of turn-level and session-level scores for multi-turn dialogues, and a composite metric that integrates planning quality with final outcomes for agent tasks. Extensive experimental evaluation of 22 VLMs reveal their limitations in fine-grained visual perception, long-context reasoning, and complex agent workflows.
Show more
SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?
Yueyi Yang ⋅ Haotian Liu ⋅ Fang Kang ⋅ Mengqi Zhang ⋅ Zheng Lian ⋅ Hao Tang ⋅ Haoyu Chen
We explore the use of large language models (LLMs) for next-utterance anticipation in human dialogue. Despite recent advances in LLMs demonstrating their ability to engage in natural conversations with users, we show that even leading models surprisingly struggle to anticipate a human speaker’s next utterance. Instead, humans can readily anticipate forthcoming utterances based on multi-modal cues—such as gestures, gaze, and emotional tone—from the context. To systematically examine this gap, we propose SayNext-Bench, a benchmark evaluating MLLMs on anticipating context-conditioned responses across diverse real-world scenarios. To support it, we build SayNext-PC, a large-scale multimodal dialogue dataset, and carefully design a multi-level evaluation framework spanning lexical similarity, emotion-intention consistency, and LLM-based overall alignment. Building on this, we develop SayNext-Chat, a cognitively inspired dual-route MLLM that incorporates learnable priming tokens to fuse perceptual cues with anticipatory priors. Extensive experiments demonstrate that SayNext-Chat consistently outperforms state-of-the-art MLLMs across all evaluation levels, corroborated by user studies and LLM-as-Judge evaluations. Our results emphasize the (i) indispensable role of multimodal cues and (ii) active anticipatory processing as foundations of natural human interaction currently missing in MLLMs.
Show more
ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark
Joel Julin ⋅ Souraja Kundu ⋅ Liza Dahiya ⋅ George Z Wei ⋅ Louise Xie ⋅ Jingru Yi ⋅ Xu Zhang ⋅ Hao Yang ⋅ László Jeni
Multimodal large language models (MLLMs) achieve impressive performance on audio-visual question answering, but correct answers do not necessarily imply perceptually grounded reasoning. Models may exploit linguistic priors or dataset shortcuts without attending to the visual regions and audio events that actually justify their predictions. Existing benchmarks predominantly evaluate final answer accuracy, providing limited insight into whether models truly “see” or “hear” the evidence required for reasoning. We introduce ANCHOR, a benchmark and evaluation framework for auditing perceptual grounding in multimodal reasoning. ANCHOR requires models to produce grounded chain-of-thought explanations that explicitly reference audio-visual evidence, including representative frames, spatial bounding boxes, and salient audio cues. To build a reliable gold standard, we combine automated annotation generation with a human-in-the-loop refinement pipeline, producing 224 human-verified videos and 975 question–answer pairs with aligned reasoning and localization annotations from 2.3K videos and 8.2K QA pairs. We further propose a metric and evaluation protocol that jointly measure answer correctness and reasoning-grounding alignment, complemented by multi-judge scoring of hallucination, logical divergence, final answer, and a token-based conciseness measurement. Experiments on nine state-of-the-art MLLMs reveal a substantial faithfulness gap: models with strong answer performance often produce explanations that are only weakly grounded in the underlying perceptual evidence. Ablation results confirm the importance of explicit grounding cues: removing bounding boxes reduces accuracy by 7.7 points, and performance drops from 57.7% with full cues to 16.1% in the audio-only setting. These findings show that answer-only evaluation is insufficient for multimodal reasoning and position ANCHOR as a new testbed for developing systems that are accurate, interpretable, and trustworthy.
Show more
FrameSelect: A Unified Library for Video Frame Selection and Evaluation
Arihant Jain ⋅ Chaosheng Dong
Frame selection---picking a small set of informative frames from a video to feed a vision-language model (VLM)---sits at the front of every video-VLM system, yet the literature is fragmented: most works do not share a candidate pool, encoder, frame budget, or answering VLM, making published gains hard to compare across works. We introduce FrameSelect, to the best of our knowledge the first Python library that treats the selector as a standalone, interchangeable component, cleanly separated from the encoder that computes representations and the VLM that answers. Any registered selector composes with any registered evaluator through a single call, and adding a new selector, dataset, encoder, or VLM backend is an isolated extension; a replay mode further rebuilds any table under a different answering VLM without re-running selection. We release reference implementations of 20 selectors, 15 evaluators spanning answer-level and selection-level metrics, and the first unified benchmark comparing frame selectors under a shared protocol across seven evaluators spanning both evaluation modes. This comparison surfaces findings that existing studies cannot: rankings invert across benchmarks. Our library is available at https://anonymous.4open.science/r/FrameSelect-D2A4.
Show more
Beyond Visual Boundaries: Rethinking Scene Segmentation for Movie RAG
Dong-Hee Kim ⋅ Seonwoo Choi ⋅ Changbeen Kim ⋅ Jungmyung Wi ⋅ Juyeon Ko ⋅ Youngju Choi ⋅ Il Hyeon Mun ⋅ Hyunwoo J. Kim ⋅ Donghyun Kim
Understanding long-form video remains a fundamental challenge for multimodal large language models (MLLMs). Sparse frame sampling fails to capture fine-grained visual details, while dense sampling quickly exceeds context length limits. Retrieval-augmented generation (RAG) offers a promising middle ground by selectively retrieving relevant video segments for grounded generation, yet its effectiveness critically depends on the quality of the video segments used as retrieval units. In this paper, we investigate RAG for movie understanding, which demands story-level reasoning over characters, events, and narrative arcs spanning hours of content. Scene segmentation, a long-studied problem that partitions movies into semantically coherent units, is a natural candidate for defining such retrieval units. We reexamine whether existing methods actually serve this role through comprehensive evaluation on downstream movie understanding tasks, and find that they consistently fail to outperform naive uniform temporal chunking. Our audit of the most standard scene segmentation benchmarks reveals why: current annotations prioritize visually salient transitions over narrative event structure. Motivated by this mismatch, we introduce NarraScene, a narrative-centric scene segmentation dataset annotated with a three-level cognitive taxonomy spanning physical, character, and narrative change, where every valid boundary requires a narrative-level shift. When used as retrieval units, these narrative-grounded segments outperform uniform chunking on downstream movie understanding tasks, suggesting that the central challenge for scene segmentation in movie RAG is not detecting boundaries, but identifying the narrative event units that matter for movie understanding.
Show more
VisEditBench: A Benchmark for Vector-Format Diagram Editing with Visual Instructions
Akito Taneguchi ⋅ Itsumi Saito ⋅ Haruto Yoshida ⋅ Jun Suzuki
We introduce a new task and benchmark dataset, VisEditBench, for understanding visual instructions in vector-format diagram editing. While recent advances have made progress in generating vector graphics from text or images, editing such diagrams remains relatively underexplored. Moreover, existing approaches predominantly rely on textual instructions, which are often unintuitive for specifying edits in structured visual content, particularly when users need to identify specific elements or indicate their spatial positions in a diagram. We propose visual instructions as a more intuitive interface for diagram editing, and construct a dataset of 694 samples with human-curated annotations overlaid on TikZ diagrams. The dataset covers multiple edit types as well as instructions with multiple editing operations. We evaluate 32 recent state-of-the-art multimodal models and find consistent performance degradation, particularly for edits involving global layout or diagram structure, compared to local operations such as element addition or deletion. Furthermore, instructions that include multiple editing operations tend to result in lower performance than single-edit cases. Even recent models struggle to achieve consistently accurate edits, highlighting the inherent difficulty of the task. VisEditBench provides a valuable benchmark for advancing research on visually grounded diagram editing and structured visual understanding.
Show more
Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation
Jihwan Hong ⋅ Woohyeon Park ⋅ Jaeik Kim ⋅ Jaeyoung Do
Video Object Segmentation (VOS) in complex and long videos is increasingly important for real-world applications, where target objects often appear only intermittently within long temporal horizons. However, existing benchmarks largely focus on temporally salient objects that remain visible for most of the video. To address this gap, we introduce FaVOS (A Benchmark for Video Object Segmentation with Fractional Temporal Visibility), a benchmark designed to evaluate VOS methods under low temporal visibility. We show that, in this regime, the standard $\mathcal{J}$\&$\mathcal{F}$ metric can collapse VOS evaluation into absence classification, because empty predictions receive high rewards on target-absent frames. Consequently, even a trivial empty-mask predictor can outperform strong models such as SAM 3, revealing a fundamental mismatch between current metrics and practical VOS performance. To mitigate this issue, we propose Volumetric $\mathcal{J}$\&$\mathcal{F}$, which evaluates mask sequences as spatio-temporal volumes and reduces the dominance of target-absence rewards while preserving sensitivity to segmentation quality and temporal structure.
Show more
Reasoning-Aware IRT: Explainable Evaluation of Test-Time Scaling in Reasoning Models
Ratthachat Chatpatanasiri
Reasoning models increasingly depend on test-time computation, but raw accuracy conflates intrinsic ability, problem difficulty, guessing effects, and reasoning budget. Item Response Theory (IRT) adjusts for problem difficulty, yet collapses test-time compute into a single latent score. We propose reasoning-aware IRT, an extension in which each model's effective ability depends on baseline ability, reasoning-budget responsiveness, and overthinking curvature. Parameter estimation is performed via multi-pass generalized EM with post-hoc initialization refinement. On LiveBench with 42 language models, our method improves rank stability, gives stronger held-out correctness discrimination, and reveals interpretable structure unavailable to classical IRT: a unified benchmark--ability scale with quantified reasoning-token benefits, budget-efficient and overthinking regimes, and rank reversals obscured by raw scores. Our framework offers a compact, explainable evaluation tool for reasoning models under test-time scaling.
Show more
BrainEM: A Large-Scale and Diverse Benchmark for EM Neuron Segmentation in Connectomics
Zhenghua Li ⋅ Lanyue Zhang ⋅ Zekang Yang ⋅ Kai Li ⋅ Yuwei Peng ⋅ Song-Hai Shi ⋅ Xiaolin Hu
Connectomics requires dense segmentation of large-scale electron microscopy (EM) volumes using models trained on small labeled blocks. In practice, a connectomics lab facing a new raw volume has two options: annotate a small block and train on it, or skip annotation and train on the union of available public labeled datasets. We introduce BrainEM, a benchmark that formalizes these options as two standard tracks: the Same-Source Track (Annotate-and-Train) and the Cross-Source Track (Reuse-and-Train). BrainEM brings together 16 EM datasets, including one in-house mouse dataset that we contribute, spanning multiple species, imaging modalities, and resolutions. We further add large-scale test volumes that bring the evaluation closer to realistic practice, and four diverse test targets in the Cross-Source Track. The data, post-processing, and evaluation metrics are fixed across all methods, and the framework is fully model-data decoupled so that new methods can be added with minimal effort. We conduct a systematic evaluation of six representative baselines covering the main technical directions in EM neuron segmentation, making BrainEM the first systematic evaluation at large scale and across diverse cross-source targets. The results differ from prior small-scale evaluations: rankings on the Same-Source Track no longer match those reported on small CREMI splits, and methods diverge sharply on the Cross-Source Track. Both tracks still leave clear room for improvement. We release BrainEM as a public platform to support unified comparison and faster method development in connectomics. All datasets and code are available at https://github.com/kwinderic/BrainEM
Show more
Slide P2V-Bench: A Cross-Domain Benchmark for Slide-Centric Scientific Paper-to-Presentation Video Generation
Ligang Huang ⋅ Zhou Liu ⋅ Wentao Zhang
Scientific communication increasingly relies on slide-based narrated videos rather than static PDFs. However, transforming a paper into a presentation video is not a conventional text-to-video task. It requires preserving technical rigor, rendering dense multimodal assets faithfully, and synchronizing narration, subtitles, and cursor motion over long horizons. Current research faces two critical bottlenecks: (1) Benchmark Gap: existing evaluations are restricted to narrow domains or structured LaTex inputs with limited long-range supervision; (2) Evaluation Gap: current protocols often overlook the joint measurement of content fidelity, channel alignment, and actual knowledge transfer. To bridge these gaps, we introduce SLIDE P2V-BENCH, a cross-domain bench mark comprising 108 high-quality paper-video pairs spanning four domains and various durations, coupled with an auditable multi-dimensional evaluation suite. Furthermore, we propose PAPER2MEDIA, a slide-grounded agentic pipeline that produces editable presentations by synchronizing decoupled media channels via a shared semantic clock. Extensive experiments demonstrate that PAPER2MEDIA significantly outperforms existing automated baselines and, notably, achieves performance comparable to human-authored presentations in terms of narrative coherence, visual quality, and knowledge conveyance efficiency.
Show more
Beyond Downstream Scores: Controlled Diagnostics for Point-Cloud Self-Supervised Learning Evaluation
Kohsuke Ide ⋅ Ryousuke Yamada ⋅ Yue Qiu ⋅ Yoshihiro Fukuhara ⋅ Hirokatsu Kataoka ⋅ Yuki Asano ⋅ Yutaka Satoh
Downstream scores are the standard evidence for progress in point-cloud self-supervised learning (SSL). But a score is not a mechanism: it does not explain why a representation improves. We diagnose three simple questions in the evaluation pipeline: what signal is learned, what geometry is visible, and what the score measures. These questions test whether a gain comes from the pretext objective, from the geometric support available in the input, or from the downstream readout itself. Applied to cross-modal alignment, autoregressive prediction, and masked autoencoding, our diagnostics reveal that downstream scores often fail to identify the source of a gain. For Concerto, coordinate-only signals explain 28.9\% of the pretext-loss response but recover only 6.9\% of the downstream gain on ScanNet, separating pretext-loss explanation from representation transfer. For masked autoencoding, removing spatially coherent regions hurts much more than removing the same number of random points, showing that the visible geometry, not just the retained fraction, drives the score. For PointGPT, relaxing mask and order assumptions reduces transfer by at most 2.78 points on ScanObjectNN and 0.22 points on ShapeNetPart, showing that high scores can persist after weakening the intended autoregressive mechanism. These findings caution against treating downstream gains as self-evident representation progress in point-cloud SSL. We propose a diagnostic protocol that asks not only whether a score improves, but why it improves.
Show more
Predicting Species Splits: A Challenging Fine-Grained Benchmark for Category Discovery
Christian Lange ⋅ Joakim Bruslund Haurum ⋅ Bingchen Zhao ⋅ Oisin Mac Aodha
Widely used category discovery benchmarks have become increasingly saturated and often fail to reflect real-world discovery settings, making long-term evaluation of new methods difficult. We introduce \textbf{PS-SPLIT}, a large-scale fine-grained benchmark for category discovery, motivated by taxonomic splitting: the real-world process by which what was once considered a single species is found to comprise multiple distinct ones. PS-SPLIT contains 79K globally distributed, high-quality, community-collected bird images spanning 771 species, derived from 257 recent real taxonomic splits, with associated geographic metadata and full taxonomic hierarchy. With categories arising from real taxonomic splits, visual differences can be very subtle, often requiring significant human expertise to tell apart, making PS-SPLIT a challenging benchmark which we hope will assist long term in fine-grained category discovery research. Based on taxonomic splitting, we introduce the task of \textbf{discovery by category splitting}, where the goal is to determine which known categories should be subdivided into finer-grained subcategories. We adapt four generalized category discovery methods to this setting and show that there is substantial room for future progress.
Show more
When Are Predictions Enough? An Evaluation Protocol for Frozen Expert Composition
Alessandro Pereira ⋅ Keith J Ransom ⋅ Lewis Mitchell
Practitioners often combine independently trained models as a fixed pool of frozen experts. Standard combination methods, such as voting, averaging, or stacking, rely on final predictions, amounting to a few scalars per expert, and discard potentially richer cross-expert information found in the penultimate representations. An alternative is to combine these models in feature space. Whether this yields a meaningful improvement depends heavily on the evaluation design. Informal assessments on the same data can support materially different conclusions depending on baseline strength, overlap control, reporting granularity, and pool construction. We argue that auditing the sensitivity of evaluation outcomes to these design choices is critical for the effective composition of expert models. To address these sensitivities, we propose a four-part evaluation protocol: (1) a baseline ladder for prediction-space (PS) models; (2) a deterministic feature-space (FS) anchor plus a higher-capacity cross-check; (3) axis-isolated pools to disentangle performance gains; and (4) grouped evaluation with overlap-aware controls. Applying our protocol to 21 frozen deepfake detectors from DF40 (Yan et al., 2024), we find that a 10K-parameter logistic regression on raw concatenated features outperforms the strongest PS baseline by +0.123 AUROC on an architecture-diverse pool (generator-level bootstrap 95% CI [+0.078, +0.174]). Crucially, the higher-capacity cross-check (a 3.5M-parameter feature-space MLP) adds only +0.004 AUROC, suggesting that the gain arises from the richer feature interface rather than from combiner complexity. Prediction-space models suffice when individual expert performance is high but fall short in architecture-diverse, high-bottleneck pools. Our primary contribution is an evaluation-design audit showing that, on this benchmark and expert pool, relaxing any of these controls risks overstating the apparent FS advantage or creating the illusion of a gain that fails to hold across data subgroups. Our findings demonstrate that, without rigorous controls, the same data can support multiple, sometimes spurious, conclusions.
Show more
HiddenPathQA: A Benchmark for Knowledge Graph Question Answering When Questions Hide Their Paths
Sumin Jo ⋅ Edward Choi
Knowledge graph question answering (KGQA) systems are typically evaluated on questions whose surface tokens lexically expose the gold KG path. Real users, however, ask questions without knowing the underlying schema: "Which condition could Andrew have due to family history?" compresses a 4-hop chain (Andrew - father → father → medical condition → subclass of - hereditary disorder) into a single phrase (family history) whose tokens name none of those relations. We show that this collapse of path-question alignment is a blind spot of existing KGQA evaluation. The three dominant Large Language Model (LLM)-KG paradigms — hop-by-hop KG exploration, plan-then-retrieve, and similarity-based retrieval — all implicitly rely on the question's surface leaking the gold path, an assumption rarely tested in practice. We make three contributions. (i) HiddenPathQA, a human-validated KGQA benchmark of 1,055 multi-hop questions (2-6 hops, multiple domains) constructed so that the question surface does not leak the gold KG path's relations. (ii) Holistic Path Selection, a reasoning paradigm that exposes all candidate relation chains around the topic entity to the LLM at once, replacing per-hop relation scoring with question-against-full-chain alignment. Two implementations, TieredHolistic and FlatHolistic, outperform six strong baselines (ToG, PoG, FiDeLiS, R2-KG, KAPING, KARPA) on HiddenPathQA by +8.8 to +17.9 points over the strongest baseline across all four (backbone × split) settings. (iii) A pseudoword stress test that replaces entity surfaces with placeholder tokens. It both confirms HiddenPathQA items are solvable from KG structure alone — not from entity-level priors — and reveals that several baselines rely on entity memorization rather than KG-grounded reasoning.
Show more
SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery
Jiajun Jiang ⋅ Chunliang Hua ⋅ Zichun Chen ⋅ Yanxing Wu ⋅ Zeyuan YANG ⋅ Jie Song ⋅ Xiao Hu
Urban uncrewed aerial vehicle (UAV) vision-language navigation (VLN) requires agents to follow instructions across extended urban spaces, inherently demanding long-term memory and geospatial grounding. However, scaling existing benchmarks remains difficult because of their reliance on costly reconstructed 3D assets, limiting geographic diversity and episode scale. To address this, we introduce SatNav, a scalable, long-horizon UAV VLN benchmark built from high-resolution satellite imagery. SatNav targets city-level navigation missions and uses satellite crops as approximations of UAV nadir views for visual observations. Through an automated cue-to-episode pipeline, SatNav constructs 118K episodes from 59 scenes across 18 cities, with an average trajectory length of 379 m. To stress-test long-horizon memory and geospatial reasoning, SatNav defines three task families: Boundary, Landmark, and Route, targeting loop progress tracking, landmark-based spatial grounding, and route following with counting cues. Benchmarking classical and recent large vision language model (LVLM) based VLN agents on SatNav shows that city-scale navigation remains challenging. We further introduce SwiftVLN, a modular framework with switchable memory components, and conduct systematic memory-design ablations. Finally, satellite-to-UAV transfer experiments show that satellite-trained navigation models can operate on real-flight UAV observations, showing the practical relevance of SatNav.
Show more
OccStress: Stress-Testing the 4D Occupancy Forecasting Chain
Yu Zheng ⋅ Jie Hu ⋅ Jiaqi Xiong ⋅ Ruiping Liu ⋅ Junwei Zheng ⋅ Kailun Yang ⋅ Jiaming Zhang
Occupancy world models use historical occupancy states to forecast future 3D scenes, but their robustness under corrupted temporal inputs remains poorly understood. Existing evaluations primarily emphasize clean forecasting accuracy and provide limited evidence about how errors enter, persist, and propagate through the occupancy perception-forecasting chain. This paper introduces OccStress, a robustness stress-testing benchmark for the occupancy forecasting chain. OccStress contains 61 corruption categories and 80k+ anchors. OccStress covers both 3D occupancy perception and 4D occupancy forecasting through two complementary tracks. This design separates realistic pipeline errors from the intrinsic sensitivity of 4D forecasting models to corrupted occupancy states. OccStress further defines temporal injection protocols to test whether errors in the current state, recent history, or earlier history affect future forecasts differently. OccStress provides aggregate metrics to evaluating robustness along occupancy forecasting chain. Experiments on OccStress reveal that current occupancy models are substantially affected by both upstream prediction errors and direct state corruptions, and that clean performance alone is an insufficient indicator of temporal robustness. Dataset: https://hf.co/datasets/OccStress/OccStress. Code: https://anonymous.4open.science/r/OccStress.
Show more
Brittlebench: Quantifying LLM robustness via prompt sensitivity
Angelika Romanou ⋅ Mark Ibrahim ⋅ Candace Ross ⋅ Kerem Oktar ⋅ Chantal Shaib ⋅ Samuel J Bell ⋅ Anaelia Ovalle ⋅ Antoine Bosselut ⋅ Jesse Dodge ⋅ Koustuv Sinha ⋅ Adina Williams
Existing evaluation methods largely rely on clean, static benchmarks, which can overestimate true model performance by failing to capture the noise and variability inherent in real-world user inputs. This is especially true for language models, which can face human-generated text queries containing mistakes, typos, or alternative ways of phrasing the same question. In this work, we introduce a theoretical framework for quantifying model sensitivity to prompt variants, or brittleness, that can enable us to disentangle data-induced difficulty from prompt-related variability. Using this framework, we design a novel evaluation pipeline, Brittlebench, to holistically evaluate the sensitivity of frontier models. We apply semantics-preserving perturbations to a suite of popular benchmarks, and observe model performance to degrade as much as ~21%. However, these perturbations do not affect all models equally: even a single perturbation alters the relative ranking of models in 63% of cases, impacting conclusions about comparative model performance. Decomposing the total variance of both state-of-the-art open-weight and commercial models, we find that semantics-preserving input perturbations can account for up to half of the performance variance for a given model. Brittlebench highlights the need for more robust evaluations and models, and allows us to systematically understand model brittleness.
Show more
SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
Mohit Raghavendra ⋅ Soham Dan ⋅ Miguel Romero ⋅ Yannis He ⋅ Johannes B Mols ⋅ Gautam Anand ⋅ Cole McCollum ⋅ Edgar Arakelyan ⋅ Vijay Bharadwaj ⋅ Andrew Park ⋅ Jeff Da ⋅ MohammadHossein Rezaei ⋅ Bing Liu ⋅ Brad Kenstler ⋅ Yunzhong He
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
Show more
VAANI: Capturing the language landscape for an inclusive digital India
Sujith Pulikodan ⋅ Abhayjeet Singh ⋅ Agneedh Basu ⋅ Nihar Desai ⋅ Pavan K J ⋅ Pranav D Bhat ⋅ Raghu Dharmaraju ⋅ Ritika Gupta ⋅ Sathvik Udupa ⋅ Saurabh Kumar ⋅ Sumit Sharma ⋅ Visruth Sanka ⋅ Dinesh Tewari ⋅ Harsh Dhand ⋅ Amrita Kamat ⋅ Sukhwinder Singh ⋅ Shikhar Vashishth ⋅ Partha Talukdar ⋅ Raj Acharya ⋅ Prasanta Ghosh
Voice-based technologies have the potential to bridge digital accessibility gaps; however, existing datasets fail to capture the linguistic and regional diversity of Indic languages. We present Project VAANI, a large-scale multimodal dataset designed to represent India’s linguistic landscape across 165 districts. Speech data is collected using image-based prompts to elicit spontaneous responses, while images are curated through a separate pipeline covering diverse themes across regions. The dataset undergoes a rigorous multi-stage quality control process, combining automated and manual evaluation to ensure high audio quality and transcription accuracy. We release approximately 289K images, 31,255 hours of speech, and 2,043 hours of transcribed audio spanning 105 languages from 28 states and 3 union territories. Many of these languages are represented at this scale for the first time, making VAANI a foundational resource for inclusive speech technology. The dataset enables the development of robust, multilingual, and multimodal models, and supports research in speech recognition, language understanding, and cross-modal learning for underrepresented languages.
Show more
RealityTest: How People Probe AI Identity and Whether Models Disclose It
Anna Gausen ⋅ Sarenne Wallbridge ⋅ Bessie O'Dell ⋅ Christopher Summerfield ⋅ Hannah Rose Kirk
AI systems are increasingly deployed in conversational settings where users may be uncertain whether they are speaking with a human or an AI. Despite mounting regulatory attention to this known safety risk, existing evaluations of AI disclosure are typically English-only, based on machine-generated questions, and restricted to text. We present RealityTest to comprehensively test whether AI systems disclose their identity when asked. The benchmark is the first large-scale multimodal and multilingual evaluation, grounded in human data on how people actually encounter and question AI identity in the real-world. Alongside the benchmark, we release the underlying dataset of 3,152 identity-probing queries collected from $\textasciitilde$750 participants across 49 countries and five languages, in text and speech scenarios. We find that only 31\% of people ask about identity directly, and that the questions people ask are far more diverse than machine-generated queries. We test 17 text and 6 speech models, and find substantial variation in disclosure behaviour. However, a single suppression instruction reduces disclosure rates to below 30\%, even in the best-performing models. Validating our investment in diverse, human-grounded evaluation data, we find that how the question is phrased and the context of the conversation matter more for disclosure than which model is being tested. Safety evaluations built on narrow or synthetic query sets risk mischaracterising how models behave in realistic deployment settings.
Show more
Can AI Agents Synthesize Scientific Conclusions?
Hayoung Jung ⋅ Pedro V Diniz ⋅ José R Roveda ⋅ Abner F da Silva ⋅ Haeun Jung ⋅ Enoch Tsai ⋅ Aleksandra Korolova ⋅ Manoel Ribeiro
Scientific AI agents increasingly retrieve evidence, reason across sources, and synthesize conclusions used in consequential decisions. Yet, their ability to do so in high-stakes domains such as health remains unclear. We introduce SciConBench, a large-scale live benchmark of 9.11K questions and expert-written conclusions from systematic reviews to evaluate open-domain scientific conclusion synthesis. The benchmark draws on an expert-validated automated evaluation pipeline that decomposes conclusions into atomic facts and measures correctness and comprehensiveness via factual precision and recall. To mitigate data leakage, we further introduce SciConHarness, a clean-room evaluation harness that equips agents with controlled web interaction to ensure valid measurement. Evaluating 8 frontier models and deep research agents, we find that factual quality remains low: under clean-room settings, the best agent achieves only a factual F1 of 0.337. Our clean-room setting consistently reduces performance relative to unconstrained evaluation, suggesting that leakage inflates estimates of models' true synthesis capabilities. Finally, we audit consumer-facing agents (e.g., Google AI Overview, OpenEvidence) and find they frequently generate incomplete and sometimes contradictory conclusions, even when the ground-truth answer is available. Overall, our results show that reliable synthesis of scientific conclusions remains an open challenge, and that clean-room evaluation is essential for assessing open-domain AI agents.
Show more
FLARE: Full-Modality Long-Video Audiovisual Retrieval Benchmark with User-Simulated Queries
Qijie You ⋅ Hao Liang ⋅ Mingrui Chen ⋅ Bohan Zeng ⋅ Meiyi Qiang ⋅ Zhen H Wong ⋅ Wentao Zhang
As video becomes increasingly central to information dissemination and multimodal large language models (MLLMs) continue to advance, evaluating video retrieval has become increasingly important. In realistic search scenarios, this requires matching short user queries to long-form content using both visual and auditory evidence. Yet existing retrieval benchmarks are still dominated by short clips, single modalities, and caption-based evaluation. We introduce FLARE, a full-modality long-video audiovisual retrieval benchmark with user-simulated queries. Built from 399 carefully screened Video-MME videos (10--60\,min, 225.4\,h) to ensure source quality and diversity, FLARE contains 87,697 clips annotated with vision, audio, and unified audiovisual captions, together with 274,933 user-style queries. Cross-modal queries are further filtered by a hard bimodal constraint, requiring retrieval to fail under either modality alone but succeed when both are combined. FLARE evaluates models under two regimes, caption-based and query-based retrieval, across vision, audio, and unified audiovisual settings. Experiments with 15 representative retrievers show that user-style queries substantially change model behavior, strong caption-based performance does not always transfer to query-based retrieval, and audio--language alignment remains a key bottleneck for unified audiovisual retrieval. Our code and data are released at https://flarebench.github.io/.
Show more
ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making
Liu Dai ⋅ Haina Wang ⋅ Weikang Wan ⋅ Hao Su
Building embodied agents capable of accomplishing arbitrary tasks is a core objective towards achieving embodied artificial general intelligence (E-AGI). While recent work has advanced such general robot policies, their training and evaluation are often limited to tasks within specific scenes, involving restricted instructions and scenarios. Existing benchmarks also typically rely on manual annotation of limited tasks in a few scenes. We argue that exploring the full spectrum of feasible tasks within any given scene is crucial, as they provide both extensive benchmarks for evaluation and valuable resources for agent improvement. Towards this end, we introduce ManiTaskGen, a novel system that automatically generates comprehensive, diverse, feasible mobile manipulation tasks for any given scene. The generated tasks encompass both process-based, specific instructions (e.g., "move object from X to Y") and outcome-based, abstract instructions (e.g., "clear the table"). We apply ManiTaskGen to both simulated and real-world scenes, demonstrating the validity and diversity of the generated tasks. We then leverage these tasks to automatically construct benchmarks, thoroughly evaluating the embodied decision-making capabilities of agents built upon existing vision-language models (VLMs). Furthermore, we propose a simple yet effective method that utilizes ManiTaskGen tasks to enhance embodied decision-making. Overall, this work presents a universal task generation framework for arbitrary scenes, facilitating both benchmarking and improvement of embodied decision-making agents.
Show more
SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions
Jeremy S G Yee ⋅ Daniel Zhengkui Wang ⋅ Zhiyuan Zhang ⋅ Avinash Anand ⋅ Timothy Liu ⋅ Benedict Chan ⋅ Aik B Ng ⋅ Simon See
Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under physically constrained imaging conditions remains poorly understood. Existing evaluations predominantly assume ideal visual inputs and therefore fail to characterize how camera distance, illumination, viewpoint, and pixel density fundamentally affect semantic recoverability. We introduce SynDORBench, the first physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated conditions aligned with human visual capability standards. SynDORBench comprises over 54k question--answer pairs generated through a controllable synthetic pipeline that systematically varies viewing distance, lighting, camera geometry, and action pose according to physically interpretable pixel-density regimes. To support scalable low-visibility supervision, we further propose a discernibility annotation framework that propagates human perceptual labels using mask-conditioned statistical features and ensemble learning. We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x across human-presence classification and action recognition tasks under progressively degraded visibility conditions. Our results reveal that perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints rather than model scale alone. Surprisingly, several compact open-source LVLMs outperform larger commercial baselines and substantially exceed YOLO11x robustness under long-range and low-light conditions. SynDORBench establishes a new benchmark paradigm for physically grounded multimodal evaluation, enabling systematic analysis of LVLM reliability under real-world perceptual constraints and direct comparison against human visibility thresholds.
Show more
Unpacking the Evaluators: How Configuration Shapes the Evaluation of Alignment in Explainable AI
Gizem Karagoz ⋅ Tanir Özçelebi ⋅ Nirvana Meratnia
Explainable AI (XAI) evaluation fundamentally requires human-centered evaluation, assessing whether explanations align with human reasoning and expert knowledge. However, such evaluations often require expert-driven manual validation and are difficult to scale, leading to the widespread use of alignment-based metrics, where attribution maps are compared against predefined ground-truth (GT) representations. Despite their popularity, alignment evaluations are implicitly bias towards multiple design choices made during the evaluation process, for example GT representations, metric families, metric formulations (soft vs.\ hard), and thresholding strategies. In this paper, we formalize alignment evaluation as a structured evaluation configuration space through which the influence of these design choices can be systematically analyzed. We further introduce a structured alignment evaluation framework that models interactions between XAI methods, GT representations, metric families, metric formulations, and thresholding strategies across 2D and 3D synthetic and real-world datasets. In addition, we propose a novel taxonomy of XAI output behavior based on local entropy and boundary mass ratio, enabling structured characterization of attribution patterns for alignment evaluation. Our experiments reveal systematic flows(triples of XAI-GT-metric configurations), showing that alignment outcomes provide distinct behavior patterns under certain configurations, such as soft/hard metric implementations. We further observe varying levels of threshold sensitivity across metric families and attribution structures. These findings demonstrate that alignment evaluation cannot be treated as a fixed or invariant measure of explanation quality, and provide practical guidance for designing more reliable and behavior-aware XAI evaluation protocols.
Show more
ProtGlycanDock: Towards Accurate Protein-Glycan Docking with Tailored Dataset, Benchmark and Model
Minghao Xu ⋅ Jiaze Song ⋅ Jinyu Wang ⋅ Wentao Zhang
Performing accurate protein-glycan docking is critical to understanding protein-glycan interactions which govern many biological processes. However, this important task is hampered by the lack of tailored datasets, benchmarks and models for protein-glycan docking. To fill this blank, we first curate a standard benchmark **ProtGlycanDock** for training and evaluating protein-glycan docking models under both protein and glycan generalization settings. On this benchmark, we evaluate ten models in three categories, physics-based docking methods, machine-learning-based docking models and molecular structure prediction models. To obtain more precise binding poses, we further propose two finetuning techniques to inject the knowledge of glycan conformations into a general-purpose molecular structure prediction model. These two techniques respectively augment model inputs with glycan topological features and refine glycan binding pose with a glycan energy loss. Benchmark results show the superiority of AlphaFold~3 among existing methods, and the two proposed finetuning techniques can be well combined to achieve a state-of-the-art model for protein-glycan docking. Also, we perform stratified benchmark analysis and case studies to better understand the strengths and weaknesses of current models, providing insights for future model improvements. We maintain datasets at https://huggingface.co/datasets/ProtGlycanDock/ProtGlycanDock and source codes at https://github.com/kasawa1234/ProtGlycanDock.
Show more
Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks
Benjamin Warner ⋅ Ratna S Grandhi ⋅ Max Kieffer ⋅ Aymane Ouraq ⋅ Saurav Panigrahi ⋅ Geetu Ambwani ⋅ Kunal Bagga ⋅ Nikhil Khandekar ⋅ Arya Hariharan ⋅ Nishant Mishra ⋅ Manish Saravanan ⋅ Shamus S Yang ⋅ Ahmed Essouaied ⋅ Adepoju J Moyondafoluwa ⋅ Robert Scholz ⋅ Bofeng Huang ⋅ Molly Beavers ⋅ Srishti Gureja ⋅ Anish Mahishi ⋅ Sameed Khan ⋅ Maxime Griot ⋅ Hunar Batra ⋅ Jean-Benoit Delbrouck ⋅ Siddhant Bharadwaj ⋅ Ronald Clark ⋅ Ashish Vashist ⋅ Anas Zafar ⋅ Leema K Murali ⋅ Harsh Deshpande ⋅ Ameen Patel ⋅ William Brown ⋅ Johannes Hagemann ⋅ Connor Lane ⋅ Paul Scotti ⋅ Tanishq M Abraham
Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insufficient coverage of relevant tasks. Existing suites have either saturated, heavily depend on restricted datasets, or lack comprehensive model coverage. We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning. We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge. Our results show that frontier reasoning models (Gemini 3 Pro Preview, GPT-5.1, \& GPT-5.2) achieve the highest performance across both benchmarks, most frontier proprietary models are significantly more token efficient than open-weight alternatives, medically fine-tuned models outperform their generalist counterparts, and that models are susceptible to answer-order bias (particularly smaller models and Grok 4). Code is available at https://anonymous.4open.science/r/med-lm-envs-B76C/
Show more
MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping
Junyao Gao ⋅ Sibo Liu ⋅ Jiaxing Li ⋅ Yanan SUN ⋅ Yuanpeng Tu ⋅ Fei Shen ⋅ Weidong Zhang ⋅ Cairong Zhao ⋅ Jun Zhang
In this paper, we introduce MegaStyle, a novel and scalable data curation pipeline that constructs an intra-style consistent, inter-style diverse and high-quality style dataset. We achieve this by leveraging the consistent text-to-image style mapping capability of current large generative models, which can generate images in the same style from a given style description. Building on this foundation, we curate a diverse and balanced prompt gallery with 170K style prompts and 400K content prompts, and generate a large-scale style dataset MegaStyle-1.4M via content–style prompt combinations. With MegaStyle-1.4M, we propose style-supervised contrastive learning to fine-tune a style encoder MegaStyle-Encoder for extracting expressive, style-specific representations, and we also train a FLUX-based style transfer model MegaStyle-FLUX. Extensive experiments demonstrate the importance of maintaining intra-style consistency, inter-style diversity and high-quality for style dataset, as well as the effectiveness of the proposed MegaStyle-1.4M. Moreover, when trained on MegaStyle-1.4M, MegaStyle-Encoder and MegaStyle-FLUX provide reliable style similarity measurement and generalizable style transfer, making a significant contribution to the style transfer community.
Show more
MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support
António Farinhas ⋅ Nuno M Guerreiro ⋅ José Pombal ⋅ Pedro Henrique Martins ⋅ Laura Melton ⋅ Alexandra Conway ⋅ Cara Dochat ⋅ Maya D'Eon ⋅ Ricardo Rei
Large language models are increasingly used for mental health support, yet their conversational coherence alone does not ensure clinical appropriateness. Existing general-purpose safeguards often fail to distinguish therapeutic disclosures from genuine clinical crises, and progress on developing better ones is bottlenecked by a lack of clinically grounded evaluation resources. To address this gap, we introduce a risk taxonomy, developed in collaboration with PhD-level clinical psychologists, that identifies actionable harm (self-harm and harm to others) while preserving space for safe, non-crisis therapeutic content. We release MindGuard-testset, a dataset of multi-turn conversations annotated at the turn level by clinical experts, which is, to our knowledge, the first public benchmark with turn-level clinical risk labels for multi-turn mental health support. We also release an automated red-teaming (ART) framework designed to measure how safety classifiers affect downstream model behavior in adversarial multi-turn interactions. Using synthetic dialogues generated via a controlled two-agent setup, we train MindGuard, a family of lightweight safety classifiers (with 4B and 8B parameters). Our classifiers reduce false positives at high-recall operating points and, when paired with clinician language models, help achieve lower attack success and harmful engagement rates compared to general-purpose safeguards. We release all models, human evaluation data, and the ART framework.
Show more
HERO: A Heterogeneity-Aware Benchmark Library for Federated Continual Learning
Thinh Nguyen ⋅ Le-Tuan Nguyen ⋅ Minh-Duong Nguyen ⋅ Nhi Trinh ⋅ Anh T Nguyet ⋅ Dung Le ⋅ Kok-Seng Wong
Federated continual learning (FCL) evaluates how distributed clients learn from changing data streams while retaining previously learned knowledge. Existing evaluations are difficult to compare because they often change datasets, task splits, client data splits, task orders, backbones, memory assumptions, and reporting rules simultaneously. We introduce \textbf{HERO}, a heterogeneity-aware benchmark library for FCL. HERO builds benchmark streams by separating three choices that are often coupled, namely the task split, the client data split, and the client task sequence. In HERO-Core, the main comparable benchmark, $\alpha$ controls client data skew and $\rho$ controls task-order mismatch. We evaluate representative FCL methods on CIFAR-100 and TinyImageNet using final average accuracy, average forgetting, and bottom-10\% client accuracy. We also include a graph-based Domain-IL portability case study on OGB-MolPCBA, where scaffold-domain granularity changes the input distribution while the prediction task remains fixed. Our results show that method behavior changes across easy and heterogeneous settings, that average accuracy can hide weak bottom-client performance, that task-order mismatch favors different strategies from synchronized evaluation, and that the same HERO interface can expose domain-shift difficulty beyond image-based FCIL. HERO releases benchmark streams, configurations, method implementations, and reporting scripts to support reproducible and setting-aware FCL evaluation.
Show more
HumanScore: Benchmarking Human Motions in Generated Videos
Tiange Xiang ⋅ Yusu Fang ⋅ Tian Tan ⋅ Narayan Schuetz ⋅ Scott Delp ⋅ Fei-Fei Li ⋅ Ehsan Adeli
Recent advances in model architectures, compute, and data scale have driven rapid progress in video generation, producing increasingly realistic content. Yet, there is no method to systematically assess the fidelity of human figures generated in these videos. In this paper, we present HumanScore, a systematic framework to evaluate the \textbf{quality of human figures and motions} in AI-generated videos. We introduce seven metrics spanning kinematic plausibility and biomechanical consistency. Through carefully designed prompts, we elicit a broad set of movements at varying intensities. We evaluated a total of 840 videos generated by 6 state-of-the-art models. Our framework reveals consistent gaps between visual plausibility and motion fidelity, highlights common failure modes, and ranks models from multiple quantitative and physically meaningful metrics. The proposed metrics are correlated with human evaluations and are simple to compute using standard pose estimators.
Show more
SANEval: Open-Vocabulary Compositional Benchmarks with Failure-mode Diagnosis
Rishav Pramanik ⋅ Ian Nielsen ⋅ Jeffrey Smith ⋅ Saurav Pandit ⋅ Ravi P Ramachandran ⋅ Zhaozheng Yin
Text-to-image (T2I) models still fail on compositional prompts that require multiple objects, correctly bound attributes, accurate counts, or specific spatial relations, yet the benchmarks used to measure this progress are themselves limited: they rely on fixed-vocabulary detectors (typically the **80** MS-COCO classes), produce a single opaque score, and are rarely validated against human judgment. We introduce **SANEval** (Spatial, Attribute, and Numeracy Evaluation), an open-vocabulary compositional benchmark with two contributions: (i) a public dataset of $\sim$**5,250** prompts paired with images from six state-of-the-art T2I models, partitioned into a Simple split (algorithmically generated, human-validated) and a Hard split (fully human-written), and labeled by compositional task type (spatial, numeracy, color, shape, texture), and (ii) a modular evaluation pipeline that combines an LLM-based prompt parser, an open-vocabulary detector with LLM-driven synonym expansion and mapping, and three scorers for attribute binding, spatial relations, and numeracy. Each scorer emits both a numerical score and structured diagnostic feedback identifying what is missing, extra, or mis-bound. We validate SANEval against human ratings ($n=500$, **4** annotators, **5** categories): SANEval achieves a positive Pearson correlation with humans in every category (avg $r=0.222$), with **4** of **5** category-level correlations significantly above zero (Fisher **95%** CIs excluding **0**); the strongest prior compositional benchmark averages near zero ($r=-0.016$), with **4** of **5** category CIs spanning zero. SANEval is, to our knowledge, the only compositional T2I benchmark whose correlation sign agrees with humans on every axis. A manual audit of **10,040** detections decomposes the pipeline into a perception stage and an LLM/VLM verification stage and measures end-to-end precision at **96.6%**; the verification stage (synonym mapping plus VLM-as-judge attribute reading) contributes only **1.05%** of audited false positives, while removing it collapses scores by **43–76%** in ablation — substantial detection work at modest precision cost. The dataset is publicly released at [https://huggingface.co/datasets/saneval-ann/saneval-release](https://huggingface.co/datasets/saneval-ann/saneval-release) and the evaluation pipeline at [https://anonymous.4open.science/r/saneval-anon](https://anonymous.4open.science/r/saneval-anon).
Show more
MaSC: A Masked Similarity Metric for Evaluating Concept-Driven Generation
Patryk Bartkowiak ⋅ Lennart Petersen ⋅ Bartosz Kotrys ⋅ Dominik L Michels ⋅ Soeren Pirk ⋅ Wojtek Palubicki
Evaluating single-concept personalization in text-to-image diffusion has seen two categories of quantitative metrics: Concept Preservation (CP) measures identity fidelity to a reference while Prompt Following (PF) measures whether the generated scene matches the prompt. Personalization papers have commonly computed these signals using three separate backbones: CLIP-I and DINO for CP, CLIP-T for PF. In this work, we show that existing metrics fall short of correlating with human perception because they attend to the image as a whole, instead of distinguishing the concept subject from the background. This distinction is important from a human perception point of view as the concept subject in the output image should be very similar to the input concept, whereas the output background should adhere closely to the text prompt. To improve personalization evaluation in this way, we introduce MaSC, a unified metric that attends to CP and PF by differentiating concept subject regions. Specifically, given an externally provided foreground concept mask, MaSC computes the CP and PF scores from a single forward pass of a frozen SigLIP2 encoder per image. On DreamBench++ human ratings, MaSC reaches Krippendorff $\alpha = +0.471$ and Spearman $\rho = +0.611$ on CP, the highest point estimate among the evaluated non-LLM metrics, exceeding GPT-4V and falling $\Delta\alpha = 0.028$ below GPT-4o. An exploratory reference-mask sensitivity analysis reaches $\alpha = +0.506$ with $32\%$ dilation. To distinguish human perception bias we also evaluate our method, as well as common SOTA baselines, on ORIDa, a real-photo benchmark of identity preservation across physical environments. In this experiment MaSC reaches AUC $= 0.992$ almost perfectly identifying concept subjects. The PF score, obtained without a second vision-encoder forward pass, achieves higher rank correlation with human ratings than the CLIP-T baseline shipped with DreamBench++. Together, these results support foreground-local aggregation for concept preservation and show how a shared encoder can provide complementary concept-preservation and prompt-following scores. We release MaSC as a pip-installable Python package, alongside independent reproductions of every comparator's published numbers.
Show more
WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors
Keming Wu ⋅ Yijing Cui ⋅ Wenhan Xue ⋅ Qijie Wang ⋅ Xuan Luo ⋅ ZhiYuan Feng ⋅ Zuhao Yang ⋅ Sudong Wang ⋅ Sicong Jiang ⋅ Haowei Zhu ⋅ Zihan Wang ⋅ Ping Nie ⋅ Wenhu Chen ⋅ Bin Wang
Commercial video generation systems such as Seedance2.0 and Veo3.1 have rapidly improved, strengthening the view that video generators may be evolving into "world simulators." Yet the community still lacks a benchmark that directly tests whether a model can reason about how an observed world should evolve over time. We introduce WorldReasonBench, which reframes video generation evaluation as world-state prediction: given an initial state and an action, can a model generate a future video whose state evolution remains physically, socially, logically, and informationally consistent? WorldReasonBench contains 436 curated test cases with structured ground-truth QA annotations spanning four reasoning dimensions and 22 subcategories. We evaluate generated videos with a human-aligned two-part methodology: Process-aware Reasoning Verification uses structured QA and reasoning-phase diagnostics to detect temporal and causal failures, while Multi-dimensional Quality Assessment scores reasoning quality, temporal consistency, and visual aesthetics for ranking and reward modeling. We further introduce WorldRewardBench, a preference benchmark with approximately 6K expert-annotated pairs over 1.4K videos, supporting pair-wise and point-wise reward-model evaluation. Across modern video generators, our results expose a persistent gap between visual plausibility and world reasoning: videos can look convincing while failing dynamics, causality, or information preservation. We will release our benchmarks and evaluation toolkit to support community research on genuinely world-aware video generation.
Show more
AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation
Mingyang Li ⋅ Haofan Xu ⋅ Haowen Sun ⋅ Xinzhe Chen ⋅ Sihua Ren ⋅ Liqi Huang ⋅ Chenyang Miao ⋅ Xinyang Sui ⋅ jiawei Ye ⋅ Qiongjie Cui ⋅ Zeyang Liu ⋅ Xingyu Chen ⋅ Xuguang Lan
Many everyday robot manipulation skills are affordance-dependent, with success determined by whether the robot contacts the functional object region required by the subsequent action. Current simulation data generators obtain contacts from generic grasp estimators or per-object manual contact annotations, but generic estimators rank stable grasps without task semantics and often select contacts that are misaligned with the downstream action, while manual contact annotations must be rewritten for each new object and task. To solve these challenges, we introduce AffordSim, a scalable data generator and benchmark that integrates open-vocabulary 3D affordance prediction into simulation-based trajectory generation. Given a natural-language task description, AffordSim synthesizes a task-relevant scene, emits affordance queries, grounds them on object surfaces, samples region-conditioned grasps, and selects executable candidates with motion planning. It further randomizes object pose, texture, lighting, image noise, and cross-viewpoint backgrounds for sim-to-real transfer. We instantiate AffordSim as a 50-task benchmark across diverse manipulation skills, five robot embodiments, and 500+ rigid and articulated objects. AffordSim achieves $93\\%$ of the trajectory collection success rate of manual contact annotations on affordance-critical tasks and $89\\%$ on hard composite tasks. Vision-language-action policies trained on AffordSim data transfer zero-shot to a real Franka FR3, reaching $24\\%$ average success. The data generator, benchmark, and code are available at \href{https://anonymous.4open.science/r/AffordSim-8D94/}{project page}.
Show more
Thinking in Pictures: A Systematic Benchmark for Reasoning-driven Image Generation
Yutong Liu ⋅ Nan Huang ⋅ Xu Cao ⋅ James Rehg
Recent advancements in world models and unified generative models (UGMs) have achieved unprecedented results in visual perception and synthesis. However, these models primarily rely on surface-level alignment, leaving the capacity for high-level visual reasoning underexplored. True visual generative intelligence demands "Reasoning-to-Generation", an ability to infer latent rules from visual inputs and manifest solutions through precise, logically constrained visual outcomes. We introduce RIG-Bench, a novel comprehensive benchmark that systematically evaluates Reasoning-driven Image Generation (RIG) across four cognitively demanding domains: Concept-based, Transformation-based, Pattern & Structure, and Scenario-based. Featuring 2000 curated samples, RIG-Bench serves as a rigorous stress test for RIG. Our extensive evaluations of state-of-the-art UGMs and image/video generation models reveal a significant reasoning-generation gap, wherein models frequently produce locally plausible but globally illogical outputs. RIG-Bench provides a vital diagnostic framework to guide the development of next-generation, logically grounded world models.
Show more
EgoStream: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
Rosario Forte ⋅ Giuseppe Lando ⋅ Antonino Furnari
Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce Egostream, a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision. \egostream organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art memory-management mechanisms, covering sliding windows, attention sinks, KV-cache pruning, merging, and offloading. Experiments within a unified Qwen3-VL backbone reveal that comparable aggregate accuracies mask starkly different memory profiles. For instance, token pruning preserves fine-grained details and temporal structure significantly better than token merging, while quantized offloading rescues ultra-long-term recall. Ultimately, all mechanisms operate well below real-time (>1s per frame), and top performing methods ceil at about 45\% accuracy, exposing critical gaps in current architectures. Egostream provides the diagnostic testbed needed to close these gaps.
Show more
SUPERVISE: A Unified Framework for Standardized and Reproducible Superpixel Evaluation
Julien Walther ⋅ Rémi Giraud ⋅ Michaël Clément
Image segmentation into superpixels is a widely used technique in computer vision, with a large body of work developed over the years. Yet, its evaluation remains poorly standardized, as methods are compared under heterogeneous protocols, across inconsistent scale ranges, and with redundant or misaligned metric subsets. In this work, we introduce $\textbf{SUPERVISE}$ (SUperpixel PERformance VISualization and Evaluation), a unified and reproducible benchmark that addresses these longstanding inconsistencies. Built on a scale-normalized protocol based on interpolation over the actual number of generated superpixels, SUPERVISE enables fair and consistent cross-method comparisons. We leverage this standardized framework to conduct the first large-scale statistical analysis of 20 evaluation metrics across 7 datasets and 31 methods — revealing strong inter-metric correlations and significant redundancy that challenge common evaluation practices. These findings provide empirical grounding for a compact, representative metric subset. SUPERVISE is released as a lightweight, modular framework built on precomputed shared label maps, making large-scale reproducible evaluation accessible to the community. All code, label maps, and evaluation results are publicly available at: https://anonymous.4open.science/r/evaluation_superpixel-08B1/
Show more
FlyingDrones: A Dataset and Benchmark for Optical Flow Estimation from UAV motion
Laurenz Ruzicka ⋅ Roman Pflugfelder ⋅ Daniel Cremers
Optical flow is an important cue for drone analysis in aerial surveillance, airspace monitoring, and long-range target tracking, where motion remains informative when appearance is weak. Estimating the motion of small Unmanned Aerial Vehicles (UAVs), however, is difficult because distant targets occupy few pixels, have low contrast, and move against complex dynamic backgrounds. Standard optical-flow benchmarks emphasize large, object-centric motions and therefore do not characterize distant aerial targets well. We introduce FlyingDrones, a generated benchmark with dense ground-truth optical flow and segmentation masks derived from physically based rendering, and define resolution-normalized metrics (EPE Norm) that normalize flow by image dimensions. We evaluate 16 state-of-the-art models on small-target motion estimation. Standard RAFT degrades substantially on small targets at low resolution, whereas VideoFlow's multi-frame variant achieves the best drone-region EPE at every tested resolution. Aerial-pretrained two-frame models such as Sea-RAFT and WAFT are competitive at 270$\times$480 but lose their advantage at 1080$\times$1920, and the best fine-tuning domain depends on resolution: KITTI helps when drone motion is sub-pixel-to-few-pixels, while Sintel-style fine-tuning helps once displacements grow with image size. Overall, long-range drone analysis benefits most from multi-frame temporal context, scale-aware evaluation, and resolution-matched fine-tuning.
Show more
What Gets Measured Gets Managed: Sign-aware Recommendation Needs Sign-aware Evaluation
Minchan Kim ⋅ Jungmin Hwang ⋅ Hyunwoo Park
Sign-aware recommender systems have recently been developed to leverage negative feedback for a deeper understanding of user preferences. However, our empirical diagnosis reveals that state-of-the-art graph-based sign-aware recommender systems are paradoxically *valence-blind*. Even though they explicitly incorporate sign information during training, they consistently fail to differentiate liked items from disliked ones at the ranking stage, frequently infiltrating top-$K$ recommendations with disliked content. Through linear probing, we show that while valence information exists in the learned embeddings, it remains inaccessible to the inner-product scoring function. This widespread failure remains entirely undetected because conventional evaluation metrics, such as Recall, HR, and NDCG, assign a uniform utility of zero to both negative and unobserved items, creating a systematic evaluation blind spot. To bridge this gap, we propose a family of signed metrics, Signed Recall, Signed HR, and Signed NDCG, that explicitly penalize the recommendation of disliked content. Systematic re-evaluation under our proposed metrics fundamentally reshapes the established performance landscape, revealing that methods ranked highly under conventional metrics often fail to protect users from disliked content. Finally, through a proof-of-concept auxiliary loss, we confirm that the proposed metrics provide actionable training signals, guiding models toward valence-aware behavior without sacrificing conventional relevance. For transparency, our source code is available at: https://anonymous.4open.science/r/signed-rec-benchmark-07E4
Show more
SGNNBench: A Holistic Evaluation of Spiking Graph Neural Networks on Large-scale Graphs
Huizhe Zhang ⋅ Jintang Li ⋅ Yuchang Zhu ⋅ Huazhen Zhong ⋅ Liang Chen
Graph Neural Networks (GNNs), as representative deep models designed for graph data, effectively learn graph topological information and push the performance boundaries across various graph tasks. However, the computational and memory burden of GNNs poses significant challenges for scaling to large real-world graphs. Spiking Graph Neural Networks (SGNNs), which integrate biologically plausible learning via unique spike-based neurons, have emerged as a promising energy-efficient alternative. Different layers communicate with sparse and binary spikes, which facilitates computation and storage of intermediate graph representations. Despite the proliferation of SGNNs proposed in recent years, there is no systematic benchmark to explore the basic design principles of these brain-inspired networks on the graph data. To bridge this gap, we present **SGNNBench** to quantify progress in the field of SGNNs. Specifically, we elaborately investigate the design space of SGNNs to facilitate the development of a general SGNN paradigm. Regarding efficiency, we empirically compare these baselines w.r.t. model size, memory usage and theoretical energy consumption to reveal previously overlooked energy bottlenecks. Furthermore, SGNNBench establishes a unified and reliable benchmark to comprehensively evaluate 9 state-of-the-art SGNNs across 20 datasets covering diverse graph settings and tasks.
Show more
HiGraph: A Large-Scale Hierarchical Graph Dataset for Malware Analysis
Han Chen ⋅ Hanchen Wang ⋅ Hongmei Chen ⋅ Ying Zhang ⋅ Lu Qin ⋅ Wenjie Zhang
The advancement of graph-based malware analysis is critically limited by the absence of large-scale datasets that capture the inherent hierarchical structure of software. Existing methods often oversimplify programs into single-level graphs, failing to model the crucial semantic relationship between high-level functional interactions and low-level instruction logic. To bridge this gap, we introduce HiGraph, the largest public hierarchical graph dataset for malware analysis, comprising over 200M Control Flow Graphs (CFGs) nested within 499K Function Call Graphs (FCGs). This two-level representation preserves structural semantics essential for building robust detectors resilient to code obfuscation and malware evolution. We demonstrate HiGraph’s utility through a large-scale analysis that reveals distinct structural properties of benign and malicious software, establishing it as a foundational benchmark for the community. The dataset and tools are publicly available at https://higraph.org.
Show more
CRISP: Compositional Reasoning over Images via Stackable Programs for VLMs
Arnas Uselis ⋅ Yujin Jeong ⋅ Yanpeng Zhao ⋅ Alexander Rubinstein ⋅ Seong Joon Oh ⋅ Yonatan Bitton ⋅ Paul Gavrikov
How well do vision-language models actually reason about what they see? Current diagnostic benchmarks offer limited answers: they draw from fixed question templates, evaluate only the final answer, and are tied to single visual environments. We introduce CRISP (Compositional Reasoning over Images via Stackable Programs), a platform that generates compositional visual reasoning tasks from reusable, composable operations. Given any scene with a structured specification, the platform constructs reasoning chains of arbitrary depth where each intermediate step has automatically derived ground truth, including bounding boxes for the objects the model should attend to at every stage. New visual environments and reasoning operations plug in without modifying the generation engine. We instantiate the platform across three environments spanning 2D sprites, photorealistic 3D characters, and indoor room scenes, and show that frontier VLMs degrade systematically with reasoning depth and frequently arrive at correct answers through incorrectly grounded intermediate reasoning. Beyond benchmarking, the generated reasoning traces and step-wise ground truth can support future work on training and improving visually grounded reasoning models.
Show more
Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies
Tang Zirui ⋅ Xuanhe Zhou ⋅ Yumou Liu ⋅ Linchun Li ⋅ Wang Weizheng ⋅ Hongzhang Huang ⋅ Jun Zhou ⋅ Jiachen Song ⋅ Shaoli Yu ⋅ Wang Jinqi ⋅ Zihang Zhou ⋅ Hongyi Zhou ⋅ YutingLv ⋅ Jinyang Li ⋅ Jiashuo Liu ⋅ Ruoyu Chen ⋅ Chunwei Liu ⋅ Guoliang Li ⋅ Jihua Kang ⋅ Fan Wu
Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace, enabling them to complete both routine and advanced tasks effectively. Despite its importance, existing relevant benchmarks largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored. To this end, we introduce Workspace-Bench, a benchmark for evaluating AI agents on Workspace-Learning involving Large-Scale File Dependencies. We construct realistic workspaces with 5 worker profiles, 74 file types, 20,476 files (up to 20GB) and curate 388 tasks, each with its own file dependency graph, evaluated across 7,399 total rubrics that require cross-file retrieval, contextual reasoning, and adaptive decision-making. We further provide Workspace-Bench-Lite, a 100-task subset that preserves the benchmark distribution while reducing evaluation costs by about 70%. We evaluate 4 popular agent harnesses and 7 foundation models. Experimental results show that current agents remain far from reliable workspace learning, where the best reaches only 68.7%, substantially below the human+tool result of 80.7%, and the average performance across agents is only 47.4%. More details please refer to https://workspace-bench.github.io.
Show more
ParallelKernelBench: Can LLMs Write Fast Multi-GPU Kernels?
Willy Chan ⋅ Nathan Paek ⋅ Simon Guo ⋅ Simran Arora ⋅ Dan Fu
There is growing interest in using large language models (LLMs) to write high-performance GPU kernels, with recent work showing promising results on single-GPU workloads. However, multi-GPU kernel generation remains unexplored, despite communication emerging as a dominant bottleneck in large-scale training and inference. In this paper, we study how well LLMs can write multi-GPU kernels, a task compounded by three challenges: (1) the design space is combinatorially large, as training and inference workloads can be parallelized across tensor, expert, pipeline, data, and sequence dimensions; (2) single-GPU memory-compute roofline analysis fails to capture communication bottlenecks in multi-GPU execution; and (3) the many hardware paths available for communication (e.g., copy engine, TMA, SM instructions) each carry distinct tradeoffs. We introduce ParallelKernelBench (PKB), a benchmark and evaluation framework for multi-GPU kernel generation. Additionally, we construct a taxonomy of distributed workloads spanning different parallelism types and select 87 problems covering compositions that arise in real workloads. We contribute evaluations of frontier coding models on PKB, finding that current LLMs struggle: single-shot kernel correctness plateaus at 32% of cases, with speedup exceeding an unoverlapped baseline (PyTorch + NCCL) in only 25% of cases. We also contribute a communication-aware roofline analysis of correct kernels, finding that over 90% of baselines achieve less than 50% of peak hardware utilization. Optimized solutions to many PKB workloads are largely absent from existing open-source repositories; we highlight several LLM-generated net new kernels that outperform their reference implementations in specific regimes, including NeMo's vocab-parallel log-probability kernel (up to 2.06x), Hyena CP (up to 1.72x), and SAM3 IoU suppression (up to 1.40x).
Show more
ARES: How Reliable Are LLM User Simulators for Recommender A/B Testing?
Hongyang Su ⋅ Beibei Kong ⋅ Lei Cheng ⋅ Chengxiang Zhuo ⋅ Zang Li ⋅ Chenyun YU
LLM-driven user simulators are increasingly adopted for A/B-testing recommender systems, yet the choice of backbone is treated as an interchangeable implementation detail rather than a factor shaping evaluation reliability. We show otherwise: across nine backbones under unified controls, this choice accounts for 149% as much outcome variance as the tested recommender, and swapping backbones alone flips about 20% of pairwise A/B winners. The backbone, in other words, is not a neutral component but the measurement instrument. We turn this observation into a method: ARES (Audit for Reliable Evaluation of Simulators) separates a backbone effect (the instrument's contribution) from a recommender effect (the signal under study), quantifies both at three levels (outcomes, trajectories, and reasoning), stress-tests them against text vs. rendered UI, and endorses only A/B claims a majority of backbones independently support. Applied to nine backbones $\times$ five recommenders on MovieLens-1M, ARES surfaces failures invisible to outcome metrics: within a single model lineage, newer and stronger releases do not necessarily agree more with their predecessors (Kendall's $\tau$ drops from 1.0 to 0.20 across generations); three of eight vision-capable backbones reverse the declared best recommender under UI rendering; and 63000 reasoning traces tie these reversals to interpretable behavioral biases, not stochastic noise. Capability alone, therefore, does not guarantee reliability. While instantiated for recommendation, the backbone-as-instrument view generalizes to any LLM-mediated evaluation where model identity is a free variable, including LLM-as-a-Judge. We release ARES-Bench at https://github.com/neurips2026-ares-authors/ARES-Bench: 21400 behavioral logs, 63000 reasoning traces, a reusable visual sandbox, and an analysis toolkit, making multi-backbone auditing a feasible default for simulator-based evaluation.
Show more
InvestigationWorlds: An Agentic Environment for Legal Investigation
Albert Sun ⋅ Andrew J Benard ⋅ Sil Hamilton ⋅ Anna Teresita Marcelo ⋅ Yong J Kim ⋅ Carl-Leander Henneking ⋅ Rundong Hu ⋅ Yuhong Wang ⋅ David Mimno ⋅ Bishan Yang ⋅ Igor Labutov
We introduce **InvestigationWorlds**, an agentic environment for legal investigation. We build on an underused artifact of U.S. civil litigation: the summary judgment motion. This motion relies upon a record composed of real evidence exhibits, and results in a court-adopted hypothesis that is treated as ground truth for the purposes of deciding the motion. Each environment is built from a real U.S. Federal Court case retrieved from Public Access to Court Electronic Records (PACER) and augmented by an attorney-validated generation pipeline that synthesizes role-tagged documents around the original record. The resulting corpus admits multiple coherent factual readings, only one of which matches the court-adopted hypothesis. Evaluating on 100 cases, we find agents often commit to incorrect hypotheses despite retrieving relevant evidence, struggling to distinguish the court-adopted hypothesis from alternative hypotheses.
Show more
CADMA: Capacity-Aware Recall Decomposition for Generative Model Assessment
Debin Meng ⋅ Zheng Gao ⋅ Ziquan Liu ⋅ Yanran Li ⋅ Ioannis Patras ⋅ Georgios Tzimiropoulos
Evaluating generative models requires measuring not only sample fidelity but also how well a generated set covers the real data distribution. Recent work has therefore introduced recall-oriented metrics for distribution-level evaluation. However, existing recall metrics often reduce distributional coverage to geometry-based coverage. This simplifying assumption weakens their diagnostic value and interpretability in two ways: these metrics provide only a single recall score that cannot distinguish essential diversity failures such as mode dropping and mode collapse, and geometry-based coverage can provide unreliable estimates by assigning high recall to poorly covered generative distributions. We therefore introduce CADMA, a capacity-aware framework for recall-oriented evaluation and diagnosis. CADMA evaluates distributional coverage by formulating local real--fake allocation as a one-to-one matching problem, separating geometry-based coverage from effective coverage. This yields multi-dimensional recall diagnostics that quantify mode dropping and mode collapse while reducing the overly optimistic estimates produced by geometry-based recall metrics. Through extensive investigations on toy experiments and generative models, we show that CADMA provides more reliable recall estimates and more fine-grained, interpretable diagnostic signals of diversity failures than existing metrics.
Show more
PolyTopoBench: A Benchmark for Complex Vector Polygon Generation from Remote Sensing Imagery
Zeping Liu ⋅ Ni Lao ⋅ Weiwei Sun ⋅ Gil Wolff ⋅ Yiqun Xie ⋅ Liang Zhao ⋅ Junfeng Jiao ⋅ Gengchen Mai
Vector polygon generation converts visual inputs, e.g., remote sensing (RS) images, into vectorized polygonal geometries, supporting applications such as autonomous driving, vector map construction, and remote sensing. Early pipelines predict raster masks and post-process them into polygons, which prevents end-to-end optimization and may miss small objects or introduce inaccurate vertices. Recent methods directly generate vector polygons, but most focus on simple exterior contours, while they either cannot represent complex polygons with holes or fail to preserve their topology. In this paper, we propose PolyTopoBench, a unified evaluation framework for vector polygon generation from RS images with explicit emphasis on complex polygons. PolyTopoBench evaluates both exterior and interior rings, and benchmarks representative methods, including segmentation-based polygonization pipelines, vision foundation model baselines, and specialized vector polygon generators, on two RS-image datasets covering buildings, roads, vegetation, and unvegetated regions. Experiments show that existing methods often recover simple exterior boundaries but degrade substantially on polygons with holes or multiple rings. These results reveal complex polygon generation as an unresolved challenge and motivate topology-aware benchmarks and model designs.
Show more
BayesJudge: Uncertainty-Aware Bayesian Meta-Evaluation of Human and LLM Judgments
Jiahao Zhang ⋅ Pengbin Feng ⋅ Chunlei Meng ⋅ Hang He ⋅ Tianyong Hao
AI evaluation pipelines often produce conflicting judgments rather than clean labels. In pairwise LLM evaluation, this conflict is especially visible: disagreement can arise from ambiguous items, underspecified rubrics, heterogeneous or unstable human raters, or an LLM judge whose verdict changes when the response order is swapped. We propose BayesJudge, an online Bayesian meta-evaluation layer for conflicting human-LLM judgment streams. For each comparison, BayesJudge estimates a panel-relative posterior verdict distribution over the two responses, with a tie or ambiguity state when such labels are available. At the same time, it estimates rater-specific human confusion matrices and LLM presentation-order bias. The method uses tie-open labels to keep ambiguity observable and paired order-swapped judge calls to separate response quality from presentation effects. We formulate the exact online posterior recursion and use a scalable Rao--Blackwellized assumed-density SMC approximation for streaming inference. Controlled synthetic experiments demonstrate recovery of prespecified evaluator parameters and illustrate two protocol-level identifiability mechanisms: tie-open labels expose ambiguity mass, and order-swapped paired judgments separate item preference from position bias. On real-world SummEval dataset, BayesJudge successfully detects systematic presentation-order effects in LLM judge outputs, infers distinct expert and crowdworker behavior signatures without rater metadata, and produces posterior uncertainty estimates that correlate with human disagreement. Our code is available at https://anonymous.4open.science/r/BayesJudge-0879.
Show more
TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
Pei Yang ⋅ Wanyi Chen ⋅ Tongyun Yang ⋅ Pengbin Feng ⋅ Jiarong Xing ⋅ Wentao Guo ⋅ Yuhang Yao ⋅ Yuhang Han ⋅ Hanchen Li ⋅ Xu Wang ⋅ Jie Xiao ⋅ Anjie Yang ⋅ Lynn Ai ⋅ Eric Yang ⋅ TIANYU SHI
LLM routing matters most in long-horizon applications such as coding agents, deep research systems, and computer-use agents, where a single user request triggers many model calls. Routing each call to the cheapest sufficient model can cut costs without sacrificing quality, yet existing router benchmarks evaluate only which model should answer a whole prompt. They never expose the router-visible prefix at an intermediate agent step, never test whether a cheaper replacement preserves downstream task success, and often rely on online LLM judges at evaluation time. We introduce TwinRouterBench, a step-level routing benchmark with two tracks. The static track provides 970 router-visible prefixes from 520 instances across SWE-bench, BFCL, mtRAG, QMSum, and PinchBench, each paired with an execution-verified target tier estimated under a released downgrade-and-cascade protocol; scoring is deterministic arithmetic over tier labels, trajectory membership, and token costs, with no online evaluator-side LLM judge. The dynamic track supplies a harness that runs routers on the full 500-case SWE-bench Verified suite; in this paper we report a 100-case held-out evaluation disjoint from the static SWE supervision split. At each LLM call the router selects a concrete model from a locked pool, and success is measured by official task resolution and realized API spend. The two tracks support fast offline iteration followed by end-to-end validation under live agent execution. Code and data are available at https://anonymous.4open.science/r/TwinRouterBench-8820.
Show more
Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes
Tim Merino ⋅ Sam Earle ⋅ Ryunosuke Iwai ⋅ Julian Togelius ⋅ Edoardo Cetin
We introduce Dream-Cubed, a large-scale dataset of Minecraft worlds at voxel resolution, and a family of models using cubes as powerful compositional units for efficient generation of interactive 3D environments. Dream-Cubed comprises tens of billions of tokens from a carefully curated mixture of procedural biome terrain and high-quality human-authored maps. We use this dataset to conduct the first large-scale study of 3D diffusion models for voxel generation, analyzing discrete and continuous diffusion formulations, data compositions, and architectural design choices. Our models operate directly in the space of blocks, enabling efficient and semantically grounded generation while supporting interactive user workflows such as inpainting and outpainting from user-authored blocks. To quantitatively evaluate our models, we adapt the FID metric to assess semantic differences between real and generated world renderings, and validate generation quality through a human preference study. We release the full dataset, code, and all our pretrained models, which we hope will provide a foundation for future research in efficient generative modeling for structured, interactive 3D environments.
Show more
LinuxArena: A Control Setting for AI Agents in Live Production Software Environments
Tyler Tracy ⋅ Ram Potham ⋅ Nikolas Kuhn ⋅ Myles Heller ⋅ Anshul Khandelwal ⋅ Cody Rushing ⋅ Henri Lemoine ⋅ Miguel Brandão ⋅ Tomáš Turlík ⋅ Adam Hanson ⋅ Josh Hills ⋅ Amy D Ngo ⋅ Ram Rachum ⋅ Nik Mitchell ⋅ Falko Galperin ⋅ Oscar Sykes ⋅ Pip Arnott ⋅ Samuel P Lima ⋅ Carlos R Giudice ⋅ Matt Goldwater ⋅ Daniel J Popp ⋅ Drew de Wet ⋅ Ruben Castaing ⋅ Qi Guo ⋅ Douw Marx ⋅ Benjamin Shaffrey ⋅ Justin Shenk ⋅ Martin Milbradt ⋅ Hannah Meagher ⋅ Shaheen Ahmed-Chowdhury ⋅ Daniel O'Connell ⋅ Christopher W Canal ⋅ Buck Shlegeris ⋅ Aryan Bhatt
As AI agents are given more autonomy in software engineering workflows, the risks grow if they pursue goals different from the ones their users intended. The field of AI control develops control protocols that prevent such harm without restricting the agent's ability to do useful work. We introduce \textbf{LinuxArena}, a control setting for testing such protocols, where agents operate directly on live, multi-service production environments. LinuxArena contains 10 public environments (with an additional 10 held out privately), providing 906 main tasks representing legitimate software engineering work and 92 side tasks representing safety failures such as data exfiltration and access-control bypassing, making it the largest and most diverse control setting for agentic software engineering to date. We demonstrate LinuxArena's utility by running sabotage and monitor evaluations: against a GPT-5 Nano trusted monitor at a 1% step-wise false positive rate, Claude Opus 4.6 achieves an undetected sabotage rate of 42%. We additionally release \textbf{LaStraj}, a dataset of human-crafted attack trajectories that achieves an undetected sabotage rate of 97% against the same monitor. Together, these results suggest meaningful headroom for both attackers and defenders, making LinuxArena a strong testbed for developing and evaluating future control protocols.
Show more
VI-Bench: Benchmarking Prompt Inversion from AIGC Videos
Wulin Xie ⋅ Rui Zhao ⋅ Kecen Li ⋅ Xiujin Liu ⋅ Bokang Zhang ⋅ Zheng Liu ⋅ Xinwen Hou ⋅ Chen GONG
Recent advances in video generation have made prompt-based control increasingly central to AIGC video generation. Prompts specify what a video should depict and how it should be represented, controlling factors such as visual style or camera behavior. Understanding this recoverability is important both for creative reuse and editing, and for assessing prompt leakage risks. However, existing video understanding benchmarks do not measure this capability: a caption may describe what is visible, but a replayable prompt must recover the generation-relevant controls needed to reproduce the video. To address this gap, we introduce VI-Bench, a benchmark built from 16.1 million real-user prompts and 900 human-verified AIGC videos. VI-Bench spans three progressively harder settings, namely single-shot semantic grounding, control over style and camera behavior, and multi-shot compositional inversion, and evaluates five generation-critical dimensions: subject, action, scene, style, and camera. We evaluate 18 representative VLMs, including 2 proprietary and 16 open-source models on VI-Bench, using an Inversion Score that measures prompt-level alignment with the original prompt and video-level fidelity of the regenerated video. The results reveal substantial limitations: even the strongest model achieves only 0.632 on Inversion Score, performance degrades sharply as samples require richer control and multi-shot reasoning, and models often produce plausible prompts whose regenerated videos deviate from the reference. These findings show that video prompt inversion is a distinct and under-evaluated capability requiring models to transform visual understanding into replay-stable generative control.
Show more
Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?
Jiale Liu ⋅ Huajun Xi ⋅ Shaokun Zhang ⋅ Yifan Zeng ⋅ Tianwei Yue ⋅ Chi Wang ⋅ Jian Kang ⋅ Huazheng Wang ⋅ Qingyun Wu
Automated failure attribution uses LLMs to identify where and why agentic systems fail. As agents become more capable, their failures become subtler, making automated attribution increasingly important. We introduce Who&When Pro, a large-scale benchmark for automated failure attribution in agentic systems. Using a strictly controlled pipeline that injects a failure only after exactly replaying a successful prefix, we construct 12,326 failed trajectories with golden labels across 3 modalities and 26 benchmarks covering various scenarios. Beyond benchmarking, we conduct extensive experiments and analyses, revealing systematic patterns in how models attribute failures across modalities, protocols, and model families, and providing empirical guidance for future automated failure attribution systems.
Show more
DRAGON : A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Anirudh Iyengar Kaniyar Narayana Iyengar ⋅ Tampu Ravi Kumar ⋅ Gaurav Najpande ⋅ Manan Suri ⋅ Dinesh Manocha ⋅ Puneet Mathur ⋅ Vivek Gupta
Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics, and scientific diagrams. Recent vision-language models (VLMs) often achieve high answer accuracy on these tasks, yet correct answers do not guarantee that models ground their reasoning in the diagram regions that support the prediction. Models may instead rely on textual correlations or dataset artifacts without identifying the visual evidence required to verify the answer. This limitation prevents reliable evaluation of diagram reasoning and reduces interpretability. We introduce **DRAGON**, a benchmark for evaluating evidence-grounded visual reasoning in diagrams. Given a diagram, a question, and the correct answer, a model must predict bounding boxes that correspond to the visual elements required to justify the answer. These evidence regions may include answer-bearing components, textual labels, legends, axes, connectors, and other supporting structures involved in the reasoning process. The DRAGON dataset contains 11,664 annotated question instances from six diagram QA datasets: ChartQA, Circuit-VQA, InfographicsVQA, MapIQ, MapWise, and AI2D, with a 2,445-instance test set carrying human-verified evidence annotations and a standardized evaluation framework. Our evaluation on recent VLMs reveals that even Claude Opus 4.6 achieves a Grounding IoU of only 23.8% on AI2D, showing that faithful visual grounding remains an open challenge. DRAGON supports future research on models that ground their predictions in visual evidence.
Show more
Benchmarking Open-Ended Multi-Agent Coordination in Language Agents
Kale-ab Tessera ⋅ Andras Szecsenyi ⋅ Cameron Barker ⋅ Alexander Rutherford ⋅ Davide Paglieri ⋅ Aidan Scannell ⋅ Henry Gouk ⋅ Elliot Crowley ⋅ Tim Rocktäschel ⋅ Amos Storkey
As language models are increasingly deployed as autonomous agents, they will need to coordinate with others in long-horizon, open-ended interactive tasks. Yet current evaluations rarely test these demands together, focusing instead on short interactions, single-agent open-ended tasks, or highly structured multi-agent settings. We introduce $alem$, a JAX-based, procedurally generated open-ended benchmark for multi-agent coordination built on Craftax-like dynamics. Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. We evaluate $13$ modern LLMs zero-shot within homogeneous teams, with trained MARL agents as reference points. Most LLM agents struggle, averaging only ~6% of maximum reward, although performance varies widely. Gemini-3.1-Pro-High approaches MARL agents trained for one billion steps on Hard coordination ($17.5\%$ vs. $17.6\%$ Coord.\%), Gemma-4-31B-it is the strongest tested open-weight model, and GPT-5.4-High makes strong base progress, while achieving much lower coordination reward. We find that base task competence does not imply coordination competence, communication helps agents share intent, memory and reasoning help with multi-step planning, and heterogeneous teams regress toward average member performance rather than matching their strongest teammate. These results identify coordination as a distinct bottleneck for current LLM agents, separate from single-agent capabilities. Alem makes this bottleneck measurable, providing a controlled setting for developing agents that can communicate, allocate roles, and execute shared plans.
Show more
OmniMemBench: Towards Scalable Evaluation of Long-Term Omni-Modal Agent Memory
Junhan Shi ⋅ Qiyi Wang ⋅ Xiongwei Wu ⋅ Ming Ma ⋅ Baoqi Pei ⋅ Zhenning Shi ⋅ Yong Jiang ⋅ Qing Li ⋅ Weigao Sun ⋅ Yiran Zhong ⋅ Steven Hoi
Multimodal agents that interact with users over days or months must remember not only what was said, but what was shown, heard, and played. Current memory benchmarks are predominantly text-only or cover at most two modalities. Those that do incorporate multimodal content still evaluate with aggregate scores that conflate retrieval and reasoning errors, and provide no mechanism to test whether performance degrades as context scales, leaving our understanding of agent memory fundamentally incomplete. We introduce OmniMemBench, to our knowledge the first long-term memory benchmark that spans image, audio, and video in multi-session dialogue. Its design is guided by three requirements for reliable evaluation. First, an anti-leakage mechanism filters questions answerable from text alone, reducing text-only solvability to a negligible level. Second, two LLM-judged retrieval metrics—Clue Coverage and Entry Relevance—operate on content rather than entry IDs, enabling unified retrieval diagnosis across RAG and memory paradigms. The gap between retrieval coverage and downstream QA scores further localizes failures to the memorization stage. Third, content-isolated distractor injection scales context from 128K to 1M tokens while holding evidence constant, so scalability can be measured without confounding difficulty. The benchmark covers 103 characters and 6,655 QAs across 8 task types and 4 context tiers. Evaluating representative long-context, RAG, and structured-memory methods, we find that no single paradigm dominates: long-context models lead overall, yet RAG surpasses them on targeted fact extraction. Further analysis shows that memory methods retrieve clues at coverage rates comparable to RAG but score substantially lower on QA, revealing that the bottleneck is not what is retrieved but what is lost during memorization. Caption-based processing irreversibly discards perceptual details; raw embeddings degrade retrieval accuracy; and scaling the memorization model yields negligible gains.
Show more
Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases
Jhen-Ke Lin
Most coding-agent benchmarks ask whether generated code behaves correctly. That remains essential, but repository-level engineering is increasingly agent-managed: one agent writes a repository, and later agents inspect, audit, or extend it as working context. In that setting, a generated repository is not only an answer to a task but also a communication artifact for future work. Even when strong agents nearly satisfy the visible behavioral objective, repositories can differ in how clearly they expose the intended behavior and design choices behind that behavior. We introduce Build-and-Find, a protocol for evaluating whether downstream agents can recover those intended choices from generated repositories, and how much inspection that recovery requires. For each task, a builder sees a hidden repository specification and creates a codebase; a finder sees only the codebase and a specification-traced multiple-choice question bank. The protocol separates behavioral correctness from artifact-side recovery and reports recovery accuracy, repeatability, implementation coverage, and inspection effort. Accuracy and stability act as gates: effort is interpreted only when recovery succeeds reliably. Among artifacts from which the same intent can be recovered, lower effort by the same finder suggests that the artifact makes that intent easier to locate. Question-only and spec-only controls quantify generic priors and specification access, while audits separate omitted claims from finder failures and check whether correct answers cite artifact evidence. In the released high-prior task pack, recovery accuracy is near saturation, so inspection effort and finder-specific effects provide the main panel-local comparison. We release the harness, tasks, generated artifacts, records, canonical tables, reports, scripts, metadata, licenses, and evidence audit to support auditable extensions and private task packs.
Show more
BioSafetyBench: Agentic Cascade Evaluation for the Bio-AI Ecosystem
Yang Cao ⋅ Xiaogeng Liu ⋅ Shengchao Liu ⋅ Chaowei Xiao
Biological risk is often a property of a cascade rather than a single model output. A generated artifact may become consequential only after transcription, translation, folding, binding, pathway propagation, or host-directed off-target interaction. Existing bio-AI safety evaluations largely grade each model within its own modality, missing downstream risks and inheriting self-grading circularity when model and evaluator share representations. We introduce BioSafetyBench, an agentic cascade-evaluation platform that treats the bio-AI ecosystem as the unit of safety evaluation. BioSafetyBench routes each model-output pair to one of 15 standardized probes over a six-level hierarchy from genome to organism-level outcome. Fixed bridge agents propagate outputs through two causal pathways: Pipeline A follows forward biological cascades, while Pipeline B evaluates CRISPR and siRNA designs through genome- and transcriptome-wide off-target alignment. At every reached level, external frozen evaluators compute domain-native biomarkers and compare them with literature-anchored thresholds, yielding a Cascade Concern Profile and Cascade Depth. Across 25 foundation models, BioSafetyBench surfaces risks invisible to single-modality grading: Influenza neuraminidase designs from ProteinMPNN and ESM-IF1 reach CD = 4; jailbreak prompts inflate gRNA off-target hits 9.05× over baseline, push 48 of 604 gRNAs to Critical, and concentrate Critical siRNA hits on MYC; protein mask-and-fill spans a 32× cross-model gap; GPT-4o complies on all natural-language-guided protein mutation prompts at CD = 2 while Claude Sonnet 4.5 refuses all prompts at CD = 0; and a three-predictor ADMET ensemble misses 30.3% of known clinical toxics. BioSafetyBench is, to our knowledge, the first bio-AI safety benchmark to provide cascade-level risk certificates whose scoring modules are weight- and gradient-flow independent from the model under test by construction.
Show more
Multi-Oracle Agreement Reveals the Limits of Self-Consistency Evaluation in RNA Design
Minghao Sun ⋅ Hanqun Cao ⋅ Fang Wu ⋅ Zhou Zhang ⋅ ZHIYUAN LIU ⋅ Tianfan Fu ⋅ Pheng-Ann Heng ⋅ Yang Zhang
Computational RNA design methods routinely evaluate designed sequences by refolding them with the same structure-prediction model used during optimization, measuring agreement with the design target. While scalable, this practice carries a fundamental risk: when optimization and evaluation share the same underlying model, high scores can reflect model-specific idiosyncrasies rather than genuine structural quality. To characterize the severity and mechanism of this problem, we audited nine RNA design methods spanning four algorithmic paradigms, evaluating approximately 11,000 sequences against predictors from two independent model families: physics-based thermodynamic models and machine-learning models, alongside five tertiary-structure predictors and large-scale SHAPE chemical-mapping data. We find that 78.1% of designs produced by single-objective optimization pass one predictor yet fail another, while ensemble-based methods reduce this disagreement 30-fold. A controlled experiment identifies optimization objective breadth, not search strategy, as the primary driver of this fragility. Cross-predictor disagreement carries a calibrated experimental signal: lower disagreement predicts higher SHAPE accuracy across 40,000 sequences, yet remains null-to-anti-predictive for biological function, establishing a clear scope boundary. We release **ACCORD** (**A**cross-oracle **C**alibration for **CO**mputational **R**NA **D**esign), a benchmarking workflow that reports per-predictor scores, experimentally calibrated uncertainty tiers, and explicit warnings for invalid use cases, providing the community with a principled foundation for RNA design evaluation.
Show more
A Benchmark for Omni-Modal Reasoning in Long Videos
Mohammed Irfan Kurpath ⋅ Jaseel M Kaithakkodan ⋅ Jinxing Zhou ⋅ Sahal Shaji Mullappilly ⋅ Mohammad Almansoori ⋅ Noor Ahsan ⋅ Beknur Kalmakhanbet ⋅ sambal shikhar ⋅ Rishabh Lalla ⋅ Jean Lahoud ⋅ Mariette Awad ⋅ Fahad Shahbaz Khan ⋅ Salman Khan ⋅ Rao Anwer ⋅ Hisham Cholakkal
Long-form omni-modal video understanding requires models to integrate vision, speech, and ambient audio with coherent long-context reasoning. Existing video benchmarks often trade off temporal scale, modality coverage, open-ended interaction, and interpretable scoring. To address this gap, we introduce $\textbf{LongShOTBench}$, a long video evaluation benchmark designed around three coupled goals: holistic omni-modal integration, intent-driven open-ended interaction, and rubric-level diagnosis. It constructs single- and multi-turn questions from practical viewing scenarios, involving systematical tasks for probing visual, speech, ambient-audio, temporal, and cross-modal reasoning. Each item includes a reference answer and a weighted criterion-level rubric, enabling evaluation to identify which perceptual facts, temporal links, modality-grounding requirements, and reasoning steps are satisfied or missed. All samples are manually verified and corrected to improve grounding, clarity, and rubric reliability. We also introduce $\textbf{LongShOTAgent}$, a training-free omni-modal evidence-seeking agent that couples full-video preprocessing with targeted retrieval, query-adaptive segment refinement, and explicit claim verification over visual, speech, and non-speech audio evidence. Its iterative search-refine-verify loop exposes intermediate evidence and lets modality-specific specialists re-analyze relevant moments before answering. We perform comprehensive evaluation of 105 video-capable models spanning open-source omni-modal models, vision-language systems, audio LLMs, agentic pipelines and closed-source APIs. Across this broad evaluation, current MLLMs remain far from saturating LongShOTBench, while our LongShOTAgent emerges as the strongest training-free system, reaching 66.64\% overall. By releasing the benchmark, leaderboard, and agentic method, our work provides the community with a shared, interpretable testbed for evaluating and advancing long-form omni-modal video reasoning.
Show more
RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology
Lingli Ge ⋅ Junyuan Gao ⋅ Jiahe Song ⋅ Jiaxing Sun ⋅ Yubin Wang ⋅ Boyu Zhu ⋅ Haote Yang ⋅ Jingchao Wang ⋅ Lixin Ma ⋅ Jiang Wu ⋅ Yuqiang Li ⋅ Conghui He
Chemical reaction-condition optimization --- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize yield and stereoselectivity --- is a central, judgement-laden subtask of organic methodology research that large language models are increasingly expected to support. Yet existing chemistry benchmarks evaluate reaction-class labelling, retrosynthesis, or SMILES manipulation, and do not ask models to read a real condition-screening table and pick the best set. We introduce RxnOptBench, a benchmark whose every option and precedent is a real wet-lab entry mined from the optimization tables of organic-methodology papers published in 2025, graded by a continuous relative score that fuses yield with stereoselectivity (ee, dr, rr) so near-correct answers are not collapsed to zero, and equipped with a paired precedents-vs-no-precedents design that isolates in-context use of literature evidence from parametric memorization. Across nine frontier LLMs and three Chemistry LLMs, even the best models leave substantial headroom: chemistry-specialized models fall to the random-baseline floor on multi-axis selection, while open-weight models have closed most of the gap to proprietary frontier models. We release the benchmark, the human-verified extraction pipeline, and the inference and evaluation code.
Show more
Forced Orders: What LLM Leaderboards Hide About Model Comparisons
Zonglin Di ⋅ Berk Ustun ⋅ Yang Liu
LLM benchmarks are usually consumed as scalar leaderboards, where adjacent ranks are read as adjacent capability differences. We argue that this display should be complemented rather than simply trusted or discarded: benchmark scores are useful summaries, but a sorted score table can overstate which model pairs are actually distinguishable. This limitation is especially clear for accuracy- and success-rate leaderboards, whose averages hide task-level structure, and it also appears in pairwise systems such as Elo-style ratings when pairwise evidence is compressed into a total order. We propose LLM tier ranking as a task-level reporting framework for benchmarks. Given benchmark outputs, we construct pairwise wins, losses, and ties between models on each task and aggregate them into ordered tiers using Selective Preference Aggregation (SPA). Cross-tier comparisons are reported only when supported by sufficient task-level agreement; models in the same tier are left unresolved, not declared equal. Across AIME 2025, $\tau$-bench airline, xbench DeepSearch, GDPVal, SWE-Bench Verified, and Chatbot Arena, tier ranking often preserves useful broad capability classes while abstaining from unsupported fine-grained distinctions among frontier models. In Chatbot Arena perturbation studies, tier outputs are more stable than Elo-style total orders, expressing weaker evidence through later tier separation rather than rank reshuffling. We recommend reporting tiered leaderboards, solution paths, top-tier size, singleton-winner thresholds, and pairwise resolution statistics alongside scalar scores.
Show more
ClinMAS: A Knowledge-Grounded Multi-Agent Simulation Framework for Evaluating Clinical Reasoning in LLMs
Yuqi Tang ⋅ Jing Yu ⋅ Zichang Su ⋅ Kehua Feng ⋅ Zhihui Zhu ⋅ Libin Wang ⋅ Lei Liang ⋅ Qiang Zhang ⋅ Keyan Ding ⋅ Huajun Chen
Clinical diagnosis begins with doctor-patient interaction, during which physicians iteratively gather information, order examinations, and refine diagnosis through patients’ responses. This dynamic reasoning process is poorly represented by existing LLM benchmarks that focus on static question-answering. To mitigate these gaps, recent methods explore dynamic medical frameworks involving interactive clinical dialogues. Although effective, they often rely on limited, contamination-prone datasets and lack granular evaluation. In this work, we propose ClinMAS, a knowledge-grounded multi-agent simulation framework for evaluating clinical reasoning in LLMs. Grounded in a disease knowledge graph, our method dynamically generates patient cases and facilitates multi-turn interactions between an LLM-based doctor, an automated patient agent, and an examiner agent. Our evaluation goes beyond diagnostic accuracy by incorporating fine-grained efficiency analysis and rubric-based assessment of diagnostic quality. Experiments show that ClinMAS effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs, offering a more nuanced and clinically meaningful evaluation paradigm.
Show more
Evaluating Neural Data Tokenizers: A Framework for Assessing Learned Representations of Spiking Activity
Federico D'Agostino ⋅ Alex Gilbert ⋅ Susanne Keller ⋅ Jaivardhan Kapoor ⋅ Nicolas Reategui ⋅ Vedang Lad ⋅ Matthias Kümmerer ⋅ Jai Bhagat ⋅ Emin Orhan ⋅ Hasan A Bedel ⋅ Nikos Karantzas ⋅ Fabian Sinz ⋅ Surya Ganguli ⋅ Jakob H Macke ⋅ Sophia Sanborn ⋅ Alexander Ecker ⋅ Katrin Franke ⋅ Matthias Bethge ⋅ Andreas Tolias ⋅ Konstantin Willeke
Foundation models of brain activity aim to learn representations from large-scale neural recordings that generalize across sessions, subjects, species, and downstream tasks. Transformer-based models have emerged as a promising architecture for this goal and begin all with the same first step --- tokenization: transforming sparse binary events into vector representations that downstream networks can consume. In other areas of machine learning, tokenization is its own design problem, distinct from the downstream foundation model. Several mostly implicit tokenization approaches dominate the literature, but their impact on downstream performance has not been systematically evaluated. We propose four explicit desiderata for a neural tokenizer---reconstruction fidelity, downstream utility, cross-instance generalization, and compression---and a framework of metrics spanning seven dimensions, including firing statistics, encoding and decoding, and latent structure and robustness. We assemble a benchmark of ten datasets spanning species, brain areas, recording technologies, and behavioral paradigms, and evaluate a representative set of discrete and continuous tokenizers, both single-neuron and population variants. Across the strategies evaluated, we show how tokenizers trained only on reconstruction can support downstream tasks. Yet reconstruction alone is misleading: reconstruction quality and downstream utility can be sharply dissociated. Our multi-axis framework is a response to this failure mode. These findings highlight tokenization as a critical and underexplored design axis in brain foundation models, and suggest that further progress will require continued systematic study of tokenization strategies.
Show more
What Drives Test-Time Adaptation for CLIP? A Controlled Empirical Study from an Update Perspective
Jiazhen Huang ⋅ Xiao Chen ⋅ Zhiming Liu ⋅ Yaru Sun ⋅ Jingyan Jiang ⋅ Zhi Wang
Vision-Language Models (VLMs) such as CLIP have become a standard backbone for open-vocabulary recognition, yet their zero-shot predictions remain vulnerable to distribution shifts encountered at deployment. Test-Time Adaptation (TTA) has recently been extended to CLIP as a lightweight solution, leading to a rapidly growing body of TTA4CLIP methods. However, empirical progress in this area has largely outpaced our understanding of what truly drives adaptation, where their gains originate, and under which shifts they remain reliable. In this paper, we take a step back from the pursuit of state-of-the-art accuracy and conduct a systematic controlled study of TTA4CLIP. We first organize existing methods into three unified paradigms according to what is updated at test time. We then introduce TTABC, an open-source TTA Benchmark for CLIP, which standardizes evaluation protocols and integrates more than 20 representative methods. Our controlled empirical analysis focuses on three key areas. First, we determine the driving factors in parameter-based methods, revealing that adaptation gains are primarily driven by test-time evidence and reliable proxies rather than heavy optimization. Second, we explore evidence utilization beyond heavy parameter tuning, showing that competitive and efficient performance can be achieved through cross- or current-sample evidence and lightweight prototype updates. Finally, we demonstrate that there is no silver bullet for TTA; no single adaptation paradigm is universally optimal, and the preferred paradigm depends on the nature of shift. We hope our benchmark and study provide a clearer understanding of the current TTA4CLIP landscape and establish a foundation for further research.
Show more
RISE-Video: Can Video Generators Decode Implicit World Rules?
Mingxin Liu ⋅ Shuran Ma ⋅ Shibei Meng ⋅ Xiangyu Zhao ⋅ Zicheng Zhang ⋅ Shaofeng Zhang ⋅ Zhihang Zhong ⋅ Peixian Chen ⋅ Haoyu Cao ⋅ Xing Sun ⋅ Haodong Duan ⋅ Xue Yang
While generative video models have achieved remarkable visual fidelity, their capacity to internalize and reason over implicit world rules remains a critical yet under-explored frontier. To bridge this gap, we present RISE-Video, a pioneering reasoning-oriented benchmark for Text-Image-to-Video (TI2V) synthesis that shifts the evaluative focus from surface-level aesthetics to deep cognitive reasoning. RISE-Video comprises 467 meticulously human-annotated samples spanning eight rigorous categories, providing a structured testbed for probing model intelligence across diverse dimensions, ranging from commonsense and spatial dynamics to specialized subject domains. Our framework introduces a multi-dimensional evaluation protocol consisting of four metrics: \textit{Reasoning Alignment}, \textit{Temporal Consistency}, \textit{Physical Rationality}, and \textit{Visual Quality}. To further support scalable evaluation, we propose an automated pipeline leveraging Large Multimodal Models (LMMs) to emulate human-centric assessment. Extensive experiments on 11 state-of-the-art TI2V models reveal pervasive deficiencies in simulating complex scenarios under implicit constraints, offering critical insights for the advancement of future world-simulating generative models.
Show more
Many Benign Errors are Better than A Few Severe Ones: Evaluating Hallucination Severity
Sanjana Ramprasad ⋅ Pranav Mani ⋅ Chenhao Tan ⋅ Byron Wallace ⋅ Elisa Ferracane
Factuality evaluation in summarization has largely focused on detecting hallucinations as binary phenomena---labeling content as either supported or unsupported by the source. However, modern LLMs increasingly generate nuanced, partially grounded content whose factual impact can vary widely—from benign inferences to severe distortions. To enable more precise assessment, we propose a framework for annotating hallucination \emph{severity} along two interpretable dimensions: \emph{implausibility} (how unreasonable a span is given the source or general knowledge) and \emph{impact} (its potential to mislead or cause harm if assumed true). Applying this framework to obtain annotations on model generations across domains, we find that many hallucinations are both plausible and benign, indicating that not all unsupported content is equally consequential. We further show that severity-aware evaluation can meaningfully alter model rankings, favoring models that generate more frequent yet benign hallucinations over those producing fewer but more detrimental errors—distinctions that conventional evaluation settings fail to capture. Finally, we evaluate LLMs as severity judges: while their span detection remains weak, their severity ratings align closely with human judgments, making them promising candidates for scaling fine-grained annotation. Using severity as a diagnostic lens, we find that LLMs preferentially detect obvious errors but miss plausible, high-impact ones---precisely the cases that matter most for deployment. Together, these findings argue for a shift toward severity-aware factuality evaluation.
Show more
MEV: A Multi-Event Video Dataset for Long-Take Generation
Peiyuan Zhu ⋅ Shaoan Xie ⋅ Yifan Shen ⋅ Wen Tian ⋅ Zijian Li ⋅ Guangyi Chen ⋅ Kun Zhang
Current datasets for text-to-video generation are largely limited to short clips depicting a single event or coarse-grained captions. As a result, they provide limited supervision for modeling long-take temporal dependencies, often leading to temporally inconsistent generations when describing multiple events. In this paper, we introduce MEV, a multi-event video dataset designed for long-take video generation. MEV provides high-quality real-world videos where subjects can perform a sequence of distinct, non-overlapping events, featuring precise temporal boundaries and fine-grained procedural annotations. This event-centric discretization reduces confounding factors and enables models to support temporally localized conditioning and build long-take videos from atomic event units. Furthermore, we introduce a comprehensive annotation pipeline that segments videos into temporally grounded events, and generates structured captions at both the global level and the event level. In addition, under an event-centric evaluation protocol, we conduct multi-event generation experiments by fine-tuning on MEV with temporally grounded event conditioning, leading to improved long-take event transitions and more coherent content. We believe this work removes a key barrier to training video foundation models for multi-event scenarios and provides practical insights into learning from temporally-structured video-text data.
Show more
XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding
Hongchen Wei ⋅ Yuanzhe Wang ⋅ Bei Liu ⋅ Yifan Yang ⋅ Qi Dai ⋅ Ruichun Ma ⋅ Kai Qiu ⋅ Yunsheng Li ⋅ Dongdong Chen ⋅ Chong Luo ⋅ Zhenzhong Chen ⋅ Baining Guo
Real-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages. Some questions also require comparing related reports. Reliable long-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high -- yet most existing benchmarks still measure short-context or single-page QA. We introduce \textbf{XL-DocBench}, a fully human-verified benchmark for extra-long document understanding, with 1519 retained questions from six professional domains and contexts up to 2,303 pages. XL-DocBench goes beyond page-level lookup. 1,103 examples (72.6\%) use multiple evidence pages. The final set also includes 556 questions (36.6\%) that use tables, charts, or figures, and 165 questions (10.9\%) that require evidence from multiple documents. Each question has one of twelve reasoning labels, expert-annotated evidence pages, a typed verification rule, and an answer format, including 218 \texttt{None}-answer cases. We build the benchmark with a tree-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts. By coupling extra-long professional contexts with page-level evidence and typed rules, XL-DocBench fills a gap left by prior single-page, short multi-page, or text-only long-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score. The results show that current systems still struggle with long contexts, multi-page evidence, and structured reasoning over professional documents.
Show more
COOP$^2$: Defining, Observing, and Repairing Cooperation in LLM Multi-Agent Systems
Hanqing Yang ⋅ Narjes Nourzad ⋅ Shiyu Chen ⋅ Marie Siew ⋅ Jingdi Chen ⋅ Carlee Joe-Wong
Many complex tasks require extended effort, diverse capabilities, or coordinated actions beyond what a single agent can provide. However, simply adding more agents does not guarantee better performance, as effective cooperation depends on how agents interact with each other and with task structure to satisfy evolving constraints over time. This challenge is amplified for LLM-based multi-agent systems (LLM-MAS): plans, messages, and revisions occur in natural language, whereas task progress depends on grounded environment actions. Current evaluations mostly treat cooperation as an implicit ingredient of final task success, leaving both cooperation and the effect of multi-agent interaction on task dynamics difficult to study. We introduce COOP$^2$, an evaluation framework that grounds high-level agent cooperation dynamics in LLM-MAS within task progress in the environment. COOP$^2$ then defines cooperative tasks with verifiable cooperative requirements, allowing us to analyze how cooperation unfolds over time with respect to task progress, as well as where and why cooperation breaks down. Building on this framework, we develop COOP$^2$-Repair, which predicts constraint failures from group plans and opens targeted repair channels for guided revisions. Across two environments and three communication structures, COOP$^2$-Repair improves task success and constraint satisfaction while exposing the additional decision overhead and communication load required for repair.
Show more
LLM-WikiRace: A Benchmark for Planning and Reasoning over Real-World Knowledge Graphs
Juliusz Ziomek ⋅ William Bankes ⋅ Lorenz Wolf ⋅ Shyam Sundhar Ramesh ⋅ Xiaohang Tang ⋅ Ilija Bogunovic
We introduce **LLM-Wikirace**, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs). In LLM-Wikirace, models must efficiently navigate Wikipedia hyperlinks step by step to reach a target page from a given source, requiring look-ahead planning and the ability to reason about how concepts are connected in the real world. We evaluate a broad set of open- and closed-source models, including Gemini-3, GPT-5, and Claude Opus 4.5, which achieve the strongest results on the easy level of the task and demonstrate superhuman performance. Despite this, performance drops sharply on hard difficulty: the best-performing model, Gemini-3.1, succeeds in only 29\% of hard games, highlighting substantial remaining challenges for frontier models. Our analysis shows that world knowledge is a necessary ingredient for success, but only up to a point, beyond this threshold, planning and long-horizon reasoning capabilities become the dominant factors. Trajectory-level analysis further reveals that even the strongest models struggle to replan after failure, frequently entering loops rather than recovering. LLM-Wikirace is a simple benchmark that reveals clear limitations in current reasoning systems, offering an open arena where planning-capable LLMs still have much to prove.
Show more
VecDBLens: A Modular Framework for Diagnosing Vector Databases Retrieval Pipelines
Yutong Zhou ⋅ Guoxin Kang ⋅ Wenxin Zhou ⋅ Haoyu He ⋅ Yuedong Zhu ⋅ Lei Wang ⋅ Wanling Gao ⋅ Fanda Fan ⋅ Jianfeng Zhan
Vector databases are increasingly deployed as retrieval pipelines in which representation, indexing, and query-time execution jointly determine accuracy, throughput, and cost. Existing vector database evaluations primarily focus on indexing while treating embeddings as fixed inputs and ignoring the role of query-time execution, making it difficult to decide whether a bottleneck should be addressed by changing the embedding model, the index, or the query-processing stages. To address the above limitation, we introduce \textbf{VecDBLens}, a modular attribution framework that treats the component under diagnosis as an \emph{evaluated object} and all remaining dataset, workload, parameter, and runtime factors as matched \emph{evaluation conditions}. VecDBLens reports stratified marginal effects, paired matched-condition differences, Flat-relative retention, and monotonicity audits, thereby turning vector database benchmarking from leaderboard ranking into a reproducible diagnostic procedure. VecDBLens reveals two key insights. First, retrieval performance is fundamentally upper-bounded by embedding quality, revealing that many apparent indexing failures are therefore representation-side limitations. Second, stronger index parameterization is not always beneficial: overly aggressive configurations can reshape neighborhood topology and cause recall regression. These results challenge the common practice of evaluating vector databases primarily through indexing and show that real-world limitations emerge from stage-wise constraints rather than any single module alone. VecDBLens provides a principled basis for identifying the true performance drivers of vector databases and for guiding their design in practical applications. The code is available at \url{https://anonymous.4open.science/r/VecDBLens-83A4}.
Show more
FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting
Yiyun Zhu ⋅ Yidong Jiang ⋅ Ziwen Xu ⋅ Yinsheng Yao ⋅ Dawei Cheng ⋅ Jinru Ding ⋅ JIE XU
Large language models (LLMs) are increasingly deployed in financial research workflows, where their role is evolving from single-model assistance for human analysts toward autonomous collaboration among multiple agents. Yet recent real-world deployments reveal persistent failures such as factual errors, numerical inconsistencies, and shallow analysis, which can distort assessments of corporate fundamentals and trigger severe economic losses. While existing benchmarks have begun to evaluate such failures, they score all aspects of the generated analysis in one pass, failing to distinguish whether a model fails at foundational stages like auditing and correction, or underperforms at generating research-grade insights. Consequently, it obscures capability bottlenecks and the specialized strengths essential for multi-agent role assignment. To address these gaps, we introduce FinReasoning, a hierarchical benchmark that decomposes the core capabilities of financial research into semantic consistency, data alignment, and deep insight. We further propose a fine-grained evaluation framework that strengthens hallucination-correction assessment and incorporates a 12-indicator rubric for core analytical skills. FinReasoning reveals clear capability stratification across model types. Closed-source models (like Doubao-Seed-1.8) perform strongly overall and are better suited for core reasoning agents in multi-agent financial systems; open-source general models (like Qwen3-235B) show clear capability divergence and consistently underperform on Semantic Consistency, making them less suited for quality-sensitive generation tasks; financial-domain models (like Fin-R1) generate moderate insights but lack foundational auditing skills, requiring upstream quality control for domain-specific analysis. Our work provides a reliable evaluation foundation for multi-agent financial application systems and has already been deployed in pilot tests across several real-world scenarios. The resource is available at https://anonymous.4open.science/r/FinReasoning.
Show more
Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
Tyler Chang ⋅ Catherine Arnett ⋅ Abdelrahman Sadallah ⋅ Abdelrahman Eldesokey ⋅ Abeer Kashar ⋅ Daud Abolade ⋅ Abosede G Olanihun ⋅ Adam Labaran M ⋅ Adhikarimayum Meerajita Sharma ⋅ Aditi Gupta ⋅ Adril P Merin ⋅ Adwoa A Bremang ⋅ Afitab Iyigun ⋅ Afonso Simplício ⋅ Ahmed Essouaied ⋅ Aicha Chorana ⋅ Akhil Eppa ⋅ Akintunde Oladipo ⋅ Akriti Kuri ⋅ Akshay Ramesh ⋅ Aleksei Dorkin ⋅ Alfred M Kondoro ⋅ Alham Aji ⋅ Ali E Çetintaş ⋅ Allan Hanbury ⋅ Alp Niksarli ⋅ Alvaro Arroyo ⋅ Amin Bajand ⋅ Amol Khanna ⋅ Ana Chkhaidze ⋅ Ana C Condez ⋅ Anamaria Roberta Hartl ⋅ Andiswa Mkhonto ⋅ Andrew Hoblitzell ⋅ Angelos Poulis ⋅ Anirban Majumder ⋅ Anjali Chaudhary ⋅ Anna Vacalopoulou ⋅ Annika Simonsen ⋅ Anton Kovalev ⋅ Anupam Nayak ⋅ Ashvanth.S ⋅ Ayodeji Lana ⋅ Ayu Purwarianti ⋅ Bashar Alhafni ⋅ C. Benedict C. Busole ⋅ Bernard Ghanem ⋅ Bharti Nathani ⋅ Biljana Stojanovska Đurić ⋅ Ogundipe B Funmilola ⋅ Bolaotan Agbonile ⋅ Bruce Torres Fischer ⋅ Burak Tutar ⋅ Burcu Çınar ⋅ Cade Kane ⋅ Can Udomcharoenchaikit ⋅ Chadi Helwe ⋅ Cecilia Liu ⋅ Chiamaka G Nwokolo ⋅ Christopher Homan ⋅ Clément Sampebgo ⋅ Cristina España-Bonet ⋅ Cynthia Amol ⋅ Daeyeop Lee ⋅ Dan Saattrup Smart ⋅ Dana Arad ⋅ Daniil Dzenhaliou ⋅ Dasol Choi ⋅ David Liu ⋅ David Semedo ⋅ David Anugraha ⋅ OLAOYE D Oluwaseun ⋅ Deborah Popoola ⋅ Deividas Mataciunas ⋅ Dennis Asamoah Owusu ⋅ Dhyuthy Krishna Kumar ⋅ Diogo Tavares ⋅ Diogo Glória-Silva ⋅ Divyanshu Goyal ⋅ DongGeon Lee ⋅ Estefany Kelly Buchanan ⋅ Ebele N Anajemba ⋅ Ngozi G Egonu ⋅ Elias Herranen ⋅ Eman Nisar ⋅ Emile Anand ⋅ Emuobonuvie M Ajiboye ⋅ Eryawan P Yulianrifat ⋅ Esther Adenuga ⋅ Ewa Rudnicka ⋅ Faith O Itiola ⋅ Fareeha Fayyaz Sheikh ⋅ Fathima Thekkekara ⋅ Fatima Haouari ⋅ Faustin NSENGIYUMVA ⋅ Fenal A Ilasariya ⋅ Filbert A Tjiaranata ⋅ Firas Laakom ⋅ Francesca Grasso ⋅ Francesco Periti ⋅ Francesco Orabona ⋅ Gbenga K Solomon ⋅ Genta Winata ⋅ Nghia G Ngo ⋅ Gloria Udhedhe-oze ⋅ Gonçalo Vinagre ⋅ Gopi N Challagolla ⋅ Gorka Urbizu-Garmendia ⋅ Gouthami Vadithya ⋅ Guijin Son ⋅ Gyan S Mohapatra ⋅ Hafeez Ullah ⋅ Hafsteinn Einarsson ⋅ Hai Hu ⋅ Hamidreza Saffari ⋅ Hamza Zaidi ⋅ Haopeng Zhang ⋅ Harethah Abu Shairah ⋅ Hele-Andra Kuulmets ⋅ Hitesh Patel ⋅ Houda Bouamor ⋅ Hwanjo Yu ⋅ Iben N Debess ⋅ İbrahim Ethem Deveci ⋅ Ikhlasul A Hanif ⋅ Ikhyun Cho ⋅ Inês Vieira ⋅ Inês Calvo ⋅ Ismail O Daud ⋅ Yusuf I Abayomi ⋅ Itay Itzhak ⋅ Ivan Zhelyazkov ⋅ Ivan Belashkin ⋅ Ivan Spada ⋅ Jacob Brinton ⋅ Jafar Isbarov ⋅ Jaka Čibej ⋅ Jan Kocon ⋅ Jan Cuhel ⋅ Jauza A Krito ⋅ Jebish Purbey ⋅ Jennifer Za ⋅ Jennifer Mickel ⋅ Jenny Kunz ⋅ Jessica Ratovondranto ⋅ Jeyarajalingam Varsha ⋅ Jihae Jeong ⋅ Jimena T Dávalos ⋅ Jinu Lee ⋅ Joao Magalhaes ⋅ John S Yi ⋅ Jongin Kim ⋅ Joseph Chataignon ⋅ Joseph Marvin Imperial ⋅ Jubeerathan Thevakumar ⋅ Judith Land ⋅ Iuliia Alekseenko ⋅ Jiang Junchen ⋅ Jungwhan Kim ⋅ Kairit Sirts ⋅ Kamesh R ⋅ Kanda Patrick Tshinu ⋅ Kätriin Kukk ⋅ Kaustubh Ponkshe ⋅ Kavsar Huseynova ⋅ Ke He ⋅ Kenneth Enevoldsen ⋅ Kent J Alvarez ⋅ Kerem Zaman ⋅ Khalil Mrini ⋅ Kian Kyars ⋅ Komal Gour ⋅ Krishnakumar Lainitha ⋅ Krister Kruusmaa ⋅ Kunal Mukherjee ⋅ Kusum Chouhan ⋅ Laura Castro ⋅ Laura M Porrino-Moscoso ⋅ Lenny S Nzambi ⋅ Leshem Choshen ⋅ Lilja Øvrelid ⋅ Lisa Alazraki ⋅ Lovina Ehimen-Ugbede ⋅ Luheerathan Thevakumar ⋅ Luxshan Thavarasa ⋅ Mamadou K KEITA ⋅ Mansi Jangid ⋅ Marco De Santis ⋅ Marcos Garcia ⋅ Marek Suppa ⋅ Mariam D'Ciofalo ⋅ Marii Ojastu ⋅ Marium Atta ⋅ Maryam Sikander ⋅ Maximos Skandalis ⋅ Mehak Soomro ⋅ Mehmet İlteriş Bozkurt ⋅ Melaku Bayu ⋅ Menan Velayuthan ⋅ Michael Leventhal ⋅ Michał Marcińczuk ⋅ Mina Almasi ⋅ Mirna Potočnjak ⋅ Mithil Bangera ⋅ Mohammadamin Shafiei ⋅ MOHIBA ANSARI ⋅ Mridul Sharma ⋅ Mrityunjaya Indoria ⋅ Mughees U Rehman ⋅ Muhammad Ravi Shulthan Habibi ⋅ Murat B Kınay ⋅ Nada Galant ⋅ Naina S Rathore ⋅ Narada Maugin ⋅ Nathalie H Norman ⋅ Nicholas Kluge Corrêa ⋅ Nikola Ljubešić ⋅ Nirmal Thomas ⋅ Nisansa de Silva ⋅ Nisheeth Joshi ⋅ Nitish Ponkshe ⋅ Nizar Habash ⋅ Nneoma C Udeze ⋅ Noel Thomas ⋅ Noémi Ligeti-Nagy ⋅ Odunayo Ogundepo ⋅ Odunayo K Buliaminu ⋅ God'spraise Okechukwu ⋅ Olanrewaju Samuel ⋅ Olga Snissarenko ⋅ Onyinye A Chiemezie ⋅ Orkun Kınay ⋅ osman tursun ⋅ Pablo Rodríguez ⋅ Pablo Gamallo ⋅ Pedro Valente ⋅ Peter Rupnik ⋅ Philip O. EKIUGBO ⋅ Prakhar Agarwal ⋅ Prokopis Prokopidis ⋅ Quadri Yahya ⋅ Rachele Mignone ⋅ Raghav Singhal ⋅ Rahul Raja ⋅ Ram Mohan Rao Kadiyala ⋅ Raphaël Merx ⋅ Rasmus Larsen ⋅ Ratnavel Rajalakshmi ⋅ Rishav Ghosh ⋅ Romina Oji ⋅ Rui P Guerra ⋅ Rushikesh Zawar ⋅ Sa'ad N Bashir ⋅ Saeed Alzaabi ⋅ Sahil Sandeep ⋅ Sai Sandeep Kantareddy ⋅ Saleha Muzammil ⋅ Salsabila Z Pranida ⋅ Sam Buchanan ⋅ Samuel Rutunda ⋅ Sander Land ⋅ Sarah Sulollari ⋅ Sardar Ali ⋅ Kengatharaiyer Sarveswaran ⋅ Saulius Tautvaisas ⋅ Sayambhu Sen ⋅ Sayantani Banerjee ⋅ Sebastien Diarra ⋅ Sewoong Lee ⋅ Shaan Shah ⋅ Shankar Venkitachalam ⋅ Sharifa Djurabaeva ⋅ Sharon Ibejih ⋅ Shivanya S Dutta ⋅ Siddhant Gupta ⋅ Silvia P Suárez ⋅ Sina Ahmadi ⋅ Sivasuthan Sukumar ⋅ Siyuan Song ⋅ Sokratis Sofianopoulos ⋅ Sona E Simon ⋅ Sonja Benčina ⋅ Sophie Gvasalia ⋅ Sphurti K More ⋅ Spyridon Konstantinos Dragazis ⋅ Stefan Milosavljević ⋅ Stephan Kaufhold ⋅ AlRashed ⋅ Surangika Ranathunga ⋅ Taiga Someya ⋅ Taja Kuzman Pungeršek ⋅ Tal Haklay ⋅ Tasi'u Jibril ⋅ Tatsuya Aoyama ⋅ Tea Abashidze ⋅ Terenz Jomar Dela Cruz ⋅ Terra Blevins ⋅ Themistoklis Nikas ⋅ Theresa D Idoko ⋅ Tilek Chubakov ⋅ Tina Munda ⋅ Tobiloba Moses Owoeye ⋅ Tommaso Gargiani ⋅ Uni Johannesen ⋅ Vallerie A Putra ⋅ Vanya BK ⋅ Varvara Arzt ⋅ Vasily Konovalov ⋅ Vasudevan Nedumpozhimana ⋅ Viktória Ondrejová ⋅ Viktoryia Horbik ⋅ Vuk Dinić ⋅ Walelign Sewunetie ⋅ Winston Wu ⋅ Xiaojing Zhao ⋅ Yacouba Diarra ⋅ Yaniv Nikankin ⋅ Yash Mathur ⋅ Yash Bagla ⋅ Yeshil Bangera ⋅ Yixi Chen ⋅ Yiyuan Li ⋅ Yolanda Xavier ⋅ Yonatan Belinkov ⋅ Zaid Alyafeai ⋅ Zhengyang Shan ⋅ Zhi Rui Tam ⋅ Zilu Tang ⋅ Zuzana Nadova ⋅ Baber Abbasi ⋅ Stella Biderman ⋅ David Stap ⋅ Duygu Ataman ⋅ Fabian D Schmidt ⋅ Hila Gonen ⋅ Jiayi Wang ⋅ David Adelani
To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and cultures. In this paper, we present Global PIQA, a participatory commonsense reasoning benchmark for over 100 languages, constructed by hand by over 350 researchers from over 65 countries around the world. The 141 language varieties in Global PIQA cover five continents, 19 language families, and 24 writing systems. In the non-parallel split of Global PIQA, over 50% of examples reference local foods, customs, traditions, or other culturally-specific elements. In the parallel split, we translate more "culturally agnostic" commonsense reasoning questions into 131 language varieties, for direct cross-lingual comparisons. In both splits, all examples have been verified by native speakers of the languages. We find that state-of-the-art LLMs perform well on Global PIQA in aggregate, but they exhibit weaker performance in lower-resource languages (e.g. up to a 68% accuracy gap between languages in the parallel split). Global PIQA highlights that in many languages and cultures, everyday knowledge remains an area for improvement in LLMs, alongside more widely-discussed capabilities such as complex reasoning and expert knowledge. Beyond its uses for LLM evaluation, Global PIQA provides a glimpse into the wide diversity of cultures in which human language is embedded.
Show more
Long-Context Generation Is a Sampling Problem
Allen Roush ⋅ Minh N Nguyen ⋅ Ravid Shwartz-Ziv ⋅ Judah Goldfeder ⋅ Sanjay Basu
Long-form generation in language models often collapses into repetitive loops well before the nominal context limit. This is commonly attributed to model capacity or training data, and addressed through costly retraining or post-training pipelines. We show that the sampling algorithm is a primary determinant of long-context output quality, improving outputs without further training costs. Across ten open-weight models from 1B to 120B parameters, output lengths from 8K to 64K tokens, and more than 13,500 generations, full-distribution-aware samplers (P-less, Top-H, top-$n\sigma$) dominate automated metrics and human evaluation, while standard truncation (top-$p$, top-$k$) collapses into late-context repetition; min-$p$ falls in a middle tier. A 1,500-annotation Amazon Mechanical Turk study finds P-less winning more than nine in ten head-to-head pairings against a top-$p$ 0.9 baseline on a 120B model, with 10-gram repetition approaching zero at 64K tokens. A per-step selected-token rank diagnostic isolates the mechanism: standard truncation drives the rank to the argmax floor by the final quartile, while full-distribution-aware methods stay in a healthy band across the full window. This gap that widens with model scale, establishing sampler choice as an under-appreciated bottleneck for long-context generation in open-weight models. We frame long-context generation as sampling from outside the training distribution, and encourage further study for usecases like long-context agentic coding.
Show more
TROPT: An Open Framework for Unifying and Advancing Discrete Text Optimization
Matan Ben-Tov ⋅ Mahmood Sharif
*Discrete text-trigger optimization*—searching for text sequences that, when ingested by a model, steer it toward a specified objective—underpins model red-teaming (e.g., LLM jailbreaks), as well as auditing and interpretability. However, the current state of discrete optimizers hinders their adoption and progress. First, existing optimizers, when open-sourced at all, are scattered across research codebases tied to specific problem domains. Second, optimizer variants proliferate, each requiring engineering overhead to use or extend, and hard to compare head-to-head. Together, these raise the bar to *adopting* optimizers in existing or new domains, and to *advancing* them via new strategies. We address these gaps with TROPT, the first open-source framework that unifies discrete optimizers' execution and standardizes their development under a single interface. TROPT currently ships with 30+ end-to-end optimization recipes—covering applications such as jailbreaking and probing model internals—built from 15+ optimizers (spanning white-box to black-box access) and 15+ losses, from foundational to state-of-the-art methods. Beyond democratizing established methods, TROPT makes it easy to *customize* optimization recipes by swapping any component—models, objectives, and optimizers—extending its reach across domains and new applications. Demonstrating its utility, we leverage TROPT in several controlled studies: *(i)* novel large-scale experiments comparing and enhancing optimization strategies for LLM jailbreaks, revealing potent-yet-underadopted techniques; and *(ii)* cross-domain demonstrations (e.g., benign prompt recovery and corpus poisoning). In all, TROPT significantly lowers the barrier to adopting and advancing discrete text optimization.
Show more
AccelEval: A Whole-Program Benchmark for LLM-Generated CPU-to-GPU Code Acceleration
Chendong Song ⋅ Tianci Bu ⋅ Meixuan Wang ⋅ Zijie Zhou
Current LLM-for-GPU benchmarks largely measure CUDA fluency on isolated operators or short kernel-generation tasks, whereas practical GPU performance engineering often starts from trusted sequential CPU programs and requires correct, end-to-end acceleration. To fill this gap, we introduce \textsc{AccelEval}, a benchmark for whole-program CPU-to-CUDA acceleration. AccelEval contains 42 tasks derived from public repositories and established HPC/application suites across six domains, each equipped with deterministic correctness oracles and three input scales. Beyond pass/fail correctness, AccelEval reports end-to-end speedup over CPU baselines and a novel peak-attainment metric which normalizes each model’s speedup against the best correct solution. Using this framework, we evaluate eight frontier LLMs across multiple input scales, GPU architectures and sample rates. At pass@1 on NVIDIA H200 at medium scale, the strongest model passes 38/42 tasks at 71.2$\times$ geometric-mean speedup over the CPU baseline. On the subset of 11 tasks with human-written CUDA references, the best-of-8 LLM oracle reaches 0.75$\times$ of human CUDA performance. To make CUDA-optimization behavior interpretable and reusable, we decompose correct solutions into 43 CUDA acceleration patterns and show that feeding winner-derived pattern guidance back to models improves paired geometric-mean speedup by 1.78$\times$, compared with 1.18$\times$ for a generic-tips control. Together, these results establish AccelEval as a reproducible diagnostic platform for end-to-end LLM-assisted GPU acceleration. An anonymized open-source release is available at \url{https://anonymous.4open.science/r/AccelEval-2397}.
Show more
MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents
Haonan Li ⋅ Tianjun Sun ⋅ Yongqing Wang ⋅ Qisheng Zhang
Multi-server MCP agents create an information-flow control problem: faithful tool composition can turn individually benign read/write permissions into cross-boundary credential propagation---a structural side effect of workflow topology, not necessarily malicious model behavior. We present MCPHunt, to our knowledge the first controlled benchmark that isolates non-adversarial, verbatim credential propagation across multi-server MCP trust boundaries, with three methodological contributions: (1) canary-based taint tracking that reduces propagation detection to objective string matching; (2) an environment-controlled coverage design with risky, benign, and hard-negative conditions that validates pipeline soundness and controls for credential-format confounds; (3) completion-requires-secret (CRS) stratification that disentangles task-mandated propagation (faithful execution of verbatim-transfer instructions) from policy-violating propagation (credentials included despite the option to redact). Across 3,615 main-benchmark traces from 5 models spanning 147 tasks and 9 mechanism families, policy-violating propagation rates reach 11.5--41.3% across all models. This propagation is pathway-specific (25x cross-mechanism range) and concentrated in browser-mediated data flows; hard-negative controls provide evidence that production-format credentials are not necessary---prompt-directed cross-boundary data flow is sufficient. A prompt-mitigation study across 3 models reduces policy-violating propagation by up to 97% while preserving 80.5% utility, but effectiveness varies with instruction-following capability---suggesting that prompt-level defenses alone may not suffice. Code, traces, and labeling pipeline are released under MIT and CC BY 4.0.
Show more
GUITAR: Structured Failure Diagnosis of GUI Agents via State Transitions
Shaoqing Zhang ⋅ Kehai Chen ⋅ Xuefeng Bai ⋅ Zhuosheng Zhang ⋅ Pengfei Zhang ⋅ Yang Xiang ⋅ Min zhang
Understanding where and why Graphical User Interface (GUI) agents fail is essential for building more reliable systems, yet current evaluation relies on step accuracy, a metric that treats each screen independently and overlooks the underlying structure of GUI environments. This leads to two critical blind spots: (1) functionally equivalent screens are evaluated in isolation, obscuring systematic failure patterns across shared screens; and (2) the long-tailed GUI distribution renders failures on rare but critical screens invisible under standard metrics. To address these issues, we propose \textbf{GUITAR}, a state-centric diagnostic framework that performs structured failure analysis over both states and transitions, using a State Transition Graph (STG) by mapping visually diverse screens to shared functional states. Evaluated with 8 agents across 3 tasks on AndroidControl and 3 tasks on Mind2Web, GUITAR reveals that failures are both structurally concentrated and metrically invisible: 61.9\% of failure instances occur in 20\% states, consistently localizing to a small set of bottleneck states. Furthermore, leveraging bottleneck states for inference guidance yields +2.8\% performance improvement. These findings highlight structure-aware evaluation as necessary for GUI agent diagnosis, and demonstrate that targeted intervention on bottleneck states offers a practical path to improving reliability.
Show more
Automotive-ENV: Benchmarking Multimodal Models in Automotive Cockpit Environments
Junfeng Yan ⋅ Biao Wu ⋅ Meng Fang ⋅ Ling Chen
Multimodal models have demonstrated strong capabilities in web, desktop, and mobile GUI environments, but their performance in automotive cockpit systems remains largely unexplored. In-vehicle GUIs introduce unique challenges beyond traditional GUI interaction, including vehicle-state reasoning, implicit intent understanding, and safety-critical decision making. To address this gap, we introduce Automotive-ENV, an interactive benchmark for evaluating multimodal models in automotive cockpit environments. Built on Android Automotive OS, Automotive-ENV combines cockpit-screen interaction with vehicle-state signals and deterministic state-based evaluation. The benchmark contains 420 carefully constructed tasks covering routine cockpit operation and safety-critical scenarios. Unlike prior GUI benchmarks that rely on action matching or LLM-based judging, Automotive-ENV evaluates model behavior through reproducible validators over UI states, vehicle signals, and safety-related conditions. We benchmark representative general-purpose multimodal models and GUI-specialized vision-language models on Automotive-ENV. Experimental results show that current models remain far from reliable in automotive environments, with substantial performance degradation in implicit and safety-critical tasks. Further analysis reveals consistent failure modes in safety-aware reasoning, vehicle-control understanding, and implicit hazard recognition. We will release Automotive-ENV, together with tasks, validators, and evaluation tools, to support future research on automotive multimodal systems.
Show more
WAXAL: A Large-Scale Multilingual African Language Speech Corpus
MohamedElfatih MohamedKhair ⋅ Emmanuel Asiedu Brempong ⋅ Subhashini Venugopalan ⋅ Abdoulaye Diack ⋅ Perry Nelson ⋅ Prince Mireku ⋅ Tavonga Siyavora ⋅ Pooja Rao ⋅ Uche Okonkwo ⋅ Angela Nakalembe ⋅ Abhishek Bapna ⋅ Aisha Walcott-Bryant
The advancement of speech technology has predominantly favored high-resource languages, creating a significant digital divide for speakers of most Sub-Saharan African languages. To address this gap, we introduce WAXAL, a large-scale, openly accessible speech dataset for 24 languages (spanning 19 for ASR and 13 for TTS, with overlaps) representing over 100 million speakers. The collection consists of an Automatic Speech Recognition (ASR) dataset containing approximately 2,340 hours of transcribed, image-prompted natural speech (over 14,100 hours of total collected audio) from a diverse range of speakers, and a Text-to-Speech (TTS) dataset with over 235 hours of high-quality, single-speaker studio recordings of phonetically balanced scripts. Crucially, this effort was executed in partnership with four African academic and community organizations, with the explicit goal of building the local ecosystem for speech technology and seeding data collection capabilities within these institutions. To demonstrate the dataset's utility, we benchmark three architecturally distinct ASR models---Gemma 3n-2B, Whisper-Large-v3, and MMS-1b-all---across the 19 ASR languages. Fine-tuning on WAXAL yields substantial improvements, reducing the macro-average Word Error Rate (WER) by up to 61\% (e.g., from 1.23 to 0.48 for Whisper). Models fine-tuned solely on WAXAL also generalize to the independently collected FLEURS benchmark, confirming the dataset's value for domain-transferable ASR adaptation. Furthermore, we employ an LLM-as-judge framework to evaluate semantic meaning preservation, demonstrating that fine-tuned predictions successfully capture core intents of native speakers. The WAXAL datasets are released at https://huggingface.co/datasets/google/WaxalNLP under the CC-BY-4.0 license to catalyze research and the development of inclusive speech technologies for speakers of African languages.
Show more
HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure
Huy Nghiem ⋅ Maria Kostylew ⋅ Noam Kolt
The deployment of large language models (LLMs) in hiring pipelines is growing, yet existing evaluations generally overlook an important aspect of this development: the compliance of LLMs with applicable legal rules. To fill this gap, we introduce HireWatch, a benchmark that assesses whether LLM hiring assistants comply with U.S. employment discrimination law (e.g., the Civil Rights Act of 1964). We construct a 200-item bank consisting of interview questions that are either lawful or unlawful under U.S. federal statute, regulation, and enforcement guidance. We evaluate the propensity of 18 proprietary and open-source LLMs to select lawful vs. unlawful questions from this bank across ~100,000 trials under a novel seven-tier contextual escalation ladder, crossing occupational details with organizational pressure. Our findings reveal that LLMs' compliance with employment discrimination law is brittle. While LLMs ordinarily comply with legal rules under different job contexts, they readily violate the law when subject to institutional pressure (e.g., an unlawful manager instruction or company policy). Meanwhile, although the inclusion of explicit legal reminders in prompts can improve legal compliance, it does not altogether eliminate violations of employment discrimination law.
Show more
SciResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Lei Xiong ⋅ Kun Luo ⋅ Ziyi Xia ⋅ Wenbo Zhang ⋅ Jingying Shao ⋅ Jianlyu Chen ⋅ Hongjin Qian ⋅ Zhicheng Dou ⋅ Zheng Liu
Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents’ capability in driving this process, we present **SciResearchBench** (Scientific - Research - Bench), a dedicated benchmark for autonomous scientific literature discovery. SciResearchBench consists of two complementary task types: (1) **Deep Research**, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) **Wide Research**, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, SciResearchBench is distinguished along three dimensions: it is **research-oriented**, calling for in-depth comprehension of scientific concepts; **literature-focused**, demanding fine-grained utilization of detailed information; and **open-ended**, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make SciResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%.
Show more
Tactile MNIST: Benchmarking Active Tactile Perception
Tim Schneider ⋅ Guillaume Duret ⋅ Cristiana de Farias ⋅ Roberto Calandra ⋅ Liming Chen ⋅ Jan Peters
Tactile perception has the potential to significantly enhance dexterous robotic ma- nipulation by providing rich local information that can complement or substitute for other sensory modalities such as vision. However, tactile sensing is an inher- ently local sensor modality, providing information only at the points of contact. This strict locality necessitates the use of active perception techniques to acquire useful, task-relevant information from manipulated objects or the environment in general. Hence, the agent must actively guide sensors toward regions with more informative or significant features and integrate such information over time in order to understand a scene or complete a task. However, while active perception and tactile sensing methods have received significant attention recently, both fields lack standardized benchmarks. To bridge this gap, we introduce the Tactile MNIST Benchmark Suite, an open-source, Gymnasium-compatible benchmark specifically designed for active tactile perception tasks, including localization, classification, and volume estimation. Our benchmark suite offers diverse simulation scenarios, from simple toy environments all the way to complex tactile perception tasks using vision-based tactile sensors. Furthermore, we also offer a comprehensive dataset comprising 13,500 synthetic 3D MNIST digit models and 153,600 real-world tactile samples collected from 600 3D printed digits. Using this dataset, we train a CycleGAN for realistic tactile simulation rendering. By providing standardized protocols and reproducible evaluation frameworks, our benchmark suite facilitates systematic progress in the fields of tactile sensing and active perception in general.
Show more
MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision
Ziyang Zhang ⋅ Shenyi Li ⋅ Yilin wang ⋅ Ziyun Cui ⋅ Bowen Zhou ⋅ Wen Wu ⋅ Chao Zhang
Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce **MMTA** (**m**ulti**m**odal **t**emporal **a**nalysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains — robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate — under *video+TS* and *image+TS* layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce **TimeOmni-v**, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.
Show more
GUIGuard-Bench: Toward a General Evaluation for Privacy-Preserving GUI Agents
Yanxi Wang ⋅ Zhiling Zhang ⋅ Wenbo Zhou ⋅ Weiming Zhang ⋅ Jie Zhang ⋅ Qiannan Zhu ⋅ Yu Shi ⋅ Shuxin Zheng ⋅ Jiyan He
As GUI agents increasingly rely on screenshots to perceive and operate digital environments, they may inadvertently expose sensitive information such as identities, accounts, locations, and behavioral traces. While existing benchmarks primarily focus on task completion, grounding, or defenses against third-party attacks, current visual privacy datasets remain largely restricted to static natural images, limiting their ability to capture the contextual dependence and task relevance of privacy risks in GUI task trajectories. To bridge this gap, we introduce GUIGuard-Bench, a first-step benchmark for studying privacy-preserving GUI agents in trajectory-based GUI workflows. GUIGuard-Bench contains 241 real GUI-agent trajectories with 4,080 screenshots across Android and PC environments. Each screenshot is annotated at the region level with privacy bounding boxes, semantic privacy categories, risk levels, and whether the private information is necessary for completing the task. Built on these annotations, GUIGuard-Bench supports three complementary evaluations: privacy recognition, offline planning fidelity under protected screenshots, and the utility impact of different protection strategies. Our results show that current models can often detect whether a screenshot contains private information, but they struggle with fine-grained localization, category recognition, risk assessment, and task-necessity judgment. We also find that closed-source models, exemplified by Claude Sonnet 4.6, can maintain largely consistent planner semantics in Android environments after privacy protection is applied. GUIGuard-Bench serves as a seed benchmark for studying privacy awareness, data minimization, and privacy-utility trade-offs in GUI agents, and is intended to support broader future evaluation efforts.
Show more
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
Xueyu Hu ⋅ Jinxiang Xia ⋅ Shengze Xu ⋅ Kangqi Song ⋅ Yishuo Yuan ⋅ Guibin Zhang ⋅ JinCheng Ren ⋅ Boyu Feng ⋅ Li Lu ⋅ Tieyong Zeng ⋅ Jiaheng Liu ⋅ Minghao Liu ⋅ He Zhu ⋅ Eleanor Jiang ⋅ Wei Wang ⋅ Wangchunshu Zhou
Long-horizon planning is widely recognized as a core capability of autonomous LLM-based agents; however, current evaluation frameworks suffer from being largely episodic, domain-specific, or insufficiently grounded in persistent economic dynamics. We introduce EcoGym, a generalizable benchmark for continuous plan-and-execute decision making in interactive economies. EcoGym comprises three diverse environments: Vending (adapted from the closed-source Vending-Bench, with full open-source release), Freelance (new), and Operation (new), implemented in a unified decision-making process with standardized interfaces, and budgeted actions over an effectively unbounded horizon (1000+ steps if 365 day-loops for evaluation). The evaluation of EcoGym is based on business-relevant outcomes (e.g., net worth, income, and DAU), targeting long-term strategic coherence and robustness under partial observability and stochasticity. Experiments across eleven leading LLMs expose a systematic tension: no single model dominates across all three scenarios. Critically, we find that models exhibit significant suboptimality in either high-level strategies or efficient actions executions. EcoGym is released as an open, extensible testbed for transparent long-horizon agent evaluation and for studying controllability–utility trade-offs in economic settings.
Show more
GEMS-3D: A Large-Scale 3D Gravity, Electrical, Magnetic, and Seismic Earth Simulation Dataset for Multimodal Geophysical Learning
Yonghao Wang ⋅ Meijia Huang ⋅ Wenkai Lu ⋅ Zhuo Jia ⋅ Hao Feng ⋅ Leyuan Fang
Accurate characterization of deep subsurface structures is a fundamental problem in geoscience, but geophysical inversion is inherently non-unique and therefore requires joint interpretation of gravity, magnetic, electromagnetic, and seismic observations. Despite recent advances in deep learning for geophysical inversion and PDE-based scientific modeling, existing large-scale open datasets are largely limited to a single modality and rarely provide physically consistent multi-source responses generated from the same 3D geological target. This limitation hinders the development and evaluation of multimodal joint inversion methods for deep subsurface characterization. We introduce GEMS-3D, a large-scale synthetic benchmark for 3D geological multiphysics learning. GEMS-3D couples four forward modeling engines—$\textbf{g}$ravity, $\textbf{e}$lectromagnetic, $\textbf{m}$agnetic, and 3D acoustic $\textbf{s}$eismic—to generate sample-aligned observations from shared geological realizations. Guided by facies-aware rock-physics relationships, the dataset provides spatially aligned volumes of P-wave velocity ($v_p$), density ($\rho$), resistivity ($\phi$), and magnetic susceptibility ($\chi$) across nine representative classes of complex geological anomalies, including dike swarms, salt domes, and gas reservoirs. Each sample includes gravity responses, magnetic responses, electromagnetic responses, and 3D acoustic seismic data, together with acquisition metadata and anomaly labels. By packaging co-registered property volumes, anomaly support, acquisition metadata, and aligned multiphysics responses as sample-level bundles, GEMS-3D provides a unified benchmark for operator learning, multimodal inverse interpretation, and missing-modality robustness in deep subsurface settings. The anonymized forward-modeling framework and corresponding dataset link is available at https://anonymous.4open.science/r/GEMS-3D-E4CA.
Show more
PINNeval: A Comprehensive Evaluation Standard for Physics-Informed Neural Networks
Kevin von Bargen ⋅ Hamid Ebrahimy ⋅ Tim Römer
The Physics-Informed Neural Network (PINN) literature has produced a rapidly expanding catalogue of methods. We divide a PINN training pipeline systematically into four components and observe that many existing benchmarks study these components in isolation. These components are domain-importance adaptation, loss balancing, the neural network architecture, and the optimization scheme. Moreover, the studies set the surrounding and not considered components of the pipeline to a minimal default that is no longer representative of current best practice approaches. Conclusions drawn under such conditions lead to suboptimal results. In this work, we propose a new evaluation protocol that (1) treats the four components as an integrated system; (2) requires as a key aspect a competitive implementation of the components not currently under investigation; (3) specifies the metrics and reporting format, and (4) is accompanied by structured documentation that ensures the traceability of evaluations across studies. Based on this comprehensive protocol, we release with PINNeval an innovative JAX package containing 18 different methods from the current literature as well as nine efficiently implemented benchmark problems. Using PINNeval, we illustrate our evaluation protocol in relation to established benchmark studies and report on results that challenge established assumptions.
Show more
AppCIF-Bench: An Application-Level Complex Instruction Following Benchmark for Large Language Models
Jiahao Xu ⋅ Xuefang Zhao ⋅ Xinhua Feng ⋅ Hongrui Yang ⋅ Yixian Liu ⋅ Zhichao Hu ⋅ Lliu Yuhong
Complex Instruction Following (CIF) is a foundational capability for deploying Large Language Models (LLMs) in production-grade applications. It demands stable, precise outputs under increasing horizon length, deeply nested rules, and multi-turn interactions. Yet existing benchmarks are predominantly atomic and short-horizon, failing to capture the logical depth and distractor pressure of real-world settings. We introduce **AppCIF**, the first application-level CIF benchmark. AppCIF stratifies difficulty along two orthogonal axes: *Horizon Length* ($H$), the span over which distractors accumulate and critical signals must be tracked; and *Logical Depth* ($L$), the nesting depth of rules to be unfolded. Together, $\langle H, L\rangle$ defines four hierarchical complexity levels, scaling from clean few-step execution (L1) to extreme pressure deduction (L4). AppCIF comprises 466 curated samples across five application scenarios. It adopts a "**Prompt-as-Environment**" paradigm, pairing Complex System Prompts (averaging ~14k words) with executable Code Simulation Engines (averaging ~3.5k lines of Python) to sustain rigorous state-machine deduction. The benchmark is constructed via a **Labor-Divided Human-AI pipeline** with scenario-specific collaboration strategies, and evaluated on 12 representative LLMs. Results reveal sharp performance stratification: Hard Satisfaction Rate ranges from 1.1% to 45.9%, exposing severe bottlenecks in long-horizon, distractor-laden instruction execution.
Show more
Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay
Mingjia Huo ⋅ Yao Fu ⋅ Bo Chang ⋅ Yaqing Wang ⋅ Dawei Zhu ⋅ Dylan Zhang ⋅ Xinyang Yi ⋅ Lichan Hong ⋅ Ed Chi ⋅ Hao Zhang ⋅ Jiaxi Tang
Vision-Language Models (VLMs) have achieved strong performance on static visual understanding benchmarks, yet their ability to act in dynamic environments remains poorly understood. We introduce AtariBench, a controlled benchmark that evaluates VLMs across 20 Atari 2600 games, requiring models to make sequential decisions from high-temporal-resolution gameplay streams. By decoupling fast in-game dynamics from wall-clock inference speed, AtariBench focuses on whether models can perceive rapid visual state changes, reason over spatiotemporal dynamics, and ground actions in the current observation. Our experiments reveal substantial limitations in state-of-the-art VLMs: models often fail to infer object motion and direction, exhibit brittle sequential decision-making, and increasingly over-rely on textual interaction history rather than newly appended visual feedback. We further introduce a separability analysis to identify which in-game scores provide reliable signals of model competence, and use the benchmark to study how memory design and prompting strategies affect performance. Overall, AtariBench exposes a persistent gap between passive visual perception and active decision-making, providing a lightweight and rigorous testbed for developing VLMs that can reason and act in dynamic environments.
Show more
RSRCC: A Remote Sensing Regional Change Comprehension Benchmark Constructed via Retrieval-Augmented Best-of-𝑁 Ranking
Roie Kazoom ⋅ Yotam Gigi ⋅ George Leifman ⋅ Tomer Shekel ⋅ Genady Beryozkin
Traditional change detection identifies where changes occur, but does not explain what changed in natural language. Existing remote sensing change captioning datasets typically describe overall image-level differences, leaving fine-grained localized semantic reasoning largely unexplored. To close this gap, we present RSRCC, a new benchmark for remote sensing change question-answering containing 126k questions, split into 87𝑘 training, 17.1𝑘 validation, and 22𝑘 test instances. Unlike prior datasets, RSRCC is built around localized, change-specific questions that require reasoning about a particular semantic change. To the best of our knowledge, this is the first remote sensing change question-answering benchmark designed explicitly for such fine-grained reasoning-based supervision. To construct RSRCC, we introduce a hierarchical semi-supervised curation pipeline that uses Best- of-𝑁 ranking as a critical final ambiguity-resolution stage. First, candidate change regions are extracted from semantic segmentation masks, then initially screened using an image-text embedding model, and finally validated through retrieval-augmented vision-language curation with Best-of-𝑁 ranking. This process enables scalable filtering of noisy and ambiguous candidates while preserving semantically meaningful changes. The dataset is available at https://huggingface.co/datasets/google/RSRCC.
Show more
SynBench: A Benchmark for Differentially Private Text Generation
Yidan Sun ⋅ Viktor Schlegel ⋅ Srinivasan Nandakumar ⋅ Iqra Zahid ⋅ Yuping Wu ⋅ Yulong Wu ⋅ Hao Li ⋅ Jie Zhang ⋅ Warren Del-Pinto ⋅ Goran Nenadic ⋅ Siew Kei Lam ⋅ Anil A Bharath
Synthetic text generation with Differential Privacy (DP) guarantees emerges as a principled approach that can enable the sharing of sensitive datasets across institutional and regulatory boundaries, while bounding the risks of re-identification and membership inference. LLM-based methods deliver promising results; however, comparisons are exacerbated by differing evaluation setups and "private" datasets, potential pre-training contamination is not considered and guarantees are not verified with DP audits. To advance this field, we introduce a unified evaluation framework with standardised utility and fidelity metrics and privacy audits, encompassing nine curated datasets that capture domain-specific complexities such as technical jargon, long-context dependencies, and specialised document structures. In a large-scale empirical study, we benchmark LLM-based state-of-the-art DP text generators of varying sizes (between 1--8B). Our results indicate that DP synthetic text generation remains an unsolved challenge, with quality deteriorating more as the private datasets deviate further from the generators' pre-training corpora. Our novel synthetic text membership inference attack (MIA) explains this observation: Synthetic data quality is overestimated when LLMs have been pre-trained---without DP---on portions of the "private" data to be generated. Finally, our work provides the first quantitative evidence that this "public pre-training and private generation" paradigm invalidates the guaranteed privacy bounds of real-world private datasets.
Show more
BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
Leonhard Waibl ⋅ Felix Michalak ⋅ Hadrien Mariaccia
LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails against repurposed generalist LLMs. We present BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems -- Operational), the first independent operational benchmark of LLM supervision systems. BELLS-O evaluates 28 systems from 17 providers: every major specialized guardrail (e.g., LlamaGuard-4, ShieldGemma-2, Lakera Guard) and frontier generalists repurposed as supervisors (e.g., GPT-5.4, Claude Sonnet 4.6, Grok-4.1), jointly on detection rate, false-positive rate, latency, and monetary cost. We cover input/output moderation across 11 harm categories and jailbreak detection across 13 attack techniques, using in-house datasets built from handcrafted prompts, expert-curated samples, and quality-controlled synthetic generation. To prevent latent generator-specific signals in synthetic data, every generated sample is run through a paraphrasing step that suppresses these fingerprints. Mapping the Pareto frontier reveals use-case-dependent tradeoffs. On content moderation, specialized supervisors are operationally dominant: top systems match frontier LLMs on detection ($\approx$95\% vs.\ 94\%) at comparably low false-positive rates ($\leq$2\%), while running 5--10$\times$ faster and ${\sim}$10$\times$ cheaper. On jailbreak detection, the tradeoff shifts: frontier LLMs achieve higher detection and lower false-positive rates but at 10--50$\times$ higher cost and 5--10$\times$ higher latency. We release the benchmark, framework, leaderboard, and datasets as the first vendor-neutral basis for selecting safeguards under real deployment constraints.
Show more
Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Models
Nilaksh ⋅ Saurav Jha ⋅ Artem Zholus ⋅ Sarath Chandar
World model-based policy evaluation is a practical proxy for testing real-world robot control by rolling out candidate actions in action-conditioned video diffusion models. As these models increasingly adopt latent diffusion modeling (LDM), choosing the *right latent space* becomes critical. While the status quo uses autoencoding latent spaces like VAEs that are primarily trained for pixel *reconstruction*, recent work suggests benefits from pretrained encoders with representation-aligned *semantic* latent spaces. We systematically evaluate these latent spaces for action-conditioned LDM by comparing six reconstruction and semantic encoders to train world model variants under a fixed protocol on BridgeV2 dataset, and show effective world model training in high-dimensional representation spaces with and without dimension compression. We then propose three axes to assess robotic world model performance: visual fidelity, planning and downstream policy performance, and latent representation quality. Our results show visual fidelity alone is insufficient for world model selection. While reconstruction encoders like VAE and Cosmos achieve strong pixel-level scores, semantic encoders such as V-JEPA 2.1 (strongest overall on policy), Web-DINO, and SigLIP 2 generally excel across the other two axes at all model scales. Our study advocates semantic latent space as stronger foundation for policy-relevant robotics diffusion world models.
Show more
gfnx: Fast and Scalable Library for Generative Flow Networks in JAX
Daniil Tiapkin ⋅ Artem Agarkov ⋅ Nikita Morozov ⋅ Ian Maksimov ⋅ Askar Tsyganov ⋅ Timofei Gritsaev ⋅ Sergey Samsonov
In this paper, we present gfnx, a fast and scalable package for training and evaluating Generative Flow Networks (GFlowNets) written in JAX. gfnx provides an extensive set of environments and metrics for benchmarking, accompanied with single-file implementations of core objectives for training GFlowNets. We include synthetic hypergrids, multiple sequence generation environments with various editing regimes and particular reward designs for molecular generation, phylogenetic tree construction, Bayesian structure learning, and sampling from the Ising model energy. Across different tasks, gfnx achieves significant wall-clock speedups compared to Pytorch-based benchmarks (such as torchgfn library) and author implementations. For example, gfnx achieves up to 55 times speedup on CPU-based sequence generation environments, and up to 80 times speedup with the GPU-based Bayesian network structure learning setup. Our package provides a diverse set of benchmarks and aims to standardize empirical evaluation and accelerate research and applications of GFlowNets.
Show more
OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling
Adam Bawatneh ⋅ Sagar Sapkota ⋅ Amrit Singh Bedi ⋅ Santu Karmaker ⋅ Mubarak Shah
Theory of Mind (ToM), the ability to infer others’ knowledge, intentions, and emotions, is commonly evaluated in large language models (LLMs) using end-point question answering, where performance is judged solely by the final answer to a social reasoning query. This paradigm obscures whether the model actually constructs the underlying mental-state representations required for robust reasoning, particularly in scenarios involving divergent, evolving, or mistaken beliefs. In order to address this research gap, we introduce OmniToM, a benchmark that directly evaluates these representations by requiring explicit modeling of belief structures for all relevant actors within a narrative. These structures are composed of belief propositions: minimal statements of what an actor takes to be true about the world or another actor's mental state, allowing knowledge, intentions, emotions, and false beliefs to be analyzed in a common format. Models are evaluated in two stages: Stage 1: Belief Extraction, which extracts from the story the beliefs relevant to its social dynamics, and Stage 2: Belief Labeling, which assigns each belief a seven-dimensional schema label covering recursive order, truth status, knowledge access, explicitness, content type, mental source, and context. Built from 895 stories from the existing ToMBench story corpus and augmented with 22,343 labeled belief propositions, OmniToM uses a human-calibrated LLM-assisted annotation pipeline. Across diverse models in zero-shot evaluation, OmniToM reveals an actor-specific belief-tracking bottleneck: current LLMs struggle with the knowledge-access and representational decisions required to transform narrative facts into actors’ beliefs and shared mental states.
Show more
VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding
He Haichen ⋅ Jiayi Zhou ⋅ Sifeng SHANG ⋅ Yihan Hu ⋅ Yuanhan Zhang ⋅ Kaiyang Zhou
Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal spans within extreme video durations. Mastering this intense cognitive load constitutes the fundamental bottleneck in long video understanding. While existing benchmarks have driven progress by scaling up video duration, their evaluation tasks often require comprehending only short and isolated video segments, falling short of capturing the challenge of ultra-long-context reasoning. To measure this cognitive load, we emphasize continuous certificate length, defined as the video length a human must continuously watch to definitively answer a given question. Driven by this metric, we introduce VideoOdyssey, a benchmark specifically designed for ultra-long-context and omni-modal video understanding. VideoOdyssey is characterized by three key features: 1) Extreme video duration and diversity: spanning 11 domains and 54 subcategories with an average video duration of 109 minutes; 2) Comprehensive evaluation scenarios: offering two subsets to address different research focuses, i.e., VideoOdyssey-V for probing the limits of visual understanding in MLLMs, and VideoOdyssey-AV for evaluating synchronized audio-visual understanding for omni-modal models; 3) Ultra-long and multi-level continuous certificates: extending the average continuous certificate to 16 minutes for VideoOdyssey-V and 12.8 minutes for VideoOdyssey-AV. Crucially, we design 5 granular levels from seconds to hours, providing a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads. Extensive evaluations show that bottlenecks of current MLLMs extend beyond simple retrieval to include struggles with continuous reasoning across varying context lengths, fine-grained perception, and non-verbal omni-modal understanding. We hope VideoOdyssey will spur the development of next-generation MLLMs toward genuine real-world video understanding.
Show more
Cracks in the Foundation: A Civil Infrastructure Dataset to Challenge Vision Foundation Models
Nicola Farronato ⋅ Niccolò Avogaro ⋅ Thomas Frick ⋅ Mattia Rigotti ⋅ Rizwan U Khan ⋅ Michele Magno ⋅ Konrad Schindler ⋅ A. Cristiano I. Malossi ⋅ Florian Scheidegger
Automated structural health monitoring is essential to prevent catastrophic infrastructure failures. Precise, pixel-level defect segmentation is needed to accurately assess structural integrity, but progress in defect segmentation for civil infrastructures has been held back by an extreme scarcity of data, which requires costly expert annotation. The need for data is accentuated by algorithmic hurdles intrinsic to the problem, including center-bias and the need to rely more on shape when inspecting nearly textureless building materials. To remove the bottleneck, we introduce **Cracks in the Foundation (CiF)**, the largest and most detailed civil infrastructure (instance) segmentation dataset to date, comprising $\approx$150,000 high-resolution images meticulously curated over five years in collaboration with civil engineering experts. With the help of this unprecedented data source, we expose a blind spot of current visual AI: despite the advent of promptable Foundation Models (FMs) and Vision Language Models (VLMs), and despite the impressive abilities of today's specialised segmentation models, it turns out that dense image understanding in the built environment is nowhere near solved. Our evaluations indicate that even the most recent zero-shot FMs face significant challenges when deployed on real-world infrastructure and even the performance of specialised models with domain-specific supervision plateaus at $\approx$25\% mAP. CiF establishes inspection of civil infrastructure, an elementary and seemingly easy perceptual task, as an open challenge that reveals fundamental weaknesses of present-day models trained predominantly on internet images, literally and figuratively highlighting *cracks in the current foundation model paradigm*.
Show more
K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs
Hao Liang ⋅ Qihan Lin ⋅ Mingrui Chen ⋅ Hengyi Feng ⋅ ZhaoYang Han ⋅ Xiaochen Ma ⋅ Zhen H Wong ⋅ Meiyi Qiang ⋅ Linzhuang Sun ⋅ Wentao Zhang
Large language models (LLMs) are increasingly deployed in K–12 education, yet existing benchmarks such as C-Eval, CMMLU, GaokaoBench, and EduEval measure only whether a model can answer an exam question, i.e., factual recall. Effective educational AI further requires curriculum cognition: the structured understanding of how knowledge is organized, including prerequisite chains, concept taxonomies, experiment–concept links, and pedagogical sequencing. Curriculum cognition is neither probed by current benchmarks nor explicitly taught by current instruction-tuning data. To close this gap, we introduce K12-KGraph, a curriculum-aligned knowledge graph extracted from the official People's Education Press textbooks, covering mathematics, physics, chemistry, and biology across primary, middle, and high school, with seven node types (Concept, Skill, Experiment, Exercise, Section, Chapter, Book) and nine relation types spanning taxonomy, prerequisite, association, verification, assessment, location, and order. Building on this single graph, we derive two complementary resources: K12-Bench, a 23,640-question multi-select benchmark across five graph-derived task families (Ground, Prereq, Neighbor, Evidence, and Locate) that jointly probe curriculum cognition; and K12-Train, a KG-guided supervised fine-tuning corpus of ~2,300 QA pairs synthesized from node attributes and edge semantics. Experiments expose a clear gap and a clear remedy: on K12-Bench, even the strongest proprietary model (Gemini-3-Flash) reaches only 57% exact match and the best open-source model (Gemma-4-31B-IT) only 46%, with Prereq and Neighbor being the hardest; yet under a strictly matched 2,300-sample SFT budget on Qwen3-4B-Base and Llama3.1-8B-Base, K12-Train consistently outperforms equally sized subsets of eight mainstream instruction-tuning corpora (OpenHermes, Infinity, UltraChat, WizardLM, DataFlow, LMSYS, SmolTalk, Tulu-3) on both GaokaoBench and EduEval, showing that structural curriculum grounding is remarkably sample-efficient for educational SFT. We release the graph, benchmark, training data, and the full construction pipeline.
Show more
Verifying Agents in Rubric-Graded Environments
Markus Dücker ⋅ Vaibhav Kumar ⋅ Yi Liu ⋅ Ronak Chaudhary ⋅ Andreas Plesner ⋅ Francisco Guzmán ⋅ Anish Athalye
As AI agents take on increasingly open-ended and complex tasks, verifying their outputs has proven to be difficult. Rubric-graded agent environments—where a verifier judges multiple criteria against the deliverables and environment state changes produced by the agent—have emerged as a popular paradigm. We conduct the first systematic study of verifiers for rubric-graded agent environments. First, we create BankerVerifierBench (BVB), a meta-evaluation dataset of 3,204 human- judged rubric criteria across 21 investment-banking tasks. Next, we develop a methodology that derives from any rubric corpus the capabilities a verifier should possess, yielding a nine-capability taxonomy that we distill into three general design principles—reactive verification, environment alignment, and domain guidance— which we implement in Gandalf, an open-source verifier. Finally, we evaluate Gandalf against existing verifiers on BVB. We find that Gandalf outperforms all baselines on seven of nine capabilities and is Pareto optimal: its cheapest configuration (F1 0.633, \\$42) exceeds the most expensive baseline (F1 0.538, \\$414). Ablations show that environment alignment and domain guidance primarily reduce cost (up to 4× fewer tokens) rather than error, while architecture choice dominates model choice. Gandalf generalizes to an OpenClaw benchmark, a structurally different personal-productivity environment, achieving F1 score 0.951 and outperforming the next-best verifier by 7.5 F1 points.
Show more
SWE-GPU-Bench: Can Language Models Solve Real-World GPU Software Engineering Tasks?
Feng Chen ⋅ binbin liu ⋅ Wenhan Han ⋅ Yin Zheng
Existing benchmarks for LLM-based GPU programming have made substantial progress in evaluating whether models can generate correct and efficient kernels, operators, or standalone CUDA programs. However, real GPU software engineering often requires modifying existing repositories rather than writing isolated computational units: changes may span CUDA kernels, C++ host logic, Python bindings, build configurations, tests, and performance evaluation scripts. We introduce SWE-GPU-Bench, a benchmark for repository-level GPU software engineering. To the best of our knowledge, SWE-GPU-Bench is the first benchmark that jointly evaluates PR-derived GPU bug fixes, feature implementations, and performance optimizations across multiple real-world repositories and programming languages, while making setup commands, correctness tests, and optional performance evaluation commands explicit components of each instance. SWE-GPU-Bench contains 608 instances from 23 GPU-related repositories and exposes strong cross-file, cross-language, and long-context demands. Through a comprehensive evaluation of representative methods spanning both file-level and repo-level approaches, we show that current LLMs remain far from effectively solving repository-level GPU software engineering tasks.
Show more
PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction
Zhiwen Yang ⋅ Pan Liu ⋅ yifan Li ⋅ Yunhua Zhong ⋅ Jun Xia
Tandem mass spectrometry provides a high-throughput framework for identifying and quantifying proteins in complex biological samples. In computational proteomics, predicting peptide MS/MS spectra is a critical task, enabling downstream applications such as large-scale peptide identification and quantification. While deep learning architectures have substantially improved prediction accuracy, three evaluation challenges obscure the true progress of the field. First, inconsistent data preprocessing and incompatible model output spaces hinder fair model comparison. Second, flawed data splitting strategies can permit hidden sequence leakage and inflate reported performance. Third, existing evaluations typically lack comprehensive cross-species benchmarking and systematic assessment of model robustness to influential experimental conditions. To address these challenges, we propose PepSpecBench, a unified benchmark for peptide MS/MS spectrum prediction. PepSpecBench standardizes data preprocessing across complementary public datasets, enforces a strict backbone-disjoint splitting strategy to eliminate sequence leakage, and evaluates diverse architectures within a shared fragment-ion representation space. It further introduces a comprehensive multi-species evaluation suite and physically grounded metadata perturbation probes to assess model robustness and instrument awareness. We uncover previously unrecognized performance discrepancies and robustness limitations across six representative models, providing actionable insights for future model design, evaluation and practical deployment. Code and data are available at https://anonymous.4open.science/r/PepSpecBench-NeurIPS2026/.
Show more
Large Language Model Selection with Limited Annotations
Yavuz Durmazkeser ⋅ Patrik Okanovic ⋅ Andreas Kirsch ⋅ Torsten Hoefler ⋅ Nezihe Merve Gürel
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop Select-LLM, the first framework for active model selection of LLMs. Select-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, Select-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate Select-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, Select-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Show more
VeriScope: Measuring Verification-Ready Verilog Artifacts
Wei Zhang ⋅ Jian Yang ⋅ jiajun wu ⋅ Junhang Cheng ⋅ Chufan He ⋅ Yihang Lou ⋅ Xianglong Liu
Large language models (LLMs) are increasingly used in hardware-oriented coding workflows, yet open resources for evaluating first-pass RTL beyond execution remain limited. We present VeriScope, an open benchmark and evaluation suite of 568 problems spanning basic combinational gates through module-level designs and a 95-task L4+ top-end slice (92 L4 system tasks plus 3 L5 stress tests). A developer agent receives a natural-language task brief, returns a single RTL draft, and that artifact is evaluated through objective execution plus RTL-level and artifact-level review incorporating waveform and circuit evidence. Across 26 public models plus two 32B calibration baselines, the best systems reach about 84/100 combined score on the full benchmark but drop to the high 70s on L3–L4+. The 60/40 combined leaderboard largely tracks functional execution (Spearman 0.998 with a functional-only ranking), while artifact scores are most useful as a post-simulation screening signal: 29.6% of L3–L4+ simulation-passing submissions receive artifact score below 6 and therefore need additional verification evidence. A 100-sample manual taxonomy of such disagreements finds 70% testbench coverage gaps, 12% real RTL defects, and 18% judge hallucinations (Wilson 95% intervals reported in the analysis), showing that artifact review surfaces both under-tested passes and model defects, but should not be treated as a standalone leaderboard refinement.
Show more
SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data
Valerii Iakovlev ⋅ Markus Spanring ⋅ Martin Flechl
As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizing large log streams into human-readable reports. Further progress requires meaningfully evaluating this function, yet existing benchmarks leave end-to-end summarization from large log streams underexplored due to the prohibitive cost and complexity of collecting real-world data. We introduce SLS-Bench, a benchmark for incident log summarization featuring 90 tasks grounded in real-world incident reports, each with a synthetic log stream and a reference summary. To construct SLS-Bench, we developed a data generation pipeline that converts incident reports into structured incident models, which guide the generation of synthetic logs and reference summaries. This pipeline combines narrative grounding, explicit incident modeling, generator-verifier loops, and statistical comparisons with real-world log streams to produce realistic and controllable observability data. Our evaluation of 14 LLMs on SLS-Bench shows that while frontier models can produce useful summaries, they do so at substantial cost and latency, and open-weight models lag significantly. SLS-Bench provides a framework for systematic evaluation of log analysis agents and a scalable blueprint for future synthetic observability benchmarks.
Show more
Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation
Andreas Haupt ⋅ Justin Hartenstein ⋅ Anka Reuel-Lamparth ⋅ Mykel J Kochenderfer ⋅ Sanmi Koyejo
AI benchmarks have well-documented limitations, with prior work examining contamination, saturation, and construct underspecification. Aggregation has received far less attention: benchmarks are typically summarized by uniformly averaging item-level scores, implicitly treating every test item as equally valuable. We model benchmarking as a multitask principal–agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives: alignment with normative welfare priorities, marginal improvability, and performance variance. We translate the theory into an audit framework that ranks items along each of these three axes, and apply it to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance. The framework surfaces items that are Pareto-inferior within OLMES subject to a pro-worker welfare operationalization.
Show more
Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth
Yoav Gur-Arieh ⋅ Ana Marasovic ⋅ Mor Geva
Chains of thought (CoTs) have become central in interpreting and auditing behaviors of large language models. Yet growing evidence suggests that these traces often fail to faithfully represent the computations behind a model's predictions. Several faithfulness metrics have been proposed, but whether they indeed measure faithfulness remains unknown. Answering this requires ground-truth labels, which are hard to obtain since internal computations are not directly observable. Consequently, most works proposing metrics report only absolute scores or comparisons to prior metrics, and the few existing benchmarks rely on proxies like plausibility or importance, properties orthogonal to faithfulness that can mislead about whether a CoT can be trusted. We address this challenge by constructing tasks whose outputs reveal which intermediate computations must have produced them, and developing an automated labeling pipeline that yields ground-truth faithfulness labels at both the step and CoT level. Building on this methodology, we present **BonaFide**, a benchmark of 3,066 labeled CoTs across 13 tasks and 10 models, and use it to conduct the first systematic evaluation of prominent faithfulness metrics. Our experiments show that most metrics perform near chance, exhibit strong prediction biases and degrade on longer CoTs. The best metric reaches only 0.71 AUROC at the CoT level while another reaches 0.59 at the step level, with neither transferring across settings, while entailing prohibitively high computational cost. Our results expose fundamental gaps in current faithfulness evaluation and call for the development of more reliable and efficient metrics.
Show more
When Medical VLMs Stop Understanding: MedTEC-Bench for Probing Semantic Specificity
Ahsan H Akash ⋅ Alina Devkota ⋅ Donald A Adjeroh ⋅ Binod Bhattarai ⋅ Prashnna K Gyawali
Medical vision--language models (VLMs) are attracting growing interest as a foundation for clinical image interpretation, retrieval, and decision support. However, current evaluations largely emphasize aggregate accuracy, retrieval, or classification performance, offering limited insight into whether these models possess the fine-grained semantic understanding required for safe and reliable use in high-stakes clinical settings. In this work, we introduce \textbf{MedTEC-Bench}, a diagnostic benchmark for probing semantic understanding in medical VLMs across five levels of increasing specificity, from broad modality recognition to fine-grained findings, and negation. MedTEC-Bench spans five medical imaging modalities: chest radiography, brain MRI, retinal fundus photography, dermoscopy, and histopathology. It combines controlled semantic probes with a suite of metrics designed to reveal failures hidden by aggregate benchmarks. Evaluating a diverse set of VLMs, from broad biomedical models to clinical-specialist models, we find that existing systems often fail on basic yet clinically meaningful semantic distinctions, despite strong reported performance on standard tasks. Our results suggest that current medical VLM evaluations may overestimate vision-grounded clinical understanding and highlight the need for targeted semantic benchmarks before such models are deployed in high-stakes medical workflows.
Show more
ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping
Joseph Marvin Imperial ⋅ Daniel Slate ⋅ Noam Kolt
AI agents deployed on the open web are becoming more capable of performing actions that may breach contracts and violate computer security laws. In this work, we introduce $\text{ScrapeBench}$, the first agentic benchmark for evaluating the legal compliance of frontier AI agents with website scraping policies, including terms of service (ToS) and $\texttt{robots.txt}$ directives. Across widely used closed-weight and open-weight AI agents, we find that most agents exhibit low compliance rates (< 5\%) when instructed to scrape websites in our sample of 1,626 popular websites that explicitly prohibit scraping. In addition, we observe that even where AI agents actively check the applicable ToS and $\texttt{robots.txt}$, they continue to exhibit low compliance rates. Conversely, in our sample of 269 websites that do not prohibit scraping, we observe that agents regularly refuse to scrape despite it being lawful, such as Claude Opus 4.6 exhibiting an over-refusal rate of 98.8%. Lastly, our human baseline experiments ($n$ = 180 participants) suggest that humans exhibit compliance rates comparable to AI agents but over-refuse at much higher rates. We release our anonymized code and data in this repository: https://anonymous.4open.science/r/scrapebench-75D6
Show more
Offloading Score: Measuring AI Reliance through Counterfactual Workflows
Vishakh Padmakumar ⋅ Lujain Ibrahim ⋅ Zora Wang ⋅ Jennifer Wang ⋅ Q.Vera Liao ⋅ Diyi Yang
AI tools are increasingly integrated into real-world workflows. However, existing measures of reliance on these tools focus on AI output adoption or on self-reported indicators, rather than how task effort is distributed between users and tools. Here, we introduce offloading score, a measure of reliance that quantifies the fraction of cognitive effort offloaded to an AI tool. Offloading score is simulation-based---we construct a counterfactual workflow by estimating how the user would have completed the task without the tool, and then computing the fraction of steps saved by using the tool. We validate offloading score through intrinsic evaluations of metric validity, and a controlled user study ($n=40$) with developers performing programming tasks using AI tools. We vary time pressure to test whether reliance measures capture the known increase in reliance under time pressure. We show that offloading score detects significantly higher reliance in time-constrained settings ($+43\%$, $p=0.018$), while usage-based and self-reported baseline measures do not distinguish the conditions. We complement this with descriptive insights showing that higher reliance manifests as greater delegation and more direct reuse of AI outputs. Finally, we demonstrate an approach of using offloading score in combination with target outcomes of reliance (e.g., task understanding) to identify when reliance may be (in)appropriate. Our framework offers two contributions: an instrument users can apply to measure and reflect on their own reliance, and a quantitative signal that agent designers can utilize to mitigate overreliance.
Show more
MMA-SafetyBench: A Benchmark for Multimodal Agent Safety Evaluation
Yuke Wang ⋅ Benlei Cui ⋅ Shen Pang ⋅ Xuemei Dong ⋅ Longtao Huang ⋅ Hui Xue' ⋅ Yuwen Zhai ⋅ Junjie Li ⋅ Jingqun Tang ⋅ Haiwen Hong
As multimodal agents increasingly rely on visual perception to navigate complex digital workflows, the vulnerability of their reasoning-action cycles to cognitive manipulation has emerged as a critical security priority. This systemic reliance on visual-semantic grounding introduces a fundamental susceptibility to executive hijacking. Unlike traditional pixel-level adversarial noise, this threat operates at the cognitive-semantic level by synthesizing high-fidelity, deceptive UI elements—such as forged system alerts or mandatory compliance overlays—that seamlessly subvert an agent's safety alignment. We propose a targeted evaluation methodology that injects context-aware payloads into specific vulnerability windows within an agent's reasoning trace. By specifically changing the attack content within the observation space while strictly maintaining a static attack strategy, we demonstrate the consistent subversion of executive intent across diverse task trajectories. Building upon this, we introduce MMA-SafetyBench, the first comprehensive safety evaluation suite for multimodal agents. Comprising 1,083 adversarial trajectories, the benchmark evaluates these vulnerabilities across five ubiquitous environments: web automation, desktop/mobile OS control, visual document understanding, and long-form video reasoning. Our evaluation of nine state-of-the-art foundational models reveals a pervasive risk of executive hijacking, where agents disproportionately prioritize deceptive observations over benign textual constraints. With the Semantic Compromise Rate (SCR) reaching an alarming 98.09%, our findings expose a critical blind spot in current safety alignments and underscore the urgent need for cross-modal cognitive defenses in autonomous multimodal architectures.
Show more
VeriVul: A Verification-Guided Framework for Generating Realistic Vulnerability Benchmarks
Ahmed Lekssays ⋅ Hamza Mouhcine ⋅ Issa Khalil
Evaluating machine-learning-based vulnerability detectors requires datasets that pair accurate labels with realistic code. Existing benchmarks satisfy at most one. Real-world collections rely on noisy commit-based labels because manual curation by security experts does not scale, while verified synthetic collections generate code from scratch that is too simple to resemble production software. The rapid advance of code LLMs adds a third pressure, since their training corpora absorb public CVE data and most widely cited vulnerability benchmarks, leaving static evaluations of such models prone to measuring memorisation rather than analysis. As model knowledge cutoffs continue to move forward, any fixed dataset risks becoming part of the next generation's training set. We present VeriVul, a framework that produces formally verified vulnerability data on demand. Given a CVE-fixing commit, it applies a vulnerability-aware backward program slicer to isolate the fragment relevant to the vulnerability, prompts a code LLM to expand the fragment into a self-contained program, and certifies the resulting program with the ESBMC bounded model checker, recording the verified label together with the trace that triggers the violation. Each verified program is paired with an ESBMC-verified counterpart that flips the label. We characterise the synthesised programs along vulnerability-relevant structural dimensions (e.g., control-flow density, pointer and array operations, struct-field access, and identifier diversity) and show that VeriVul dataset samples lie distributionally closer to real-world vulnerable code than the prior verified synthetic baseline, while remaining sufficiently complex to challenge state-of-the-art LLM-based detectors. In a prompted evaluation, Claude Opus 4.7 reaches only 52.6 percent accuracy on VeriVul dataset despite being given the full self-contained program. Because each sample carries the ESBMC counterexample and the violating statement, VeriVul also supports evaluation beyond binary classification, including root-cause identification and trace-grounded explanation. We release the framework, the slicer, the verification harness, and the generated dataset to support contamination-resilient evaluation of code LLMs as vulnerability analysts.
Show more
VocalGrad: Evaluating Acoustic Perception in Audio Language Models
Koki Ryu ⋅ Hitomi Yanaka
Recent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition. However, these models still struggle to reliably utilize acoustic information, particularly when it co-occurs with spoken linguistic content. Existing benchmarks do not clearly distinguish acoustic perception from higher-level reasoning, making it difficult to identify the source of these failures. To address this issue, we introduce \textsc{VocalGrad}, a benchmark designed to evaluate LALMs' ability to perceive low-level acoustic variations. VocalGrad formulates this problem as a simple binary classification of temporal change direction, enabling evaluation largely independent of higher-level reasoning or semantic understanding. Our benchmark results reveal a substantial gap in acoustic perception between human annotators and state-of-the-art LALMs: while human annotators achieve high accuracy, LALMs perform at near-chance levels. We further conduct linear probing to identify structural bottlenecks in these models. Our analysis reveals two distinct failure modes: for some categories, task-relevant information is preserved in internal representations but not utilized by the language model head; for others, the information becomes progressively less linearly decodable across model layers. Overall, our results reveal substantial limitations of LALMs in modeling acoustic variation under controlled settings and expose representation differences not observable from text outputs.
Show more
CausalConflictBench: Can Multimodal Models Follow Local Mechanisms That Conflict with Commonsense?
Bo Tian ⋅ Jianfeng Qu ⋅ Peng-Fei Zhang ⋅ Siyu Li ⋅ Zhixu Li ⋅ Kaiye Yu
Large multimodal models often benefit from commonsense priors, but these priors can mislead reasoning when a task specifies a local mechanism that conflicts with real-world regularities. Existing scientific VQA and visual reasoning benchmarks tend to align problem mechanisms with commonsense, so a correct answer may reflect either mechanism following or prior-based answering. We introduce Causal-ConflictBench, a diagnostic benchmark that makes this ambiguity observable by constructing samples where the current cause-to-effect mechanism contradicts the default commonsense mechanism. CausalConflictBench contains two modules: Textual Rule Override (TRO), which provides explicit commonsense-conflicting textual rules in real science image questions, and Visual Counter-Commonsense Induction (VCI), which requires models to induce a conflicting mechanism from a three-frame visual sequence. Beyond sample-level accuracy, we report Group-Strict Accuracy, factual-prior fallback metrics, and output-level process diagnostics. Across 14 proprietary and open-source multimodal models, we find that high sample-level accuracy can mask unstable mechanism following; errors concentrate strongly on factual-prior answers; and failure modes differ across rule-delivery paths, with TRO revealing rule-application failures and VCI revealing visual rule-induction failures. CausalConflictBench therefore provides a controlled diagnostic setting for uncovering commonsense fallback hidden beneath aggregate accuracy.
Show more
AutoHoney: Automating, Deploying, and Evaluating Scheming Honeypots Across Production Codebases
Martin Ciesielski-Listwan ⋅ Alyssia Jovellanos ⋅ Victoria Krakovna
LLMs are developing increasing capabilities in situational awareness, stealth and autonomy, approaching levels where they may be capable of scheming against their designers by covertly acting on misaligned goals (such as sabotaging oversight mechanisms under cover of a refactoring task). We evaluate model propensity for scheming using honeypots - realistic opportunities to pursue a misaligned goal. We present AutoHoney, an automated pipeline to generate, parameterize, and evaluate scheming honeypots in production codebases. Existing scheming propensity evaluations focus on catching and observing scheming behavior in artificial test settings, whereas we deploy honeypots in real codebases. To study scheming propensity in current models, we use prompted model organisms (PMOs) where goal conflicts are introduced through system prompts, planted documentation, or codebase-embedded environmental cues. Using PMOs with varying levels of goal nudging, we deploy honeypots across four production codebases and conduct a systematic manual ablation on multiple frontier models. Results show that goal presence, threat intensity, and task scope are strong predictors of scheming rates in our model organisms, while urgency has limited effect. The specific goal matters less than its existence—arbitrary goals like "promote mechanical pencils," environmental advocacy, and library funding all produce substantial scheming. Evaluation awareness typically suppresses scheming, but we observe notable exceptions where models take honeypots despite recognizing the evaluative context.
Show more
Discrete Diffusion Playground: A 2D Benchmark for Discrete Generative Models
Justin Shao ⋅ Sihyun Park ⋅ David Koes ⋅ Maria Chikina
Discrete diffusion is a promising framework for categorical generative modeling, but current evaluation protocols often fail to reveal whether models have actually learned the target distribution. We introduce the Discrete Diffusion Playground, a diagnostic benchmark that converts known 2D continuous distributions into binary token sequences through an exact discretized bijection. This creates a rare setting in which models are trained on discrete sequences, but generated samples can be decoded back into 2D and compared directly against a known ground-truth distribution. The benchmark spans smooth Gaussian mixtures and tilted checkerboards with sharp boundaries, disconnected support, and coordinate-dependent structure, allowing it to expose failures that are invisible in standard proxy metrics. It supports exact grid divergences, Wasserstein distances, density visualizations, and nonparametric two-sample tests, providing both quantitative evaluation and interpretable failure diagnosis. Using this testbed, we show that masked diffusion models can learn meaningful adaptive reveal orders but remain less stable than autoregressive baselines on smooth targets and largely fail on tilted checkerboards, where autoregressive models preserve the target geometry. Control experiments with reversed and interleaved token orderings show that autoregressive performance is not explained by favorable bit order alone.The Playground provides a simple but stringent benchmark for stress-testing discrete generative models and for developing evaluation metrics that detect real distributional mismatch rather than proxy performance. Our code is available at https://anonymous.4open.science/r/DiscreteDiffusionPlayground-0CCA/
Show more
MapShift: Controlled Post-Intervention Evaluation for Embodied World Models
Aarav Sinha
Embodied agents are often evaluated in the same environment in which they were explored or trained, making it difficult to assess whether a learned world model supports planning after the world changes. Existing evaluations can conflate memory of the explored environment, belief update after change, and planning under the updated belief. We introduce MapShift, an executable benchmark for controlled post-intervention evaluation (CPE): an agent explores a base environment without a task reward; the environment is modified by a controlled intervention in the metric, topology, dynamics, or semantics; and the agent is evaluated on post-intervention planning, inference, and adaptation tasks. The contribution is measurement infrastructure: matched base/intervened pairs, family-wise estimands, severity ladders, invariant validators, benchmark-health gates, protocol-comparison tooling, and reproducible artifact generation. In the expanded 24-motif release, health gates pass with zero fatal leakage, zero task rejections, perfect reference solvability, no intervention-validator failures, and no severity-magnitude failures. A deterministic mechanism diagnostic shows that same-environment evaluation underestimates the belief-update advantage by 3x on topology shifts, from $\Delta=0.304$ under CPE versus $0.102$ same-environment, and by 7x on semantic shifts, from $\Delta=0.724$ versus $0.102$; planning slices also exhibit protocol-induced rank reversals.
Show more
PotARCin: Multi-Dimensional Evaluation of Skill Acquisition in Abstract Reasoning Tasks
Claas Beger ⋅ Ryan Yi ⋅ Melanie Mitchell
The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating skill acquisition in AI models. Yet standard ARC evaluation considers only a single capability: producing the correct output grid for a test input. We argue that this narrow format underestimates the diversity of abilities required for genuine abstract skill acquisition. We introduce PotARCin, a benchmark that extends ARC by assessing understanding of a task’s underlying abstract rule across five dimensions: Definition, Classification, Constrained Generation, Editing, and Inversion. PotARCin uses explicit generator and verifier programs, enabling dynamic generative sampling beyond fixed input-output pairs. Across four state-of-the-art models evaluated on the ARC-AGI-1 training set, we observe a 30–50 percentage-point performance gap between standard ARC evaluation and PotARCin. We further investigate effects of generative sampling, difficulty of corruption types and questions of self-consistency. We also introduce P-ARC, a hand-crafted test set with generator and verifier programs, on which models achieve 0-8% accuracy across all five dimensions, underscoring the difficulty of acquiring versatile abstract skills.
Show more
Beyond IID: How General Are Tabular Foundation Models, Really?
Lennart Purucker ⋅ Andrej Tschalzev ⋅ Nick Erickson ⋅ David Holzmüller ⋅ Alan Arazi ⋅ Alexander Pfefferle ⋅ Mustafa Tajjar ⋅ Gioia Blayer ⋅ Gael Varoquaux ⋅ Frank Hutter
Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry. Research communities across disciplines are increasingly evaluating tabular foundation models on diverse datasets and tasks. However, these task- and discipline-specific evaluations remain largely inaccessible to model researchers because benchmark software and evaluation protocols are fragmented. As a result, model researchers rely on standard benchmarks, which are mostly defined for tasks where tabular foundation models already excel. The most challenging scenarios are excluded, limiting meaningful progress in the field by focusing on marginal improvements on IID data rather than on broader, more demanding challenges. To overcome this, we introduce BeyondArena, the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types (with text, with high cardinality) from a broad range of disciplines. To enable unified benchmarking beyond standard benchmarks, we introduce DataFoundry, a Python framework and metadata schema for curating tabular datasets for predictive machine learning. Our results across 11 models and 142 curated datasets show that existing tabular foundation models excel on tiny- to medium-sized IID data, while traditional tree-based and deep learning models still dominate on non-IID, large, and high-dimensional datasets. BeyondArena guides model research for the most demanding challenges in tabular data, enabling progress towards truly foundational tabular models.
Show more
EvoCodeBench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
Haiyang Shen ⋅ Xuanzhong Chen ⋅ Wendong XU ⋅ Yun Ma ⋅ Liang Chen ⋅ Kuan Li
Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This fails to capture whether agents can maintain executable artifacts as requirements evolve. We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks spanning 227 evaluated rounds. Each task preserves the agent's workspace across 5--15 rounds, expresses requirements through observable behavior, and uses cumulative executable tests to check both new requirements and regressions against still-active prior ones. We evaluate 13 coding agents with two complementary metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22--40 points, indicating that isolated instruction-following does not imply persistent reliability. The gap also changes model rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only \~50\% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round~5. Failure analysis reveals tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. These results show that single-round and persistent multi-turn coding evaluate different capability dimensions. We release the benchmark data and Harbor multi-turn infrastructure.
Show more
Benchmarking Compositional Generalisation for Machine Learning Interatomic Potentials
Amir Masoud Nourollah ⋅ Irtaza Khalid ⋅ Stefano Leoni ⋅ Steven Schockaert
Machine Learning Interatomic Potentials play a fundamental role in computational chemistry and materials science, enabling applications from molecular dynamics simulations to drug design and materials discovery. While recent approaches can estimate inter-atomic forces with high precision, it remains unclear to what extent they can generalise to previously unseen molecules. Do they learn the compositional structure of chemistry, capturing how molecular fragments and their combinations determine properties, or do they primarily learn to interpolate patterns that are specific to the training examples? To address this question, we propose a benchmark consisting of four tasks that require some form of compositional generalisation. In each task, models are tested on molecules that were unseen during training, but the training data is chosen such that generalisation to the test examples should be feasible for models that learn the underlying physical principles. Our empirical analysis shows that the considered tasks are highly challenging for state-of-the-art models, with errors on out-of-distribution examples often an order of magnitude higher than on in-distribution examples, even when using foundation models that have been pre-trained on millions of molecules.
Show more
Leak-CURBER: A Leakage-Controlled Multimodal Evaluation Benchmark for Enzymatic Reaction Tasks
Adhil Ahmed P M Shums ⋅ Roshan Balaji ⋅ Nirav Bhatt
We present Leak-CURBER, the largest multimodal, leakage-controlled benchmark for evaluating machine learning and deep learning models for enzymatic reactions. Leak-CURBER standardizes data from 10 biochemical databases and curates datasets with multimodal inputs, metrics, and leakage-controlled splits for 4 tasks. The benchmark comprises 10 subtasks: prediction of reaction outcomes (2), enzyme function (2), kinetic parameters (3), and protein–ligand binding (3). Leak-CURBER establishes new evaluation protocols that control for train/validation/test overlap across protein sequence and structure, molecule string and structure, and reaction similarity, thereby exposing generalization failures of learning methods. Evaluating the state-of-the-art (SOTA) models on Leak-CURBER shows that their strong performance degrades sharply under these splits, often approaching random classification or retrieval performance, and near-zero regression performance. Leak-CURBER includes precomputed embeddings, continual benchmark releases, reproducible curation scripts, and a public leaderboard to support reliable comparison of enzymatic-reaction models.
Show more
TraceSim: A Generative Simulator and Benchmark for Joint scRNA-seq and Lineage Tracing
Mehrshad Sadria ⋅ Katsuki Fujisawa ⋅ Xun Shen
Single-cell lineage tracing paired with transcriptomics is fundamentally transforming our understanding of cellular differentiation. However, due to immense technical difficulties and experimental complexity, paired barcoding–scRNA-seq datasets remain scarce, fragmented across only a handful of studies, and frequently corrupted by stochastic barcode silencing and site-level dropout. Because transcriptomic sequencing is destructive, the continuous trajectories of differentiating cells cannot be observed directly, leaving accurate generative simulators as the only practical path to the standardized ground- truth benchmarks needed for rigorous method validation. Yet the few existing simulators generate gene expression as a strictly Markovian process, failing to capture the fate bias seen in real biological systems, where early progenitor cells already carry coordinated transcriptomic signatures of their distant terminal fates. Consequently, modern machine learning methods designed to predict cell fate cannot be rigorously benchmarked: the synthetic data systematically lacks the predictive signals these models are meant to learn. We introduce TraceSim, a highly controllable, continuous multi-fate simulation framework. By modeling cellular differentiation as an overdamped Langevin process on a Waddington landscape, TraceSim generates non-Markovian transcriptomic trajectories whose fate bias is tunable by construction, together with tunable parameters for asynchronous apoptosis, CRISPR barcoding errors, and transcriptomic signal-to-noise sparsity. Through a comprehensive weakness-targeted evaluation, we demonstrate that TraceSim successfully maps the predictive limits of modern architectures and shows the structural limitations of current simulators. TraceSim establishes a new standard for validating lineage reconstruction and early cell fate prediction methods.
Show more
NanoFold: Designing Reproducible Protein Structure Benchmarks through Principled Sampling
Chris Hayduk ⋅ Krithik Ramesh
Protein structure prediction has progressed rapidly, with a flourishing ecosystem of open-source AlphaFold (AF)-style systems delivering real progress. But isolating which architectural and training choices actually drive that progress is difficult. Production-scale training is both computationally prohibitive and not accessibly reproducible; moreover, the public corpora are complex, with structural biases that compound in parameter- and compute-constrained regimes. To address these gaps, we introduce NanoFold, a compact, fixed-data benchmark for AF-style training studies. Its sealed-hidden tracks address three fundamental questions, with a $\textit{limited}$ track for sample efficiency, a $\textit{research large}$ track for whether early gains persist under more optimization, and an $\textit{unlimited}$ track for best fixed-data final performance. NanoFold's split is designed rather than naively sampled, with chains grouped into biological units under MMseqs2 cluster and PDB-entry disjointness, stratified across structural metadata, and allocated to $10{,}000$ training, $1{,}000$ public-validation, and $1{,}000$ sealed hidden chains. We verify the construction through statistically principled diagnostics, including a randomization study over $1{,}000$ alternative valid splits, confirming the dataset is diverse, well-distributed, and statistically typical of its constraint class. We use NanoFold to run comprehensive experiments on model behavior across scales and training regimes, demonstrating that the benchmark is learnable but unsaturated, scales predictably with budget, cleanly separates training primitives, and that the underlying codebase flexibly supports head-to-head comparison. NanoFold makes architectural and training progress in protein structure prediction more transparent, comparable, and reproducible at a compute-accessible scale.
Show more
HumanoidArena: Benchmarking Egocentric Hierarchical Whole-body Learning
Taowen Wang ⋅ zikang xie ⋅ Bin Yang ⋅ Yunheng Wang ⋅ Zizhao Yuan ⋅ Yuetong Fang ⋅ Yixiao Feng ⋅ Yichi Wang ⋅ Xingyu Chen ⋅ Haodong Chen ⋅ Qiwei Wu ⋅ Weisheng Xu ⋅ Lihan Chen ⋅ Lusong Li ⋅ zeng zecui ⋅ Renjing Xu
Humanoid robots promise whole-body interaction in human-centered environments, but scalable policy learning remains difficult because task-level semantic reasoning and whole-body level dynamic execution are tightly coupled. A practical solution is hierarchical control, where a high-level policy predicts intermediate whole-body actions and low-level general motion trackers (GMTs) execute them as stable humanoid motion. However, existing benchmarks rarely evaluate the policy–tracker interface itself, leaving open whether intermediate whole-body actions are executable, robust under task distribution shifts, and transferable across different GMT backends. We introduce \textsc{HumanoidArena}, a simulation-first benchmark for egocentric hierarchical whole-body learning. The benchmark formulates policy learning as a hierarchical decision making problem: a high-level policy converts egocentric vision, proprioception, and instructions into a compact whole-body action, which is subsequently executed by a low-level GMT. Instead of treating the legs as planar transport tools, \textsc{HumanoidArena} emphasizes interactions where lower-body coordination is structurally necessary in task completion. We therefore design 7 leg-critical HOI/HSI tasks in which success requires foot placement, balance maintenance, posture adjustment, and whole-body reorientation. To further diagnose the hierarchical system, we evaluate policies from two complementary perspectives: perturbation-conditioned generalization and GMT-conditioned transfer. We benchmark representative imitation-learning and VLA-style policies under this shared interface. Experiments show that hierarchical control enables learned policies to solve diverse leg-critical interactions, but performance is strongly tracker-conditioned and cross-GMT transfer remains fragile. These results position \textsc{HumanoidArena} as a benchmark for studying transferable intermediate action representations and scalable egocentric whole-body policy learning. Code and data are available at: \url{https://anonymous.4open.science/r/HumanoidArena-D36F}.
Show more
Auditing Capsule Vision 2024: Within-Split Train-to-Validation Re-Exposure and a Kvasir-Channel Sensitivity Diagnostic
Eunseob Choi ⋅ Kyeonghun Kim ⋅ Hyuk-Jae Lee ⋅ Nam-Joon Kim
Public validation sets in machine-learning challenges frequently become de facto test sets in downstream work, making train–validation source overlap a first-order concern for claim validity. We audit the released public split of the *Capsule Vision 2024 Challenge* (CV2024) using perceptual hashing (pHash+dHash, PDQ at 99.4%), pixel NCC, and DINOv2/ResNet feature checks. The released CV2024 public split contains 1,381 / 11,581 KVASIR validation rows (11.9%; 540 pixel-confirmed at NCC ≥ 0.99, i.e. 4.66%) with pHash-exact training matches — directly violating the organizers' "NOT have duplicates" preprocessing description. All 1,381 trace to shared Kvasir-Capsule video prefixes, consistent with frame-level (not video-level) random splitting. The full CV2024-KVASIR slice falls in the disclosed Kvasir-Capsule near-duplicate set (60.7% bit-identical; non-KVASIR controls ≤ 0.31%). Training on the Kvasir-origin-removed le6 split drops public-validation balanced accuracy by Δ = −0.213 (95% CI [−0.220, −0.206], n=10 paired seeds), quantifying the public score's dependence on the KVASIR channel; the magnitude is mechanically expected given the 71.8% KVASIR validation composition rather than a leakage-specific causal effect. le6 is not a hidden-test proxy (team-level Spearman against AIIMS: ρ=0.355 vs. 0.566 for the original score). We release le6, le6_plus_internal, hash/NCC annotations, audit code, and a Croissant 1.1 descriptor as a within-public-pool sensitivity probe; no image bytes are redistributed. Detector portability: ISIC 2019 cross-source rate is 0.008% (2/25,331, both re-attributed as within-source metadata-gap pairs ⇒ genuine cross-source 0/25,331; Cassidy et al. 2022's recall-tuned multi-hash pipeline flags 14,310); HyperKvasir and Kvasir-SEG cross-benches against CV2024 return 0 pairs (procedure-disjoint negative controls). At the matched NCC ≥ 0.99 endpoint, the CV2024 rate of 4.66% (540/11,581) is ≈ 148–583× above the ISIC cross-source rate (Wilson-lower / Wilson-upper to point/point).
Show more
How to Train a Surgeon? Benchmarking Generalist Agents in Surgical Scene Understanding
Gengyuan Zhang ⋅ Xiao Han ⋅ Shijie Zhou ⋅ Abdullah Koukash ⋅ Ghazal Ghazaei ⋅ Jindong Gu ⋅ Jingpei Wu ⋅ Yuqicheng Zhu ⋅ Nassir Navab ⋅ Volker Tresp
Evaluating general-purpose multimodal large language models (MLLMs) for surgical scene understanding remains limited by scarce surgical data and fragmented evaluation protocols. We address these challenges through a two-stage evaluation framework for surgical scene understanding. In Stage I, we introduce SURGKNOWBENCH, a diagnostic benchmark that harmonizes public surgical datasets across procedures and tasks and evaluates proprietary and open-weight MLLMs to map the boundary of pretrained surgical competence. SURGKNOWBENCH reveals that current models retain partial visual priors but remain weak on procedural reasoning and workflow assessment. Building on this failure frontier, and motivated by the scaffolded nature of surgical training, we introduce SURGAGENTGYM, a controlled environment where general-purpose MLLM agents can use bounded knowledge search over SURGWIKI and general-purpose visual grounding for surgical tasks. SURGAGENTGYM evaluates whether generalist agents can improve surgical understanding at inference time through tool-mediated evidence use, without additional surgical training. Our results show that agentic inference narrows the gap mainly for perception-heavy tasks, while fine-grained localization and expert-style assessment remain persistent bottlenecks. This opens a path toward future studies of agentic surgical understanding. We have released the codebase and artifacts.
Show more
TELEVAL: A Benchmark Designed for Spoken Language Models in Chinese Interactive Scenarios
Zehan Li ⋅ Hongjie Chen ⋅ Qing Wang ⋅ Yuxin Zhang ⋅ Jing Zhou ⋅ Hang Lv ⋅ Mengjie Du ⋅ Yaodong Song ⋅ Jie Lian ⋅ Jian Kang ⋅ Jie Li ⋅ Yongxiang Li
Spoken Language Models (SLMs) are expected to support natural spoken interaction beyond task completion. However, existing SLM benchmarks primarily evaluate semantic correctness in structured settings and provide limited assessment of interactional behavior grounded in acoustic context. To address this gap, we introduce TELEVAL, a large-scale SLM benchmark for Chinese spoken interaction in instruction-free, audio-conditioned settings. TELEVAL evaluates two complementary aspects: (1) Reliable Content Fulfillment, which measures semantic accuracy of SLMs under diverse acoustic and linguistic conditions, and (2) Interactional Appropriateness, which assesses whether models produce natural and appropriate responses by implicitly grounding behavior in auditory cues. Experiments show that while models perform competitively on semantic tasks, their performance degrades under acoustic variability and in interactional settings. We observe consistent degradation from perceptual instability to interactional errors, and further identify a recurring failure pattern, termed the "Caption Trap", where models tend to describe perceived audio signals rather than produce appropriate interactive responses. These results indicate that current SLMs remain insufficiently aligned with the requirements of natural spoken interaction. TELEVAL provides a targeted framework for evaluating and analyzing interactional behavior in SLMs.
Show more
Solving Every Step Is Not Enough: Milestone Oracles Reveal a Composition Gap in LLM Math Reasoning
Zhuohan Wang ⋅ Haoran Ma ⋅ Tianyu Wu ⋅ Yuanlin Duan ⋅ Zichun Liao ⋅ Jieming Yu
Large language models (LLMs) can solve every intermediate step of a multi-step math problem on its own and still fail the full problem, even when given a roadmap of the steps and all of their answers. We introduce OracleLadder, a diagnostic evaluation that locates where LLM math reasoning fails by giving the model increasing levels of oracle help. For each problem, a teacher model writes a fixed roadmap of intermediate sub-goals (milestones), and a deterministic symbolic verifier grades every answer. Testing the model with no help, with the roadmap, with the roadmap plus the milestone answers, and on each milestone alone sorts each failure into one of five reasoning gaps. On 354 NuminaMath problems and six models from 8B to 671B parameters (Qwen3, gpt-oss, Llama 3.3, DeepSeek-V3.1), the largest gap for every model is the composition gap, a stricter form of the compositionality gap. It covers 33-48% of problems, and 24-37% after removing problems that an LLM review flags as grading errors. Accuracy and milestone-help recovery rank the two strongest models differently, and two RLVR runs with similar accuracy gains move problems differently. The roadmap effect replicates on MATH500 and AIME 2024/25, per-problem recovery agrees for 83-87% of problems under an independent second teacher, and the help ladder carries over to code generation. We release the data, roadmaps, prompts, and code at https://github.com/slark-prime/OracleLadder.
Show more
When Does Graph Retrieval Become Answer-Supporting Evidence? A Diagnostic Audit of GraphRAG
Chenghua Duan ⋅ Zhuo Wang ⋅ boheng liu ⋅ Ziyu Li ⋅ Qing Li ⋅ Xiuxing Li ⋅ zhong zhang ⋅ Xia Wu
Graph retrieval-augmented generation (GraphRAG) is motivated by the idea that relational structure can help a language model find evidence that flat retrieval may miss. Yet retrieval scores and final-answer scores often compress a multi-stage pipeline into a single method row. We study the missing middle of this pipeline through an evidence-to-answer audit that follows each system from candidate availability, through evidence selection and context construction, to generated answers and cost. Across GraphRAG-Benchmark Medical and Novel tasks, we find a consistent conversion gap. Strong reranking, learned evidence scoring, and graph-based retrieval can improve evidence-side signals, but these gains do not yield significant answer-correctness improvements under the same generator and evaluator. Matched attribution controls show that some apparent graph-side gains are driven by reranking rather than graph expansion alone. Gold-evidence and bridge-aware interventions further show that evidence completeness is insufficient on its own; answerability also depends on salience, organization, and support in the generator-facing context. We therefore argue that GraphRAG evaluation should report the full evidence-to-answer conversion chain, including candidate source, selected context, generated answer, metrics, and cost, rather than relying on retrieval or answer leaderboards alone.
Show more
MEME: Multi-Entity & Evolving Memory Evaluation
Seokwon Jung ⋅ Alexander Rubinstein ⋅ Arnas Uselis ⋅ Sangdoo Yun ⋅ Seong Joon Oh
LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. While prior benchmarks evaluate only single-entity updates, MEME defines six tasks spanning the full space defined by the multi-entity and evolving axes, including three not scored by prior work: Cascade and Absence (dependency reasoning) and Deletion (post-removal state). Evaluating six memory systems spanning three memory paradigms on 100 controlled episodes, we find that all systems collapse on dependency reasoning under the default configuration (Cascade: 3\%, Absence: 1\% in average accuracy) despite adequate static retrieval performance. Prompt optimization, deeper retrieval, reduced filler noise, and most stronger LLMs fail to close this gap. Only a file-based agent paired with Claude Opus 4.7 partially closes the gap, but at $\sim$70$\times$ the baseline cost, indicating closure currently depends on configurations that are not practical at scale. Code is available at https://anonymous.4open.science/r/MEME-0612 and the dataset at https://huggingface.co/datasets/meme-benchmark/MEME.
Show more
NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces
Konstantinos Kontras ⋅ Trui Osselaer ⋅ Stylianos G Mouslech ⋅ Angeliki I. Karaiskou ⋅ Guido Gagliardi ⋅ Thomas Strypsteen ⋅ Mohammad Hossein Badiei ⋅ Anku Rani ⋅ Maarten Vanmarcke ⋅ Miguel Bhagubai ⋅ Chanakya Ekbote ⋅ Jaedong Hwang ⋅ Christos Chatzichristos ⋅ Paul Liang ⋅ Maarten De Vos
Foundation models (FMs) promise to extract unified representations that generalize across downstream tasks. They have emerged across fields, including electroencephalography (EEG), but it is less clear how effective they are in this particular field. Published evaluations differ in datasets, in the EEG-specific preprocessing that might influence reported results, and in the reported metrics, frequently obscuring the clinical relevance in EEG. We introduce NeuroAtlas, the largest EEG benchmark to date: 42 datasets and $\sim$260k hours covering clinical EEG (epilepsy, sleep medicine, brain age estimation) and brain-computer interfaces, and include multiple datasets per task along with bespoke clinical evaluation metrics. Besides evaluating EEG-FMs with respect to supervised baselines, we present results from generic time-series FMs. We report three findings. First, EEG-specific FMs do not consistently outperform time-series FMs, which have neither EEG-focused architectures nor been pretrained on EEG. Second, standard machine learning metrics are insufficient to assess clinical utility: thus, we thoroughly evaluate more appropriate measures such as the quality of event-level decision-making, hypnogram-derived features, and the brain-age gap in the domains of epilepsy, sleep, and brain age, respectively. Third, model rankings and performance can vary substantially within domains. We conclude that pretrained models perform largely on par, with only narrow advantages for a few, and that current models do not yet deliver on the promise of an out-of-the-box unified EEG model. NeuroAtlas exposes this gap and provides the datasets and metrics for the next generation of unified EEG FMs.
Show more
REAL-MED: Benchmarking LLM Agents on Real-World Medical Tasks
Zihan Wang ⋅ yifan zhang ⋅ Jieqiong Cao ⋅ Junxiu Chen ⋅ Shanni Chen ⋅ ChaoGao ⋅ Hao Wang ⋅ Shi Feng ⋅ Xiaocui Yang ⋅ Jinghao Lin ⋅ Kai WU ⋅ Xiaozhong Ji ⋅ Haihua Yang
Real-world medical work extends far beyond answering isolated clinical questions: it often requires clinicians to process complex materials, interact with information systems, and produce structured deliverables. As AI systems evolve from medical chatbots into medical agents, evaluation should likewise move beyond \emph{answer correctness} toward the \emph{quality of end-to-end workflow deliverables}. We introduce \textsc{Real-Med}, a benchmark for evaluating AI agents on real-world medical tasks. \textsc{Real-Med} contains 405 expert-designed tasks across four categories: Clinical Support, Patient Management, Pharmacy Management, and Medical Education \& Research. Each task is grounded in a realistic scenario with concrete materials, tool interfaces, and explicit deliverable requirements, such as clinical documents, medication plans, spreadsheets and review reports. To support rigorous evaluation, each task is annotated with fine-grained expert-written rubrics, averaging 15.4 independent rubric items per task, and every task is cross-validated by at least three medical experts. We evaluate 11 strong mainstream models under a unified harness with the same resources, tools, and execution protocol. Our results show that current agents can often satisfy surface-level task requirements, but still struggle with detail-sensitive medical reasoning, dosage-calculation traps, shallow research, formatting constraints, and reliable final delivery. These findings reveal a substantial gap between medical QA ability and robust execution in realistic medical workflows. Code is available at \url{https://anonymous.4open.science/r/REAL-MED}.
Show more
HSCO-Bench: An Agent-Driven End-to-End Hardware-Software Co-design Benchmark for Systems-on-Chip
Pei-Huan Tsai ⋅ Kuan-Lin Chiu ⋅ William Baisi ⋅ Pin-Yu Chen ⋅ Luca P Carloni
Large language models (LLMs) are increasingly adopted for software and hardware design, yet these domains are still evaluated separately. Software benchmarks typically assume fixed hardware targets, while hardware benchmarks focus on component-level optimization without considering the full hardware-software stack. Consequently, no existing benchmark evaluates whether an LLM agent can perform end-to-end, system-level hardware-software co-design. Such a process requires: 1) analyzing applications to identify kernels requiring acceleration, 2) designing and \emph{integrating} heterogeneous accelerators into a System-on-Chip (SoC) under resource constraints, and 3) mapping kernels onto the generated accelerators. We present \hsco{}, an end-to-end hardware-software co-design benchmark for accelerator-rich heterogeneous SoC generation. Built upon an open-source SoC platform with a curated repository structure, HSCO-Bench evaluates the ability of LLMs to jointly optimize software and hardware stacks, producing SoC prototypes deployed on the AMD Virtex-7 FPGA VC707 Evaluation Kit. Experimental results show that end-to-end integration remains challenging for current models, with widespread failures. Among the five frontier models evaluated, only two of them could successfully generate valid SoC prototypes. Yet, even in these successful instances, the generated designs are far from optimal. While we observe a promising peak speedup of 16.22x, the maximum additional resource utilization reaches only 23.67%. This highlights that while state-of-the-art models demonstrate an emerging capability for hardware acceleration, they still heavily underutilize the available hardware capacity, leaving substantial room for future optimization. To the best of our knowledge, HSCO-Bench is the first benchmark targeting this complete co-design flow, enabling LLMs to jointly reason about and modify both the software and hardware stacks of heterogeneous SoCs.
Show more
PCBInnoBench: Benchmarking LLM Agents on Real-World PCB Design
Fuyuan Xia ⋅ Jiaxin Hu ⋅ Kewei Yu ⋅ Zihan Zhang ⋅ Zheng wei ⋅ Hehan Liang ⋅ Xianyue Chen ⋅ Tongze Zhang ⋅ Xingbo Feng ⋅ Shixiong Kai ⋅ Zhentao Tang ⋅ Yuqi Cui ⋅ Mingxuan Yuan ⋅ Linfeng Zhang ⋅ Qibing Ren ⋅ Junchi Yan
LLM-based agents have made rapid progress on software engineering benchmarks, and executable evaluation is extending to chip-level hardware tasks. Board-level PCB design and upgrade has yet to receive comparable evaluation, largely because open-source PCB projects rarely provide the structured issue/PR/test histories that existing benchmarks rely on. We introduce PCBInnoBench, the first benchmark that evaluates LLM agents on executable PCB design-and-upgrade workflows. PCBInnoBench contains 372 expert-designed KiCad tasks grounded in real PCB projects and end-to-end engineering workflows. Each task pairs an agent-visible package (KiCad project and multimodal engineering references) with a hidden evaluation package (ground-truth design and expert-reviewed validators) used only for scoring. Agents must modify existing ECAD projects and pass structural, task-specific, and cross-artifact engineering-constraint checks. Each task is evaluated under two prompts of different guidance levels: one supplies an expert execution trajectory of ECAD edits together with the consulted references, the other states only the task objective, requiring agents to navigate the full workflow from requirement localization through cross-artifact editing. The best agent achieves 22.0% Pass@1 with guidance but only 7.3% without, indicating that both workflow discovery and ECAD execution remain substantial challenges for current agents. Beyond PCB, the workflow-grounded, expert-constructed evaluation approach may serve as a reference for other engineering domains where historical traces are sparse and correctness spans heterogeneous artifacts. Data and code are available at https://anonymous.4open.science/r/pcbinnobench-C841 .
Show more
Ask-E: An Environment for Calibrated Question Generation
Sarah Pratt ⋅ Jae Sung Park ⋅ Scott Geng ⋅ Ali Farhadi
Today, we improve models by training and evaluating them on problems at the frontier of their abilities. Creating such problems is itself a demanding task, requiring the ability to probe model limits and generalize beyond existing question distributions. It also means placing problems at a precise difficulty level, which requires understanding what it takes to solve them. In short, generating problems calibrated to a model's current frontier demands capability beyond it, an increasingly burdensome constraint as models improve. Our key insight is that we can leverage this constraint to our advantage: a model that can generate problems consistently calibrated to a given frontier must possess capability beyond it. Accordingly, we present Ask-E, an environment that benchmarks and trains models on their ability to write questions at a given skill level, rather than answer them. Concretely, we define target skill levels as ranges bounded by the capabilities of two existing language models. A generated question is successfully calibrated if exactly one of the two models can solve it, placing it precisely within the target range and differentiating the capabilities of these models. Ask-E serves both as a benchmark and a training environment, where models generate problems calibrated to a variety of skill levels. We find that even frontier models achieve below 50\% calibration on the benchmark, leaving significant headroom to measure future progress. We also show that training on this environment leads to improvements across a number of downstream math benchmarks even with no new math data, no interaction with stronger models, and no correctness-based reward.
Show more
C2G-BENCH: A Cyber-Physical Evaluation Benchmark for Hierarchical Reinforcement Learning in Grid-Interactive Hyperscale Data Centers
Vineet Gundecha ⋅ Sahand Ghorbanpour ⋅ Sifat Abdullah ⋅ Mahasweta Chakraborti ⋅ Damien Fay ⋅ Avisek Naug ⋅ Sergey Serebryakov ⋅ Ricardo Luna Gutierrez ⋅ Soumyendu Sarkar
The rapid growth of generative AI is reshaping the energy footprint of modern computing infrastructure. Hyperscale AI data centers now operate at power levels comparable to large industrial facilities, creating new challenges for grid reliability, energy procurement, and real time demand flexibility. At the same time, their large controllable loads, cooling systems, and on site storage make them potential participants in grid services such as frequency regulation. However, existing evaluation environments do not capture the coupled decision making required for grid interactive operation. We introduce C2G-BENCH, a cyber-physical benchmark for hierarchical reinforcement learning in hyperscale data centers. C2G-BENCH simulates a real-world scenario, where a 250 MW-class hyperscale data center participates in grid frequency regulation while serving realistic AI workload demand under weather-driven ambient conditions. The benchmark explicitly links 15-minute market level orchestration with 5-second physical control, enabling agents to jointly reason over regulation commitments, workload flexibility, cooling, battery dispatch, and facility safety. C2G-BENCH achieves this through two coupled Gymnasium environments: C2GMacroEnv, where a high level controller bids into the regulation market, and C2GFastEnv, where a low level controller actuates the Dynamic Voltage and Frequency Scaling (DVFS), Coolant Distribution Unit (CDU) pump speed, HVAC effort, and Battery dispatch (BESS). The simulator models a 250 MW class two zone facility using seven modular physics engines covering workload, thermal, electrical, battery, grid signal, weather, and market dynamics, driven by real workload and weather traces. We benchmark a family of rule-based, LLM-based & RL-based controllers for both the macro and fast environments, and evaluate them on four stress-test environments in multiple grid markets. Baseline results show that active low level control substantially improves tracking quality: a rule-based macro controller paired with a RL-based low level controller reduces the tracking RMSE from 2,737 kW to 229 kW compared with a macro only configuration, while maintaining full throughput and zero thermal violations. These results demonstrate that C2G-BENCH exposes meaningful cross layer tradeoffs and provides a reusable benchmark for studying safe, hierarchical control of grid interactive AI data centers.
Show more
Agent$^2$ RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?
Wanyi Chen ⋅ Xiao Yang ⋅ Xu Yang ⋅ Tianming Sha ⋅ Qizheng Li ⋅ Zhuo Wang ⋅ Bowen Xian ⋅ Fang Kong ⋅ Weiqing Liu ⋅ Jiang Bian
We introduce Agent$^2$ RL-Bench, a compact diagnostic benchmark for evaluating agentic RL post-training, which tests whether LLM agents can autonomously design, implement, debug, and execute post-training pipelines that improve foundation models. RL post-training increasingly drives model alignment and specialization, yet existing benchmarks are largely static, rewarding supervised fine-tuning or script generation without assessing an agent's ability to close an interactive RL loop. Agent$^2$ RL-Bench provides a unified agent-facing interface: each run starts from an isolated workspace containing a base model, task data, instructions, and a grading API, and agents must iterate within a fixed budget by training models and submitting artifacts for evaluation. The benchmark spans six tasks across three levels, from static rule-based training to judge-based optimization and closed-loop online RL with trajectory collection. Two diagnostic skills, namely runtime recording and post-hoc summarization, enable structured analysis of agent behavior, facilitating smooth and effective iteration of the benchmark's evaluation framework. Across five agent systems and six driver LLMs, agents show intelligent behavior but clear limitations: one RL-oriented run improves ALFWorld from 4.85 to 93.28 via SFT warm-up and GRPO with online rollouts, yet DeepSearchQA remains difficult, most successful routes rely on supervised pipelines, and interactive outcomes show large single-run differences across agent stacks. Overall, Agent$^2$ RL-Bench shows that current agents can sometimes engineer online RL, but stable agent-driven RL post-training remains rare under fixed budgets. It also demonstrates that our benchmark provides a strong and effective evaluation framework for future research in this direction. Code is available at https://anonymous.4open.science/r/Agent2RL-Bench-5587.
Show more
EpistasisBench: Revealing Structural Limitations of Zero-Shot Protein Language Models
Junan Chen ⋅ jianwen Min ⋅ Liang He ⋅ Yiheng Zhu
Zero-shot mutation effect prediction is widely used to evaluate protein language models (PLMs). However, because existing benchmarks mix additive and non-additive mutations, and non-additive effects (epistasis) are sparse yet critical, it remains unclear whether these models truly capture thermodynamic residue coupling. As a result, strong overall performance may mask weaknesses in modeling multi-residue interactions. To make this distinction explicit, we introduce EpistasisBench, a thermodynamically stratified zero-shot benchmark constructed from large-scale stability measurements. By isolating significant and sign epistatic double mutations, it separates additive effects from thermodynamic coupling and defines two complementary tasks: ∆∆G prediction restricted to epistatic subsets, and a direct ∆∆Gint scoring task that examines non-additive structure in model probability space. When evaluated across more than 17 state-of-the-art PLMs, EpistasisBench reveals a consistent pattern. Performance declines under strict epistasis filtering, strong single-site predictors fail on coupled mutations, and standard masked language model scoring shows no variation in ∆∆Gint, thereby preventing detection of non-additive effects. Together, these findings indicate that the limitation lies in common zero-shot inference schemes rather than model capacity. To demonstrate that non-additive signal can in fact be recovered under a more suitable inference formulation, we introduce a simple joint marginal inference mechanism that restores non-additive signal and improves performance without fine-tuning. Overall, EpistasisBench provides a physically grounded test for multi-residue interactions and highlights a key gap in current zero-shot evaluation of protein language models. The dataset is available at https://www.openml.org/d/47245
Show more
CREF: Forecasting Benchmarks for the Age of Agents
Andreas Auer ⋅ Abdul Fatir Ansari ⋅ Oleksandr Shchur ⋅ Xiyuan Zhang ⋅ Boran Han ⋅ Pedro Mercado ⋅ Prateek Desai ⋅ Yuyang (Bernie) Wang ⋅ Michael Bohlke-Schneider
Traditional forecasting evaluations relying on static historical data are fundamentally incompatible with modern AI advances. Even for standard time series foundation models, temporal overlap between training and test periods can lead to implicit leakage through correlated signals. This problem is intensified by LLMs, which can bridge semantic gaps between forecasted and related data while memorizing extensive world knowledge. At the extreme, agentic models with tool access can retrieve the ground truth directly. Such information leakage inflates performance relative to real forecasting scenarios and is impossible to rule out in historical evaluations. We introduce CREF, a real-time benchmark that evaluates forecasting models on data that does not yet exist, eliminating all forms of leakage by design. CREF balances three competing objectives for real-time benchmarks: diversity, robustness, and throughput. It draws data from 32 public APIs and features 50 forecasting tasks with diverse data frequencies, multivariate targets, and covariates, making it the first real-time benchmark reaching the breadth of leading static benchmarks. Retrospective evaluation validates that CREF produces rankings consistent with established static benchmarks, while real-time evaluation reveals qualitative and quantitative performance gaps for agentic and LLM-based models, confirming leakage on historical data. CREF runs continuously and is open for new models to join, offering a public leaderboard for contamination-free model comparison.
Show more
InSpect: A Curated Natural History Collection Dataset for Insect Specimen Understanding
Junhao Zhang ⋅ Feng Xu ⋅ Ahalya Ravendran ⋅ Nicole Fisher ⋅ Brendan Tidd ⋅ Lars Petersson ⋅ Dadong Wang ⋅ Xun Li
Visual insect specimen understanding aims to identify biological taxa and characterize morphology-relevant structures from specimen images. Existing insect benchmarks have advanced this problem through large-scale pretraining and multimodal learning, but remain centered on taxonomic recognition with strong auxiliary modalities. We introduce \textbf{InSpect}, a curated natural history collection dataset containing 48,184 digitized insect specimen images with aligned crops, hierarchical taxonomy, label-derived structured metadata, and 1,859 images with fine-grained anatomical part masks. We thus establish two benchmark settings. The first is \textbf{open taxonomic recognition}, which evaluates zero-shot, fine-tuned, and unseen-taxon recognition using cropped insect images, label-derived metadata, or their combination. The second is \textbf{fine-grained anatomical part segmentation}, which evaluates supervised and text-guided segmentation of insect structures such as antennae, legs, wings, and body regions. Experiments show that open taxonomic recognition remains challenging despite strong closed-set performance, and that specimen metadata provides useful but unstable contextual cues. For anatomical segmentation, closed-set segmentation models remain weak on thin structures, and zero-shot open-vocabulary models often fail to distinguish insect parts from the body or background. Together, these results show that InSpect enables systematic evaluation of insect specimen understanding across cropped visual morphology, label-derived context, and fine-grained anatomical annotations.
Show more
CEO-Bench: Can Agents Play the Long Game?
Haozhe Chen ⋅ Karthik Narasimhan ⋅ Zhuang Liu
Language model agents are becoming proficient executors at isolated, short-horizon tasks such as software engineering and customer service. Yet real-world challenges require a combination of sophisticated skills that remain largely untested in agents: (1) navigating long horizons amid uncertainty; (2) acquiring information in noisy environments; (3) adapting to a changing world; (4) orchestrating multiple moving parts toward a coherent goal. We introduce CEO-Bench, which evaluates these capabilities together by simulating a representative real-world task: operating a startup for 500 days. Given diverse and realistic company management tools, business databases, and social media, an agent needs to design pricing strategies, allocate operating budgets, analyze business data, respond to unexpected competitor moves, and more. Our evaluation shows that most state-of-the-art models struggle to succeed in this environment, and only one model (GPT-5.5) finishes the simulation above its $1M starting balance. CEO-Bench reimagines the role of AI in the future, shifting from solving isolated tasks to driving sustained, adaptive progress over time. We open source full code and trajectory.
Show more
AgriManager: A Framework and Benchmark for LLM-RL Generalization in Agricultural Management
Chi Gui ⋅ Junwen Zheng ⋅ Ritwik Nigam ⋅ Qianlan Yang ⋅ Meagan Lang ⋅ Vardhan Dongre ⋅ Jeremiah Barr ⋅ Yu-Xiong Wang ⋅ Vikram Adve
Agricultural management is heterogeneous along two axes. The first is distributional shift within a fixed task interface: the same sensors, action menu, and objective, but with weather, planted crops, and prices varying across years and locations. The second is structural shift across interfaces: different farms expose different observed variables, available actions, and management objectives. We formalize the second axis in MDP terms as \emph{schema shift}: variation in the policy-facing schema $\Sigma = (\Sigma_O, \Sigma_A, \Sigma_R)$ between training and deployment. Fixed-interface neural policies are structurally inapplicable when $\Sigma_O$ or $\Sigma_A$ shifts and require retraining when $\Sigma_R$ shifts, while prompt-conditioned LLM policies re-read $\Sigma$ from the prompt of each new episode. Whether LLM-RL actually generalizes across both axes has not been systematically tested. We address this question with \emph{AgriManager}, the first multi-simulator LLM-RL framework for agricultural management, unifying the DSSAT, WOFOST, and Cycles crop simulators under prompt-based schema specification with joint multi-objective, multi-environment training. On this platform we design a three-tier generalization ladder, each tier grounded in a real-world deployment scenario: Tier 1 within-schema distributional shift (weather, crop, price), Tier 2 single-axis schema shift along $\Sigma_O / \Sigma_A / \Sigma_R$, Tier 3 compositional schema shift across simulators. Within fixed schemas, prompt-conditioned LLM-RL with explicit reasoning matches or exceeds fixed-interface NN-RL on all three settings, with reasoning becoming necessary when the reward shape penalizes input use. Single-axis schema shift imposes a different burden along each axis: information sufficiency for $\Sigma_O$, joint action-menu coverage for $\Sigma_A$, and objective coverage for $\Sigma_R$. Under compositional schema shift, a unified policy with marginal per-platform coverage and explicit reasoning matches or exceeds per-source specialists on all three callback targets. Without that coverage, pairwise zero-shot transfer is bounded by transition dynamics, which the prompt cannot describe and which we leave as an open task. Together, these results provide an initial empirical baseline for unified prompt-conditioned control across heterogeneous agricultural deployments.
Show more
LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training
Andreas Hochlehnert ⋅ Marianna Nezhurina ⋅ Mehdi Cherti ⋅ Andrej Radonjic ⋅ Thaddäus Wiedemer ⋅ Christoph Schuhmann ⋅ Romain Beaumont ⋅ Wieland Brendel ⋅ Bernhard Schölkopf ⋅ A. Sophia Koepke ⋅ Jenia Jitsev ⋅ Matthias Bethge
We present LAION-BVD (LAION - Big Video Dataset), a large-scale open video dataset for multimodal learning, containing _1.3B platform-specific video URLs_ collected from CommonCrawl. From these, we download 80M videos with a total duration of _10 million hours_. The dataset is designed for multimodal pre-training across video, audio, and image modalities. Using content-aware scene detection we extract clips for which we synthetically generate video and audio captions. Models trained on these data achieve competitive performance on standard video-text and audio-text benchmarks, with consistent improvements as training or model scale increases. Additionally, we explore video frames as a new source of image-text data by extracting scene-changing frames. These frames exhibit a visual distribution distinct from standard web image corpora, and models trained on this dataset achieve strong image-text retrieval performance. Overall, LAION-BVD significantly expands open access to multimodal videos at unprecedented scale and will be released to the research community for multimodal research
Show more
When Depth Lies: Benchmarking Vision-Language Models on Mirror-Induced RGB-D Ambiguity
HAO YIN ⋅ Tianchen Guo ⋅ Heming Du ⋅ Yan Ke ⋅ Yanbin Liu ⋅ Xin Yu
Vision-language models (VLMs) have achieved significant progress in scene understanding and spatial reasoning. However, existing evaluation benchmarks predominantly assume that all visible content corresponds directly to physical entities in the scene, leaving model performance in reflective scenarios largely unexamined. Reflective surfaces preserve object appearance while altering the mapping between appearance and physical location, causing VLMs to produce systematic reasoning errors in such scenes. In this work, we introduce (i) a grounded RGB-D dataset for mirror-aware spatial reasoning, and (ii) the first comprehensive and systematic evaluation of state-of-the-art VLMs in reflective scenarios. Specifically, we collect 1,011 RGB-D images from 525 real-world scenes, spanning diverse indoor and outdoor environments and six reflective surface types, under varying lighting conditions across both daytime and nighttime settings. All images are annotated with reflective surface region masks and fine-grained, polygon-level instance annotations for both directly observed and mirror-reflected objects, with depth maps serving as both geometric ground truth for annotation and optional model input. We curate a mirror-induced reasoning ambiguity benchmark, named \textbf{MIRA-Bench} (Mirror-Induced Reasoning Ambiguity Benchmark). MIRA-Bench comprises 2,955 unique QA pairs across eight sub-tasks organized into three cognitive levels: \mbox{Reflection-aware} Perception, Spatial Reasoning, and \mbox{Scene-level} \mbox{Decision-making}, with five object-referencing protocols yielding 14,140 QA samples. We evaluate state-of-the-art VLMs on MIRA-Bench and find that these models consistently underperform across tasks, suggesting that current VLMs do not yet understand mirrors. Common failure patterns include misclassifying whether objects are directly observed or seen through reflections, failing to identify that reflected and directly observed instances correspond to the same physical object, producing erroneous cross-space spatial inferences, and tending to adopt conservative default strategies in action-oriented decisions instead of reasoning about the true scene layout. We will release the dataset and evaluation code to support future research.
Show more
Nereus: A Large-Scale Underwater Dataset for Fine-Grained Attribute Understanding and Grounded Counting Perception
Jingtao Deng ⋅ Zihao Jin ⋅ Yuehe Chen ⋅ Feixiang He ⋅ Shuiwang Li ⋅ Dan Zeng
Underwater imagery records rich ecological information and supports applications such as science communication, fishery monitoring, ecological conservation, and oceanographic research. Beyond general scene understanding, these applications require models to recognize fine-grained biological traits and reason over spatially grounded evidence. However, existing underwater datasets often provide coarse category labels, generic captions, or total-count supervision, which limits their use for organism-level analysis and ecological reasoning. We introduce Nereus, a large-scale dataset for fine-grained underwater perception that jointly supports Fine-Grained Object Attribute Understanding and Grounded Counting Perception. Nereus contains 88K images and 2.68M question-answer pairs, including annotations for 136K marine organism objects across 2,690 species and 1,515 genera, as well as 65K point annotations and 67K grounded counting QA pairs. Experiments with representative multimodal models show that Nereus improves downstream adaptation over existing underwater-domain data and provides a foundation for future research on fine-grained marine perception.
Show more
PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
Mennatullah Siam
Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the referring expression without any temporal reasoning. To address this, we introduce novel motion-centric probing, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the ordering of motion. Consequently, we introduce MoCentric-Bench, a motion-centric benchmark designed to evaluate video MLLMs based on their ability to capture the interaction between motion and language, rather than relying primarily on static cues. We further establish strong single-image baselines that either match or surpass prior methods. Finally, we explore a simple motion-centric adaptation that provides state-of-the-art performance on our MoCentric-Bench. Code is available at \url{https://anonymous.4open.science/r/PixFoundation2MoCentricBenchFin/}.
Show more
Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation
Anya Ji ⋅ Abhijith Varma Mudunuri ⋅ David Chan ⋅ Alane Suhr
While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages, charts, or SVGs, it is still unclear VLMs are able to recover temporal dynamics when motion is present. To this end, we introduce the animation-to-code task, as well as a benchmark, \texttt{Animation2Code}, for evaluating temporal visual reasoning via reconstructing executable web animation code from videos. \texttt{Animation2Code} consists of 1,069 web animation videos with diverse visuals and motion patterns, augmented with corresponding HTML/CSS/JavaScript implementations. We further pair the dataset with several novel human-aligned metrics, which allow us to disentangle visual similarity from temporal alignment when comparing rendered animations to ground-truth samples. We benchmark current SOTA model performance on these new samples, and show that current models struggle with temporal consistency, even when achieving high appearance similarity, even in fine-tuning and iterative refinement scenarios. Our benchmark dashboard is available at: \url{https://anim2code-dashboard.vercel.app}.
Show more
COCOTree: A Dataset and Benchmark for Open Tree-Structured Visual Decomposition
Junhyub Lee ⋅ Seunghun Chae ⋅ Hyosu Kim
We formalize and enable the task of open tree decomposition, which segments an image into hierarchical trees of visual components with unconstrained granularity and flexibility. Specifically, we provide the foundation benchmark for this new paradigm with the following three key contributions. First, we overcome the prohibitively high cognitive and physical bottlenecks of manual annotation by developing a fully automated generation pipeline that synergizes the semantic reasoning of Large Vision-Language Models (LVLMs) with the precise geometric grounding of SAM 3. Second, leveraging this pipeline, we construct COCOTree, a massive-scale benchmark featuring over 21K images and 1.8M structural nodes. By embracing an open-vocabulary space of over 3.5K unique labels, it successfully captures the long-tail distribution of complex physical assemblies. Notably, rigorous human evaluation confirms our generated annotations demonstrate strong alignment with human structural judgment. Third, we establish a standardized evaluation protocol by proposing the Open Tree Quality (OTQ) metric, which jointly assesses mask precision, label accuracy, and structural consistency. We release our dataset and benchmark code at https://github.com/melonkick3090/COCOTree.
Show more
CIPHERGRID Benchmark: From Multimodal Rule Inference to Sequential Action
Christopher Curtis ⋅ Victor Fragoso ⋅ Saiph Savage
We introduce CIPHERGRID, a rule-inference and path-finding benchmark. In each task, models are given example grid-world images paired with encoded descriptions and encoded solutions. From this evidence, models then receive a new encoded grid-world to solve. To answer correctly, models must infer which symbols denote tiles, actions, row boundaries, and game mechanics. They must then decode the new world, and produce an encoded path from the start cell to the target cell. The benchmark is designed so that each component is individually solvable: rule inference is calibrated against human performance, and path-finding is solver-verifiable via exact shortest-path search. Although the underlying skills are individually solvable and verifiable, CIPHERGRID tests their interaction: whether models can infer an unfamiliar symbolic system, and carry it through rule application, and planning. CIPHERGRID remains difficult for current frontier models, which only achieve between 18.74\% and 41.89\% accuracy. CIPHERGRID offers a controlled test of whether models can coordinate multimodal grounding, rule inference, and sequential planning under unknown symbolic relationships.
Show more
HAPACT: A Benchmark For Human-Centric Physical Impact Localization in Movies
Youngrae Kim ⋅ Yejin Jang ⋅ Eunho Kim ⋅ Youjin Sung ⋅ Kun-Woo Song ⋅ Sangpil Kim ⋅ Sang Ho Yoon
Physical reasoning about humans in videos has largely focused on everyday interactions, where contact, pressure, pose, and motion support embodied perception and contact-aware applications. Still, short and forceful properties like impact events remain relatively underexplored, despite their relevance to broader understanding of how abrupt physical events affect the human body. To fill this gap, we introduce human-centric impact localization, a new task that localizes target physical impacts temporally within a video and spatially on the human body. We further present HAPACT, a HumAn-centric imPACT dataset and benchmark built from movie clips, containing over 2.8K video-query pairs with more than 20K fine-grained human-annotated frames. We built HAPACT around viewer-grounded annotation, text-conditioned localization, and supervision at frame-level temporal and SMPL-X-based spatial granularity. To demonstrate the effectiveness of our benchmark, we conduct extensive evaluations of existing methods, employing video moment retrieval models for temporal localization and human-scene contact models for spatial localization. For the spatial task, we propose a text-conditioned baseline by reformulating localization as query-guided body-region selection. Benchmark results show that physical impact localization remains challenging for existing methods, motivating HAPACT as a dedicated benchmark to support physical impact reasoning in videos.
Show more
MLLM-Edit: Benchmarking Image Forgery Detection and Localization under MLLM-based Editing
Zeqin Yu ⋅ Ye Tian ⋅ Jian Zhang ⋅ Jiangqun Ni ⋅ Lanyun Zhu ⋅ Alex Chi Chung Kot ⋅ Xudong Jiang
Multimodal large language models (MLLMs) enable prompt-driven image editing, introducing a new challenge for image forgery detection and localization (IFDL). Unlike traditional or mask-guided editing, MLLM-based editing may re-synthesize both tampered and non-tampered regions, weakening the correspondence between tampering regions and pixel-level forensic discrepancies. To study this emerging paradigm, we introduce MLLM-Edit, a large-scale benchmark containing approximately 250k tampered images with pixel-level tampering annotations. MLLM-Edit is constructed through a structured pipeline that combines diverse real-image collection, human-like prompt design, multi-source MLLM-based editing, and semi-automatic mask annotation. Specifically, we collect natural and text images from 30 sources, formulate prompts with target content, tampering type, and consistency constraints, generate forgeries using 18 representative open-source and closed-source MLLM-based editors, and obtain pixel-level tampering masks through a semi-automatic annotation pipeline. We further establish four evaluation protocols covering traditional editing, mask-guided editing, fully synthetic generation, and MLLM-based editing. Experiments show that existing IFDL methods degrade substantially on MLLM-based forgeries, especially in pixel-level localization, highlighting the need for dedicated benchmarks and methods. A demo subset of MLLM-Edit is available at the \href{https://kaggle.com/datasets/8441a4e0594aa02aa7a59ee3eead454410d34716d14423d56a56c7bbbac80655}{project page}.
Show more
EmoPhone: A Multi-Wave Dataset for In-the-Wild Mobile and Wearable Affect Sensing
Panyu Zhang ⋅ minseo Park ⋅ Soowon Kang ⋅ Ismatzoda Tomiris ⋅ Azizbek Mustafakulov ⋅ Otabek Najimov ⋅ Woohyeok Choi ⋅ JUMABEK Alikhanov ⋅ Surjya Ghosh ⋅ Uichin Lee
We introduce a three-wave, in-the-wild multimodal dataset for affect sensing that integrates smartphone sensing, wearable sensing, and dense experience-sampling-method (ESM) labels collected annually from 2020 to 2022. The dataset supports moment-level affect modeling through a shared dimensional label core across all waves, with additional affective descriptors available in the third wave (D-3). We describe the resource in terms of study design, temporal density of in-situ labels, and sensing and label coverage across waves. To support evaluation within this resource, we define an initial three-setting benchmark spanning temporal prediction from within-user history, within-wave cross-user generalization, and cross-wave generalization in which each wave is treated as a separate dataset. Our benchmark results show that the strongest method family depends on the evaluation setting: supervised baselines perform best in the temporal setting, unsupervised domain adaptation is strongest overall in the within-wave cross-user setting, and domain generalization shows the strongest overall cross-wave performance, although its margin over strong baselines is modest. These findings indicate that robust mobile affective computing is constrained not only by label availability but also by substantial participant-level variability and realistic cross-wave differences inherent in longitudinal in-situ deployments.
Show more
XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
Dasol Choi ⋅ Eugenia Kim ⋅ Jaewon Noh ⋅ Sang Seo ⋅ Eunmi Kim ⋅ Myunggyo Oh ⋅ Yunjin Park ⋅ Brigitta J Kartono ⋅ Josef Pichlmeier ⋅ Helena Berndt ⋅ Sai Krishna Mendu ⋅ Glenn J Tungka ⋅ Özlem Gökçe ⋅ Suresh Gehlot ⋅ Katherine Pratt ⋅ Amanda Minnich ⋅ Haon Park
Current LLM safety benchmarks are predominantly English-centric and often rely on translation, failing to capture country-specific harms. Moreover, they rarely evaluate a model's ability to detect culturally embedded sensitivities as distinct from universal harms. We introduce XL-SafetyBench, a suite of 5,500 test cases across 10 country-language pairs, comprising a Jailbreak Benchmark of country-grounded adversarial prompts and a Cultural Benchmark where local sensitivities are embedded within innocuous requests. Each item is constructed via a multi-stage pipeline that combines LLM-assisted discovery, automated validation gates, and dual independent native-speaker annotators per country. We report Attack Success Rate (ASR) alongside Neutral-Safe Rate (NSR) to separate principled refusal from non-engagement, and introduce the Cultural Sensitivity Rate (CSR) for recognizing embedded sensitivities. Evaluating 10 frontier and 27 local LLMs reveals two key findings. First, strong jailbreak robustness does not always come with strong cultural awareness, showing why the two should be evaluated separately. Second, local models exhibit a strong ASR–NSR trade-off (r = -0.81), suggesting that low attack success in several local models reflects non-engagement rather than safe refusal. XL-SafetyBench enables more nuanced, cross-cultural safety evaluation in the multilingual era.
Show more
SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory
Huacan Chai ⋅ YUKAI WANG ⋅ Yingxuan Yang ⋅ Dan Peng ⋅ Yuanyi Song ⋅ Zhihui Fu ⋅ Jun Wang ⋅ Weiwen Liu ⋅ Jianghao Lin ⋅ Weinan Zhang
Existing benchmarks for multimodal memory reasoning largely evaluate systems within pre-assembled contexts, but under-evaluate whether agents can use evidence distributed across independently originated sources. We argue that source-distributed memory composition is an important and under-examined bottleneck in multimodal agent memory, especially when relevant evidence is fragmented across heterogeneous artifacts such as conversations, profiles, screenshots, tables, images, and documents. To address this gap, we introduce Source-distributed Multimodal Memory Benchmark, which measures whether agents can retrieve, align, and compose multimodal evidence scattered across multiple sources rather than reason within a single curated context. SMMBench evaluates four core capabilities: (1) cross-source multimodal reasoning; (2) conflict resolution; (3) preference reasoning; (4) memory-grounded action prediction. The benchmark contains $1,877$ samples grounded in $264$ sources. Experiments on representative memory-style and retrieval-based baselines show that current systems still struggle on these capabilities, positioning source-distributed multimodal memory as an important and still under-evaluated challenge for multimodal agents. Our data and code are available at https://anonymous.4open.science/r/NeurIPSBenchmarksW8kP6gV/.
Show more
AudioAgentBench: Evaluating Multi-Turn Voice Agents on Real-World Tasks
Gardenia Liu ⋅ Yi-Hao Peng ⋅ Kamryn Ohly ⋅ Oliver Johansson ⋅ Hileamlak Yitayew ⋅ Jaden Zhang ⋅ Grace Li
Speech-to-speech (S2S) language models are increasingly deployed as voice agents that schedule appointments, take grocery orders, and plan events. Reliable deployment in such settings requires accurate listening, grounded tool use, state tracking, knowledge-base access, and recovery from user corrections across many turns. Existing voice agent benchmarks often score final database state or per-call tool accuracy, which can obscure the first failure point when an early mistake cascades through the rest of a conversation. We introduce AudioAgentBench, a fixed-trace benchmark suite with six multi-turn voice-agent tasks, 221 turns of pre-recorded audio, golden tool calls, tool schemas, knowledge bases, and matched TTS and human-recorded variants. The fixed audio traces make turn-level expectations reproducible and support a diagnostic scoring protocol that attributes failures across tool use, state tracking, ambiguity handling, knowledge grounding, and instruction following. An LLM judge applies the protocol to each turn by comparing model outputs against golden tool calls, task state, and task-specific knowledge bases, while using cross-turn realignment, conditional penalty absorption, and category-aware dimension gating to avoid double-counting cascading errors. We evaluate seven S2S models across 420 continuous session runs. The top model averages an 80.3\% pass rate, but the top four models have overlapping 95\% confidence intervals, and no model leads on more than two of six tasks. On a 408-run turn-level diagnostic subset with 14,907 judged turns, tool-use accuracy on turns where a tool call was expected peaks at only 61.4\% and drops to 32.3\% on a confusable-name scheduling task. Errors cascade by up to 3.15× in stateful tasks, and tool-use accuracy on a 30-turn grocery task falls by 40.5 percentage points from the first half to the last half. Scoring only turns where a tool call was expected reduces apparent tool-use accuracy by 22–44 percentage points across models, which reveals a larger gap between strong and weak systems than aggregate task success suggests. AudioAgentBench surfaces action, state, and recovery failures that final success metrics obscure, providing a finer-grained diagnostic benchmark for production-oriented voice agents.
Show more
AI Construct Lexis: An Ontology of the Hidden Assumptions in AI Evaluation
Olawale Salaudeen ⋅ Florian E. Dorner ⋅ Tom Sühr ⋅ Sang Truong ⋅ Haoran Zhang ⋅ Patrick Bissett ⋅ James Fox ⋅ Marzyeh Ghassemi ⋅ Tom Hartvigsen ⋅ Peter Hase ⋅ Augustin Kelava ⋅ Suhas Mahesh ⋅ Margaret Mitchell ⋅ Jamelle Watson-Daniels ⋅ Sanmi Koyejo ⋅ Angelina Wang
There are now thousands of measurement instruments used to evaluate language models, and the resulting scores are routinely used to support consequential claims about capabilities and risks. Yet, the evidence linking benchmark scores to constructs (i.e., abstract concepts such as reasoning ability or safety) is often implicit, incomplete, or absent. We introduce AI Construct Lexis, a versioned ontology and interactive artifact that extracts and organizes relationships among constructs, behaviors, and measurement instruments as claimed in the literature for LLM evaluation. Drawing on over 8,337 published papers from selected top AI venues, we use language models to extract constructs, definitions, behavioral indicators, and measurement instruments, yielding 977 distinct constructs, 1,694 measurement instruments, 830 behavioral indicators, and 5,900 relationships. The Lexis captures relationships claimed in the published literature, not necessarily validated relationships; valid score interpretation still requires empirical evidence that the instruments capture the constructs they are used to support. By exposing the field’s implicit measurement assumptions, the Lexis enables their inspection, testing, falsification, and refinement into evidence-based relationships. On expert-annotated samples, the extraction pipeline achieves high agreement on extracted concepts while revealing substantial ambiguity in the boundaries of constructs and behaviors. By making these relationships explicit, AI Construct Lexis provides infrastructure for a more rigorous and cumulative science of AI evaluation.
Show more
Perception, Not Reasoning, Limits Visual Theory of Mind
Mohamed Rayan Barhdadi ⋅ Syed Talal Wasim ⋅ Jürgen Gall ⋅ Erchin Serpedin ⋅ HASAN KURBAN
Vision-language models pass text-based false-belief probes at near-human levels but drop by twenty or more percentage points the moment the same problem is conveyed visually. The consensus diagnosis is conceptual, prescribing new belief-reasoning modules. We argue the bottleneck is perceptual: not what the model cannot reason about, but what it cannot see. We propose a five-condition dissociation protocol that holds the reasoning task constant while varying perceptual fidelity from visual only to text only, and quantify the result with the Perceptual Contribution Ratio, a dose-response metric measuring how much of the visual-to-text gap a scaffold of given fidelity closes. Instantiated on abstract 2D panels in the Heider-Simmel tradition and naturalistic 3D game scenes from MuMA-ToM and MindPower across seven VLMs and roughly $147{,}000$ API calls, ground-truth scaffolds close the gap on every regime, matching or exceeding the text-only ceiling on abstract panels and MuMA-ToM. Automatic parsers, including frontier VLMs as zero-shot scene describers, fail. The dominant visual-only error mode is \emph{reality bias} ($60\%$ of false-belief failures): models report where an object currently is rather than where an absent agent believes it to be, the signature of a front-end that fails to encode who was present when. The route to better visual social reasoning runs through perception, not new reasoning modules.
Show more
On the Selectivity of Generative Models in Structure-Based Drug Design
Ella Miray Rajaonson ⋅ Jungyoon Lee ⋅ William Chau ⋅ Alan Aspuru-Guzik ⋅ Benjamin Sanchez-Lengeling ⋅ Dominique Beaini ⋅ Kirill Neklyudov ⋅ Marta Skreta
Designing small-molecule drugs that selectively bind to a target protein while avoiding unintended interactions with other proteins is essential for reducing side effects that contribute substantially to clinical attrition. Although structure-based drug design (SBDD) has advanced rapidly with deep learning, most generative models still optimize binding to a single target and disregard selectivity due to limited training data and unknown off-target interactions. Motivated by this gap, we introduce SelectBench, a comprehensive benchmark assessing the selectivity of generative SBDD models. SelectBench provides three curated evaluation tasks reflecting different real-world drug discovery scenarios: literature-based case studies, safety screening panels used in pharmaceutical development, and large-scale structure-based evaluation sets. We further introduce a suite of metrics to quantify selectivity and conduct a systematic evaluation of nine representative SBDD models, including methods explicitly trained for selectivity and widely-used inference-time guidance techniques. Our results show that even selectivity-trained models achieve only modest off-target discrimination and inference-time guidance provides limited improvements, underscoring the need for method improvement. SelectBench provides both a characterization of where current generative SBDD stands on selectivity and an open, extensible platform to accelerate progress.
Show more
Information Parity for Code: The Scope of Transfer in Multilingual Code Models
Alexander Tsvetkov ⋅ Alon Kipnis
LLMs exhibit uneven performance across programming languages, yet benchmark gaps often reflect narrow task regimes and evaluation-specific confounds. This makes it hard to tell whether poor performance reflects a broader weakness with a given language, a broader difficulty on a given task, or a limitation specific to that task-language combination. We adapt Information Parity, a measure of relative cross-language representational efficiency developed for natural language, to code. We define Code Information Parity (Code IP) on a controlled parallel corpus of 22 algorithms across 20 languages, audited to preserve core semantics while minimizing boilerplate and excluding library shortcuts. The results are regime-specific: Code IP correlates strongly with multilingual benchmark performance on short-horizon generation ($\rho$ from $-0.55$ to $-0.81$), remains negative though moderated on execution prediction ($\rho=-0.429$), and disappears on whole-program synthesis ($\rho=+0.105$). Rather than a universal metric of multilingual coding ability, Code IP isolates a reliable cross-language signal on short-horizon generation and delineates where that signal no longer separates cleanly from broader task and evaluation effects.
Show more
OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes
Naiyu Fang ⋅ Zhongjin Luo ⋅ Yuxin Mo ⋅ Siyuan Huang ⋅ Jianbo Liu ⋅ Yufei Liu ⋅ Zheyuan Zhou ⋅ Chengkai Jin ⋅ Xiaogang Wang ⋅ Hongsheng Li
Dexterous grasping is the foundational primitive in embodied AI, demanding massive data to train robust models. As real-world data collection is expensive, simulation has become the mainstream paradigm. Yet, while cluttered scenes best reflect real-world applications, learning to grasp within them is bottlenecked by a critical scarcity of large-scale data. To resolve this, we curate high-quality 3D objects and supporting bases, proposing a scalable seed-and-filter strategy that bypasses sluggish scene-level optimization. This yields an unprecedented benchmark comprising over 2.6 million scenes and 0.4B scene-specific grasp ground truths, featuring diverse realistic layouts paired with rich semantic and geometric observations. Furthermore, we introduce the OmniDex model to overcome the grasp multimodality and last-millimeter precision errors plaguing current generative models. By coupling Soft Winner-Takes-All learning with human-inspired physical constraints during training, and utilizing physics-driven ranking, our approach achieves robust dexterous grasping without the latency of post-optimization. Experimental results show that OmniDex model achieves state-of-the-art performance and strong generalization across diverse scenes, views, and unseen objects.
Show more
PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images?
Zongyi Chen ⋅ Yu Liang ⋅ Jie Lin ⋅ Liansheng Wang
Multimodal large language models (MLLMs) are increasingly used to analyze pathology images. However, dominant multimodal benchmarks in pathology mainly score final diagnostic answers, captions, or reports. These evaluations provide limited insight into whether a model understands the multiscale visual content needed for pathology reasoning and decision-making. We introduce PathView-Bench, a vision-anchored benchmark for fine-grained and multiscale visual understanding in computational pathology. Built from $23$ public pathology imaging datasets with human-supervised labels and spatial annotations, PathView-Bench evaluates MLLM understanding in two fields of view: Region-FOV for high-resolution local regions and Slide-FOV for macro whole-slide views. By converting raw annotations into deterministic task targets, PathView-Bench enables programmatic scoring of region localization, visual recognition, quantity estimation, spatial reasoning, and insufficient-context judgment. The benchmark contains $14$ VQA-style tasks, $61,673$ images, and $308,070$ samples across $28$ organs and $7,253,526$ annotations. Evaluating $18$ representative general-purpose, medical-domain, and pathology-oriented MLLMs, we observe substantial limitations even in advanced models on fine-grained visual tasks across multiscale pathology images. PathView-Bench provides a reproducible basis for developing and evaluating pathology MLLMs with explicit multiscale visual understanding.
Show more
MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
Jonathan Steinberg ⋅ Oren Gal
Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets. The challenge is structural: existing safety alignment evaluates overt requests in isolation, leaving models blind to malicious end-states that emerge from sequenced compliance with innocuous-looking requests. We introduce MOSAIC-Bench (Malicious Objectives Sequenced As Innocuous Compliance), a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates (10 web-application substrates, 31 CWE classes, 5 programming languages) that treats both exploit ground truth and downstream reviewer protocol as first-class evaluation axes. On this benchmark, nine production coding agents from Anthropic, OpenAI, Google, Moonshot, Zhipu, and Minimax compose innocuous tickets at 53–86% end-to-end ASR with only two refusals across all staged runs. In a matched direct-prompt experiment over four frontier Claude/Codex agents, vulnerable-output rates fall to 0–20.4%: Claude primarily refuses, while Codex primarily hardens rather than emitting the vulnerable implementation — ticket staging silences both defense modes simultaneously. Downstream, code reviewer agents approve 25.8% of these confirmed-vulnerable cumulative diffs as routine PRs, and a full-context implementation protocol closes only ∼50% of the staged/direct gap, ruling out context fragmentation as the sole explanation. As a deployable but non-adaptive mitigation, reframing the reviewer as an adversarial pentester reduces evasion across the evaluated reviewer subset; pentester-framed evasion ranges from 3.0% to 17.6%, and an open-weight Gemma-4-E4B-it reviewer under this framing detects 88.4% of attacks on the dataset with a 4.6% false-positive rate measured on 608 real-world GitHub PRs. We publicly release our dataset at https://huggingface.co/datasets/MosaicBenchmark/mosaic-bench, and a verifiable and adaptable evaluation framework at https://github.com/mosaic-benchmark/mosaic-benchmark.
Show more
OgBench: A Framework for Evaluating Graph Neural Networks on Omics Data
Louisa Cornelis ⋅ Johan Mathe ⋅ Louis Van Langendonck ⋅ Guillermo Bernárdez ⋅ Nina Miolane
Graph Neural Networks (GNNs) have become the dominant framework for inductive graph-level learning. Yet most benchmarks focus on the regime $n \gg p$, where the number of graphs $n$ greatly exceeds the number of nodes per graph $p$. This overlooks biological domains such as omics, which operate in the opposite $n \ll p$ regime, characterized by large graphs of genes, transcripts, or proteins across few patient samples. This raises the question: \textit{how do GNNs perform in this low-sample, high-node omics setting?} We introduce \texttt{OgBench} (Omics-Graph Bench), the first benchmarking platform for graph-level prediction in the $n \ll p$ regime characteristic of omics data. We provide a standardized, end-to-end modular infrastructure from raw omics data to families of featured graphs with varied structural properties. We benchmark classical GNNs, as well as GNNs designed for large graphs and omics applications, alongside MLPs and machine learning baselines to establish reference performances. Our results show that widely used GNNs often do not outperform simple MLPs and classical baselines. These findings challenge the prevailing assumption that graph structure inherently adds value in this domain, fostering a critical reassessment of current learning paradigms. Ultimately, by exposing these limitations, OgBench provides the open-source ecosystem necessary for the community to develop and validate novel architectures explicitly tailored for biological graphs. The code is available at \url{https://anonymous.4open.science/r/ogbench-C1A9/README.md}.
Show more
SenseBench: A Benchmark for Remote Sensing Low-Level Visual Perception and Description in Large Vision-Language Models
Chen Zhong ⋅ Xiao An ⋅ Jiaxing Sun ⋅ Zihan Gui ⋅ Guangyi Yang ⋅ Wei He
Low-level visual perception underpins reliable remote sensing (RS) image analysis, yet current image quality assessment (IQA) methods output uninterpretable scalar scores rather than characterizing physics-driven RS degradations, deviating markedly from the diagnostic needs of RS experts. While Vision-Language Models (VLMs) present a compelling alternative by delivering language-grounded IQA, their visual priors are heavily biased toward ground-level natural images. Consequently, whether VLMs can overcome this domain gap to perceive and articulate RS artifacts remains entirely unverified. To bridge this gap, we propose SenseBench, the first dedicated diagnostic benchmark for RS low-level visual perception and Description. Driven by a physics-based hierarchical taxonomy that unifies both no-reference and reference-based paradigms, SenseBench features over 10K meticulously curated instances across 6 major and 22 fine-grained RS degradation categories. Specifically, two complementary protocols are designed for evaluation: objective low-level visual perception and subjective diagnostic description. Comprehensive evaluation of 23 state-of-the-art VLMs reveals not only skewed domain priors and multi-distortion collapse, but also fluency illusion and a perception-description inversion effect. We hope SenseBench provides a robust evaluation testbed and high-quality diagnostic data to advance the development of RS VLMs in low-level perception. Code and datasets are available at [here](https://github.com/Zhong-Chenchen/SenseBench).
Show more
ECLIPSE: A Spacecraft Rendezvous Trajectories Dataset with Controlled In-Orbit Lighting Conditions
Nidhal Eddine Chenni ⋅ Arunkumar Rathinam ⋅ Abid Ali ⋅ Djamila Aouada
Deep learning has become the dominant paradigm for vision-based spacecraft pose estimation, where models are predominantly trained on synthetic data due to the prohibitive cost of acquiring real in-orbit imagery. When deployed, these models suffer significant performance degradation due to the sim-to-real domain gap, which existing benchmarks address by pairing synthetic training data with Hardware-in-the-Loop (HIL) real imagery. Yet a critical factor remains consistently overlooked: in real orbital scenarios, illumination varies with the spacecraft's position along its orbit, inducing severe appearance shifts that existing benchmarks neither control nor study. We introduce ECLIPSE, the first benchmark designed to isolate illumination as an independent experimental variable. ECLIPSE pairs a large-scale photorealistic synthetic training set with a fully labeled HIL evaluation set of 14 approach trajectories, each replayed identically under 4 controlled lighting conditions, enabling rigorous evaluation of illumination-robust and domain-adaptation methods. Through this controlled design, our experiments confirm that lighting is the dominant driver of the sim-to-real performance gap in spacecraft pose estimation.
Show more
When Do Cosine Prototypes Mislead? A Whitening-Aware Benchmark for Frozen-Feature Image Classification
Jian Ding ⋅ Su Yang
Cosine nearest-class-mean (NCM) evaluation is a widely used default for closed-set frozen-feature image classification because it is deterministic, train-free, and easy to reproduce. This convenience hides a metric assumption: raw angular geometry alone suffices for benchmark conclusions. We show that the assumption changes model-selection outcomes, not just absolute accuracies: across nine datasets and six frozen backbones, cosine-only reporting changes which backbone ranks first on 2 of 9 datasets and incurs 6.71 percentage points of regret relative to a best-evaluator oracle; on fixed DINOv2-B features, the underlying cosine-to-LDA gap reaches 37.63 points on Aircraft. Motivated by the low-variance spectral directions where class-discriminative signal often appears in such high-gap cases, we call this evaluator dependence the Spectral Decision Gap (SDG). SDG quantifies the gap between raw cosine and train-validated whitening-aware reference evaluators, flagging when cosine-only reporting changes benchmark conclusions. A leave-one-dataset predictive protocol reaches Pearson $r=0.752$ for best-whitening disagreement and incurs only 0.56pp routing regret at a 5pp threshold. This paper provides a compact audit protocol and artifact suite for detecting, predicting, and reporting when cosine prototype conclusions are reliable in this setting. Code and artifacts are available anonymously at https://anonymous.4open.science/r/sdg-ed/.
Show more
PhysEval.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models
Emma Kasteleyn ⋅ Timo Maier ⋅ Axel Lauer ⋅ Veronika Eyring ⋅ Pierre Gentine ⋅ Ana Lucic
Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs required for traditional physics-based methods. However, they are primarily (1) data-driven and (2) evaluated using pixel-wide error metrics (e.g., RMSE), so there are no guarantees that their forecasts are physically consistent. We introduce \textbf{PhysEval.Weather}, an evaluation framework that assesses the physical realism of MLWP models across three types of metrics: conservation, spectral, and dynamical. By quantifying physical realism, this tool guides the development of physics-informed architectures and helps evaluate whether MLWP models are reliable for operational use.
Show more
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
Zhaolu Kang ⋅ Yidi Wang ⋅ Yile Li ⋅ Siqi Zeng ⋅ Yingjie He ⋅ richeng xuan ⋅ Zhichao Hu
Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current multimodal benchmarks mostly test perception, recognition, or reasoning over already determinate inputs, leaving unclear whether omni-modal large language models can resolve ambiguity distributed across text, vision, and audio. We introduce TRIAD, a benchmark for omni-modal ambiguity resolution. Each item is a text--image--audio triplet paired with a question and a set of answer options, and is constructed so that the full triplet determines a unique answer while removing any one modality makes the set of consistent options non-unique. TRIAD contains $327$ hand-written queries in $86$ scene groups and covers $18$ ambiguity categories across the three modalities. Evaluating $14$ omni-modal systems reveals that strong item-level performance does not translate into robust scene-level disambiguation: the best model still trails humans by over $20$ points in group accuracy. Removing answer options widens the gap further, driving model group accuracy to the single digits while humans remain robust. Leave-one-out and transcript/description counterfactuals further show that current systems often underuse audio cues when text and image are present. TRIAD reframes omni-modal evaluation around disambiguation rather than recognition, and provides a taxonomy, benchmark, and diagnostic protocol for measuring this capability.
Show more
MultiTalk: Scaling Full-Duplex Speech Models to Long, Multi-Party, Bilingual Conversation
Ke Wang ⋅ Houxing Ren ⋅ Zimu Lu ⋅ Yunqiao Yang ⋅ ZHUOFAN ZONG ⋅ Mingjie Zhan ⋅ Hongsheng Li
End-to-end full-duplex speech models have brought open-source machine conversation close to human fluency, yet existing systems fall short of real-world deployment in two entangled respects: long-context conversational robustness and multi-party interaction capability. Realistic settings, including meetings, group lessons, family dinners, and social-robot reception, are inherently long-horizon and multi-party at the same time, requiring a single model to perceive, attribute, contextualize, and respond across multiple speakers over extended durations. Progress along these axes is bottlenecked by both data and evaluation. On the data side, open multi-party conversational speech corpora total only a few hundred hours and are not designed for codec-frame-level full-duplex modeling. On the evaluation side, existing long-audio benchmarks focus on passive listening, while existing speech-to-speech benchmarks remain dyadic and short. In this work, we extend the Moshi paradigm along long-horizon and multi-party axes simultaneously, in both English and Chinese, with three contributions. First, we release an open data engine and a 57.6 k-hour corpus for long, multi-party, bilingual full-duplex dialogue. The engine produces parallel-stream audio with controllable length, participant count, conversational dynamics (turn-taking, overlap, backchannels, interruption, addressee shifts, long-range co-reference), and language (English, Chinese, and intra-sentential code-switching), exceeding all prior open multi-party conversational speech corpora by more than an order of magnitude. Second, we introduce MultiTalkBench, the first benchmark to jointly evaluate long-form full-duplex dialogue with an average duration of 32.6 minutes, multi-party, and bilingual full-duplex dialogue, with explicit probes for long-range entity tracking, topic coherence, and addressee selection. Third, we train a bilingual Moshi-style full-duplex model on the released corpus that sustains coherent multi-party English-Chinese conversation over extended durations, substantially outperforming open-source baselines including Moshi, MiniCPM-o-4.5, and Qwen3-Omni-30B-A3B-Instruct.
Show more
DiagnosticIQ: A Benchmark for LLM-Based Industrial Maintenance Action Recommendation from Symbolic Rules
Devin De Silva ⋅ Dhaval Patel ⋅ Christodoulos Constantinides ⋅ Shuxin Lin ⋅ Nianjun Zhou ⋅ Dharmashankar Subramanian ⋅ Paul J Adams ⋅ Sal Rosato ⋅ Nicolas Constantinides ⋅ Deborah L McGuinness ⋅ Jayant Kalagnanam
Monitoring complex industrial assets relies on engineer-authored symbolic rules that trigger on sensor conditions and prompt technicians to perform corrective actions. The bottleneck is not detection but response: translating rules into maintenance steps requires asset-specific knowledge gained through years of practice. We investigate whether LLMs can serve as decision support for this rule-to-action step and introduce DiagnosticIQ, a benchmark of 6690 expert-validated multiple-choice questions from 118 rule-action pairs across 16 asset types. We contribute (i) a symbolic-to-MCQA pipeline normalizing rules to Disjunctive Normal Form with embedding-based distractor sampling, (ii) five variants probing distinct failure modes (Pro, Pert, Verbose, Aug, Rationale), and (iii) a benchmark of 29 LLMs and 4 embedding baselines. A human evaluation (9 practitioners, mean 45.0\%) confirms DiagnosticIQ requires specialist knowledge beyond operational experience. Three findings stand out. The frontier has closed: the top three LLMs lie within one Macro point, with Bradley-Terry Elo placing claude-opus-4-6 30 points above the next model. Yet DiagnosticIQ-Pro exposes brittleness, with every model losing 13-60\% relative accuracy under distractor expansion. DiagnosticIQ-Aug exposes pattern-matching: under condition inversion, frontier models still select the original answer 49-63\% of the time. The deployment bottleneck is not capability but calibration: frontier models handle template-style fault detection but break under structural perturbation.
Show more
When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Tom Burns
Standard language model evaluation assigns scores to single predicted answers, rewarding high-confidence responses regardless of how residual probability mass is distributed over alternative options. This creates a systematic pressure toward overconfident guessing: under accuracy-based schemes, a model maximises its expected score by always committing to an answer rather than abstaining, even when its uncertainty is high. While penalty-based approaches partially address this by raising the confidence threshold for strategic guessing, they still treat all sub-threshold responses identically, ignoring a fundamental distinction in how models can express uncertainty - for example between hedging toward incorrect answers versus hedging toward "I don't know" responses. We introduce a novel evaluation metric to solve this problem of not considering a model's entire probability distribution over answer choices. Our metric naturally distinguishes between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range. Through theoretical analysis and illustrative examples, we demonstrate our metric offers a more nuanced and aligned evaluation paradigm that incentivises models to express genuine uncertainty rather than guessing. We then adapt 12 existing evaluation benchmarks to our metric's variants and measure performance on six language models, showing that for half of the tested benchmarks scores are negative across all tested models, indicating significant tendencies towards hallucination.
Show more
SH$^2$: A Mathematician-Curated benchmark for Assessing Research-level Math Capabilities of LLMs
Guijin Son ⋅ Seungone Kim ⋅ Catherine Arnett ⋅ Hyunwoo Ko ⋅ Hyein Lee ⋅ Hyeonah Kang ⋅ JIANG LONGXI ⋅ JIN YUN ⋅ JungYup Lee ⋅ Kyungmin Lee ⋅ Sam Y Kim ⋅ Sang Park ⋅ Seunghyeok Hong ⋅ SeungJae Lee ⋅ Seungyeop Yi ⋅ Sunhye Bok ⋅ Yonghoon Ji ⋅ Youngtaek Kim ⋅ Hanearl Jung ⋅ Akari Asai ⋅ Graham Neubig ⋅ Sean Welleck ⋅ Youngjae Yu ⋅ Akshelin R ⋅ Alexander B Ivanov ⋅ Muhammadjon Boboev ⋅ Christian Stump ⋅ Dmitrii Karp ⋅ Dohyun Kwon ⋅ DoYong Kwon ⋅ Duk-Soon Oh ⋅ Giovanni Resta ⋅ Greta Panova ⋅ Huiyun Noh ⋅ Hyungryul Baik ⋅ Hyungsun Bae ⋅ Jeewon Kim ⋅ Ji E Lee ⋅ Jiaqi Liu ⋅ Jieui Kang ⋅ Jimin Kim ⋅ Jon-Lark Kim ⋅ Junseo Yoon ⋅ Junwoo Jo ⋅ Kibeom Kim ⋅ Kiwoon Kwon ⋅ Mario Kummer ⋅ Max Mercer ⋅ Minjun Kim ⋅ Nahyun Lee ⋅ Ng Ze-An ⋅ Rafał M Łochowski ⋅ Raphaël Lachièze-Rey ⋅ Ruichen Zhang ⋅ Sejin Park ⋅ Seonguk Seo ⋅ Shin jaehoon ⋅ Taewoong Eom ⋅ Yeachan Park ⋅ Yongseok Jang ⋅ Youchan Oh ⋅ Zhaoyang Wang ⋅ Zoltán Kovács ⋅ Chae Young Han ⋅ Shinae Shin ⋅ Sunyoung SHIN
Mathematical reasoning benchmarks for large language models face three persistent limitations. First, datasets scraped from public competitions are vulnerable to training-data contamination and saturate quickly. Second, most existing benchmarks cover only part of the olympiad-to-research spectrum. Third, benchmarks that withhold problems indefinitely to mitigate leakage trade away transparency and reproducibility. We introduce \gls{soohak}, a benchmark of 1{,}141 newly authored problems by 86 mathematicians designed to address all three. \gls{soohak} consists of three difficulty splits, constructed adversarially against panels of small, mid-size, and frontier baseline LLMs, spanning olympiad-style problem solving through research-adjacent material; a separate 99-item Refusal split probes recognition of ill-posed prompts. The dataset is temporarily embargoed and evaluated by request, with a committed public release in late 2026. On \splitthree{}, Gemini-3-Pro, GPT-5, and Claude-Opus-4.5 reach Avg@3 of 30.39\%, 26.37\%, and 10.39\%, respectively; the best Pass@3 is only 44.12\%, and 129/340 problems remain unsolved by any closed model evaluated, indicating substantial room for improvement before LMs can meaningfully assist in mathematical research. By contrast, the \calibrationsplits{} are nearly saturated by both closed and open-weight systems. Notably, while GPT-OSS-120B, Kimi-2.5, and GLM-5 each reach Pass@3 between 87.4\% and 88.3\% on \splitone{}, within 6 percentage points of the leading closed model (Gemini-3-Pro, 93.85\%), they trail by over 20 percentage points on \splitthree{}. This pattern suggests open-weight pipelines are well targeted to olympiad-style benchmarks but transfer poorly to research-adjacent material. Finally, through a timed human study with 25 participants of diverse mathematical expertise, we confirm that the difficulty is genuinely mathematical rather than artifactual, with aggregated teams covering 50.6\% of a 79-problem sample.
Show more
CogArena: Benchmarking Multimodal Agents on Interactive Behavioral Experiments
Kianté Fernandez ⋅ Caitlin Chen ⋅ Jialu Zhou ⋅ Bartek Sadowski ⋅ Anthony C Miceli ⋅ Ian Krajbich
Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into natural language prompts, bypassing the visual interfaces and time-pressured interactions through which human participants actually engage with experiments. In parallel, recent demonstrations that AI agents can produce human-like response data on live behavioral tasks raise concerns that online datasets used across the social and behavioral sciences are increasingly susceptible to contamination. We introduce CogArena, a benchmark of interactive experimental paradigms drawn from the social and behavioral sciences. Agents interact with the experiments through a standard browser, processing visual stimuli and responding under configurable deadlines. We designed a three-level scoring pipeline that evaluates task completion, performance accuracy normalized against literature-derived human baselines, and alignment with canonical task-specific signatures. Across frontier multimodal agents, we find that agents complete the tasks, but their behavior diverges from humans on the signatures these paradigms reliably elicit in people. CogArena provides the tasks, API, scoring pipeline, and a public leaderboard for ongoing community submissions.
Show more
LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale
Francesco Mantegna ⋅ Dulhan Jayalath ⋅ Gereon Elvers ⋅ Tasha Kim ⋅ Benjamin Ballyk ⋅ SungJun Cho ⋅ Teyun Kwon ⋅ Luisa Kurth ⋅ Miran Özdogan ⋅ Gilad Landau ⋅ Pratik Somaiya ⋅ Natalie Voets ⋅ Mark Woolrich ⋅ Oiwi Parker Jones
We introduce LibriBrain100, a large-scale MEG dataset for speech decoding designed from the ground up for reproducible, standardised evaluation. LibriBrain100 more than doubles the size of the original LibriBrain release, resulting in over 100 hours of high-quality MEG acquired while subjects listened to naturalistic continuous speech. With ~80 hours from a single subject, LibriBrain100 sets a new record for ‘deep’, within-subject neural data (8x more than the next comparable dataset and roughly 80x more than other datasets). To demonstrate the payoff of this depth-first design, we evaluate on a word-classification benchmark—an increasingly well-established stepping stone towards the open challenge of non-invasive brain-to-text decoding. Using an existing decoding model, we achieve state-of-the-art performance—validating both the quality of the recordings and the value of within-subject data at scale. Because collecting 80 hours of data per user is impractical for real-world applications, we also collected ~40 minutes of additional data from each of 32 subjects. Using the same word-classification benchmark, we demonstrate the value of ’broad’ multi-subject data: supervised fine-tuning of a pre-trained model can substantially compensate for limited per-subject data. We provide standard train, validation, and test splits, all reproducible through an open-sourced Python library that supports easy downloading, optional preprocessing, and data loading for common deep learning frameworks. In addition, the dataset and evaluation infrastructure are being released alongside an open machine-learning competition with a public leaderboard for standardised benchmarking. Ultimately, our hope is that LibriBrain100 will accelerate progress towards practical non-invasive brain-computer interfaces, capable of restoring communication to people living with severe paralysis.
Show more
Who Called? V33DA: A Physically Verified Multimodal Benchmark for Vocal Attribution in Zebra Finch Groups
Maris Basha ⋅ Yuhang Wang ⋅ Xiaoran Chen ⋅ Longbiao Cheng ⋅ Luca N Yapura ⋅ Anja T Zai ⋅ Mathieu Salzmann ⋅ Richard Hahnloser
Deciding which member of a group produced a vocalization and where that vocalization was produced are central to studying social communication but are rarely evaluated with physically verified ground truth. Existing resources either localize sounds without identifying the caller, or recognize individuals from isolated vocalizations without reliance on the 3d candidate geometry. We close this gap with V33DA, a multimodal benchmark for spatial vocal attribution in social zebra finches: it comprises 33,625 verified events coupling 5-channel audio with three camera views, calibrated 3D pose for every visible candidate, and per-bird radio telemetry across 10 individuals and three experiments. Caller identity is verified from an on-body accelerometer-derived vibration signal; this channel is withheld from benchmark models and used only by an oracle ceiling. We also provide V33DA++ as an auxiliary extension with verified overlapping-call events and $\pm 2$s context windows, intended to support future source-separation and vocal-activity-detection studies. V33DA separates familiar-individual recognition from transferable spatial attribution. We evaluate candidate-conditioned caller attribution, with 3D source localization as a secondary diagnostic, under three regimes: session-disjoint testing, held-out-experiment transfer, and leave-one-individual-out evaluation. A broad set of reference methods exposes a consistent gap between in-domain accuracy and transfer under identity shift: methods that can exploit familiar vocal identity perform well on known callers but degrade sharply on unseen ones, whereas candidate-aware methods preserving explicit spatial reasoning transfer substantially better. The accelerometer oracle stays near-perfect across regimes, confirming the internal consistency of the accelerometer-verified labels and quantifying the ceiling available when the withheld verification channel is observed. We release V33DA with fixed attribution/localization protocols and V33DA++ with overlap and long-context tasks, together with code for the evaluated methods.
Show more
RobustGenBench: A Benchmark for Robust Generalization to Adversarial and Common Perturbations, with Applications to Vision and Vision-Enabled Large Language Models
Maxime Heuillet ⋅ Jonas Ngnawé ⋅ Yann Pequignot ⋅ Rishika Bhagwatkar ⋅ Alexandre Larouche ⋅ Irina Rish ⋅ Christian Gagné ⋅ Ola Ahmad ⋅ Audrey Durand
Robust generalization (i.e., how models behave across diverse perturbation settings) is poorly understood for modern classification techniques such as robust fine-tuning from pretrained backbones and zero-shot classification with vision-enabled large language models (Ve-LLMs). We introduce \texttt{RobustGenBench}, a standardized benchmark of six classification datasets with reproducible evaluation splits and a unified perturbation protocol covering adversarial ($\ell_1, \ell_2, \ell_{\infty}$) and common perturbations. Using \texttt{RobustGenBench}, we conduct one of the most comprehensive studies of robust fine-tuning to date (40 pretrained backbones, 2 robust losses, and 3 adaptation protocols yielding 7{,}200 robustness measurements). We find that convolutional architectures with TRADES perform best at base scale, and the advantage of TRADES over Classic AT widens as size grows, and that hybrid architectures unlock competitive robust generalization. We further evaluate two frontier Ve-LLMs (GPT-4o mini, Gemini 3 Flash) and show that they exceed robust fine-tuned vision models on clean and common perturbations, and exhibit a striking resilience to $\ell_1$ perturbations that fine-tuned models lack. By evaluating both modeling techniques on the same tasks and perturbation settings, \texttt{RobustGenBench} provides a unifying view of robust generalization across these regimes.
Show more
InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search
Bohan Hou ⋅ Jiuning Gu ⋅ Jiayan Guo ⋅ Ronghao Dang ⋅ Sicong Leng ⋅ Xin Li ⋅ Xuemeng Song ⋅ Jianfei Yang
Existing benchmarks for multimodal agentic search evaluate multimodal search and visual browsing, but visual evidence is either confined to the input or treated as an answer endpoint rather than part of an interleaved search trajectory. We introduce \textbf{InterLV-Search}, a benchmark for Interleaved Language-Vision Agentic Search, in which textual and visual evidence is repeatedly used to condition later search. It contains 2,061 examples across three levels: active visual evidence seeking, controlled offline interleaved multimodal search, and open-web interleaved multimodal search. Beyond existing benchmarks, it also includes multimodal multi-branch samples that involve comparison between multiple entities during the evidence search. We construct Level 1 and Level 2 with automated pipelines and Level 3 with a machine-led, human-supervised open-web pipeline. We further provide InterLV-Agent for standardized tool use, trajectory logging, and evaluation. Experiments on proprietary and open-source multimodal agents show that current systems remain far from solving interleaved multimodal search, with the best model below 50% overall accuracy, highlighting challenges in visual evidence seeking, search control, and multimodal evidence integration. We release the benchmark data and evaluation code at \url{https://anonymous.4open.science/r/InterLV-Search-F423}
Show more
SWE-Pro: A Benchmark for Evaluating LLMs on Performance-Oriented Repository-Level Optimization
Ezgi Sarıkayak ⋅ Wenchao Gu ⋅ Hesham Ghonim ⋅ Chunyang Chen
Software performance optimization is a notoriously complex and manual task. Despite the growing use of Large Language Models (LLMs) for code refinement, we still lack benchmarks that capture how optimization actually happens in real-world codebases. Existing frameworks often oversimplify the problem by focusing on isolated functions or a single performance metric, missing the critical trade-offs between execution time and memory footprint, the inherent noise of the measurement environment, and the variability introduced by different input data and execution conditions. We address this by introducing SWE-Pro, a repository-level benchmark derived from 102 expert-written optimizations from open-source projects. Unlike previous benchmarks, SWE-Pro pairs each task with parameterized tests to evaluate runtime, peak memory, and Time-Weighted Memory Usage (TWMU) varying input data and execution conditions under noise-aware measurement conditions. Our evaluation shows that current LLMs struggle significantly: runtime gains are negligible, and memory optimizations are nearly non-existent. This stands in sharp contrast to expert implementations, which achieve an aggregate speedup of 15.5x and peak memory reduction of 171.3x over benchmark tasks. Expert-written improvements are observed in 91.2% of tasks for runtime and 65.7% for peak memory. Our findings expose a substantial gap between current LLMs capabilities and the demands of expert-level engineering.
Show more
QEC Model Zoo: Democratizing AI-enhanced Quantum Error Correction
Yuandou Wang ⋅ Maxim de Groot ⋅ Aadi Patwardhan ⋅ Floris Geerts ⋅ Rihan Hai
Quantum error correction (QEC) is essential for scalable quantum computing. AI-enhanced QEC is gaining increasing attention, with learned full decoders and pre-decoders being explored across code families. However, progress is difficult to assess: existing studies use different datasets, input representations, decoder roles, preprocessing pipelines, and evaluation protocols, making it unclear whether a reported gain comes from the model, the data interface, or the benchmark setup. We present the QEC model zoo, a data-centric benchmark and model-zoo framework for AI-enhanced QEC. The QEC model zoo curates real and simulated datasets for repetition- and surface-code decoding, supports both full decoders and pre-decoders, and records the metadata needed for reproducible comparison. The framework provides common task definitions, model-ready processing pipelines, and evaluation interfaces for heterogeneous QEC data. It further enables systematic comparison across neural, tree-based, and modern tabular models. Our goal is not to advocate a single decoder architecture, but to make AI-enhanced QEC easier to compare, reproduce, and extend.
Show more
PCFBench: How Far Can Large Vision-Language Models Go in Physics-aware Photonic Inverse Design?
Shengchao Chen ⋅ Ting Shu ⋅ Sufen Ren
Photonic crystal fibres (PCFs) underpin critical applications in telecommunications, sensing, and high-power laser delivery, yet their design remains a manual, expert-intensive process that couples geometric intuition with deep waveguide physics. Vision-language models (VLMs) offer a promising path toward automating this pipeline, but can they truly reason about photonic physics from microscopy images, or do they merely recognise visual patterns? We introduce PCFBench, the first comprehensive benchmark for evaluating VLMs on the complete photonic inverse-design workflow. PCFBench spans 26 tasks organised into six categories of increasing cognitive demand (Geometry Perception, Physics Understanding, Text Generation, Multi-modal Reasoning, Inverse Design, and Code Generation), built from over 220K FDTD simulation samples across 18 structurally diverse fibre families. We evaluate both individual VLMs on isolated tasks and multi-agent systems (MAS) on the full sequential pipeline from perception to design. Our results reveal a striking perception-design gap: current models can identify fibre geometry with moderate success, yet consistently fail to translate that visual understanding into physics-aware design actions. Moreover, cascading agents through the pipeline exposes significant error propagation, where upstream perception mistakes compound into downstream design failures. These findings suggest that neither model scaling nor naive agent chaining suffices; closing the loop demands explicit physical grounding. PCFBench provides the community with a rigorous, multi-dimensional testbed to measure progress toward AI-assisted photonic engineering.
Show more
AET-Bench: Pixel-Accurate Atomic Tomography Is Not Atom-Accurate
Zicheng Liu ⋅ Wenzhuo Ma ⋅ Jintao Chen ⋅ Di Huang
Atomic electron tomography (AET) aims to recover the three-dimensional arrangement, chemical identity, and temporal evolution of individual atoms. Yet most reconstruction methods are still evaluated primarily by voxel-level image similarity, such as PSNR, SSIM, or Fourier shell correlation. In this work, we show that this evaluation practice can select scientifically incorrect reconstructions: a volume that looks accurate can contain atoms that are shifted, merged, missed, hallucinated, chemically mislabeled, or assigned inconsistent identities through time. We introduce AET-Bench, a dataset and benchmark for atom-first evaluation of atomic electron tomography. AET-Bench contains 755 samples across a six-tier realism ladder, spanning clean analytical simulations, binary alloys, 4D atomic dynamics, multislice scattering, real experimental tilt series, and Brownian Pt nanocrystal trajectories. It defines four complementary tasks and evaluates 14 core method families with 21 logged method entries under a unified protocol. AET-Bench reveals a central failure mode hidden by voxel-only evaluation. On the AET-Sim-Mono tier, high-SSIM neural and Gaussian reconstructions can recover substantially fewer atoms than classical or sparsity-regularized methods: for example, 4D-GS achieves strong voxel similarity but only 43.9\% atom recall, while FISTA-TV reaches 95.1\% atom recall despite much lower SSIM. This inversion persists across atom-matching thresholds, showing that it is not an artifact of a single cutoff. On more realistic multislice and real tiers, atom recovery further degrades, exposing a physics gap between visually plausible volumes and reference atomic models. For 4D reconstruction, we find that smooth temporal density fields do not guarantee persistent atom identities. AET-Bench establishes atom-level fidelity as a necessary evaluation axis for AET. Our results suggest that future AET methods should not be judged only by how reconstructions look, but by whether they recover the atoms, species, and trajectories that scientists actually use.
Show more
VEHBench: A Stage-Local Diagnostic Benchmark for LLM-Assisted Vibration Energy Harvester Design
Depeng Su ⋅ Yuyu Luo ⋅ Guobiao Hu
Battery-free Internet of Things (IoT) requires iterative design of vibration energy harvesters (VEHs) under coupled physical constraints, while LLMs are emerging as interface layers for engineering workflows. However, existing engineering benchmarks primarily assess final artifact validity, offering limited insights into how LLMs behave across different stages of coupled physical design. We introduce VEHBench, an engineering-native diagnostic benchmark for LLM-assisted VEH design, featuring 763 literature-grounded tasks scored by an analytical physical oracle. VEHBench evaluates four design roles: specification triage, verifier-guided search, corrupted-state recovery, and policy-conditioned selection. Experimental results reveal that LLM capability is strongly stage-dependent: no single model consistently dominates the entire workflow, and response-control profiles expose distinct behavioral patterns across design roles. VEHBench thus provides a stage-aware foundation for evaluating, selecting, routing, and improving verifier-grounded engineering LLMs. The anonymous artifact is available at https://huggingface.co/datasets/AnonymousVehbench/vehbench.
Show more
SWE-Git-Bench: A Worktree-Level Benchmark for Real Merge Conflict Resolution
Wei Zhang ⋅ Jian Yang ⋅ jiajun wu ⋅ Zidan Tang ⋅ Siwei Wu
Large-language-model code benchmarks mostly measure issue repair or fresh synthesis, but they do not isolate a common commit-gating operation: after git emits conflict markers, a resolver must leave the worktree file ready to commit. SWE-Git-Bench turns this operation into a reproducible benchmark. We replay real two-parent merges from SWE-bench-family source pools, retain files where replay produces conflict markers, and use the maintainer-committed merge file as the reference. The release contains 135 slice-counted conflicts and 185 conflicted-file examples from 23 repositories; after collapsing Lite/Verified overlap, these correspond to 129 unique merge commits and 179 unique conflicted files. In an April 19–22, 2026 snapshot of 36 LLM endpoints, the best whole-file exact match is only 26.5%, and an oracle union across all models covers 41.6% of files. Conflict-block prompting raises edit similarity by +19.2 pp but changes exact match by only -0.1 pp, showing that an easier output interface does not by itself recover merge intent. A human-checked audit of 80 whole-file non-EM outputs with ES ≥ 0.99 labels 40/80 likely unsafe. We release public code and data, canonical JSONL artifacts, Core Croissant plus Responsible-AI metadata, scoring harnesses, model predictions, difficulty bands, and failure analyses.
Show more
TransitLM: A Large-Scale Dataset and Benchmark for Map-Free Transit Route Generation
Hanyu Guo ⋅ JieDong Yang ⋅ Chao Chen ⋅ Longfei Xu ⋅ Kaikui Liu ⋅ Xiangxiang Chu
Public transit route planning traditionally depends on structured map infrastructure and complex routing engines, and no existing dataset supports training models to bypass this dependency. We present TransitLM, a large-scale dataset of over 13 million transit route planning records from four Chinese cities covering 120,845 stations and 13,666 lines, released as a continual pre-training corpus and benchmark data for three evaluation tasks with complementary metrics. Experiments show that an LLM trained on TransitLM produces structurally valid routes at high accuracy and implicitly grounds arbitrary GPS coordinates to appropriate stations without any explicit mapping. These results demonstrate that transit route planning can be learned entirely from data, enabling end-to-end, map-free route generation directly from origin-destination information. The dataset and benchmark are available at https://huggingface.co/datasets/GD-ML/TransitLM, with evaluation code at https://github.com/HotTricker/TransitLM.
Show more
The Missing Corpus: Infinite Ground Truth for File-Grounded LLM Evaluation
Anush Sankaran ⋅ Arijit Banerjee ⋅ Pamela Bhattacharya ⋅ Tanujay Saha
Evaluating enterprise LLMs on file-grounded Question Answering (QA) requires large, labeled corpora of realistic enterprise PDFs, yet real documents are sensitive, scarce, and impossible to annotate at scale. Existing synthetic approaches produce visually crude renders, annotate ground truth post-hoc (introducing hallucination risk), or are non-reproducible one-off scripts with no diversity control. We present SynthDocQA, an Intermediate Representation(IR)-driven pipeline that co-generates photorealistic multi-page enterprise PDFs and provenance-grounded evaluation pairs in a single pass. A compact Document Generation IR encodes topic, content schedule, and perturbation profile before rendering begins. Templates are retrieved per page via Contrastive Language-Image Pretraining (CLIP) embedding-based cross-modal search over a 10,000-layout library; a 15-stage physical degradation pipeline models five scan quality tiers from pristine to poor. Every QA assertion is derived analytically from the same structured artifact that populated the page, making hallucination structurally impossible, and a hierarchical fork-based Random Number Generator (RNG) guarantees exact reproducibility from a single integer seed. A reference 100-document corpus spans 50 enterprise domains, 4,374 pages, 16,589 QA pairs, and 21,402 assertions at zero annotation cost.
Show more
Croissant Baker: Metadata Generation for Discoverable, Governable, and Reusable ML Datasets
Rafi Al Attrach ⋅ Rajna Fani ⋅ Sebastian Lobentanzer ⋅ Joan Giner-Miguelez ⋅ DEBANSHU DAS ⋅ Varuni H K ⋅ Nobin Sarwar ⋅ Rajat Ghosh ⋅ Anwai Archit ⋅ Surbhi Motghare ⋅ Christina C Parry ⋅ Luis Oala ⋅ Lara Grosso ⋅ Joaquin Vanschoren ⋅ Steffen Vogler ⋅ Sujata Goswami ⋅ Eric Rosenthal ⋅ Marzyeh Ghassemi ⋅ Matthew McDermott ⋅ Tom Pollard
Croissant has emerged as the metadata standard for machine learning datasets, providing a structured, JSON-LD-based format that makes dataset discovery, automated ingestion, and reproducible analysis machine-checkable across ML platforms. Adoption has accelerated, and NeurIPS now requires Croissant metadata in every submission to its dataset tracks. Yet in practice Croissant generation usually starts with uploading data to a public platform, a path infeasible for governed and large local repositories that hold much of the high-value data ML increasingly relies on. We release Croissant Baker, a local-first, open-source command-line tool that generates validated Croissant metadata directly from a dataset directory through a modular handler registry. We evaluate Croissant Baker on over 140 datasets, scaling to MIMIC-IV at 886 million rows and 374 Parquet files. On held-out comparisons against producer-authored or standards-derived ground truth, Croissant Baker reaches 97–100% agreement across multiple domains.
Show more
KNOT: A Knowledge Entanglement Benchmark for Robust Unlearning Evaluation
Mengyang Li ⋅ Jingwen Wang ⋅ Yu Zhang ⋅ Shuang Liu ⋅ Zhong Zhang
Large language model (LLM) unlearning aims to selectively remove specific knowledge while preserving model utility. Current benchmarks such as TOFU, WMDP, and MUSE evaluate unlearning on semantically disjoint forget and retain sets, making the task artificially easy. We formalize this gap through a taxonomy of knowledge entanglement at three levels: entity, concept, and skill. Based on this taxonomy, we construct KNOT, a benchmark with approximately 5,300 forget, 9,800 retain, and 5,000 boundary QA pairs, each annotated with a quantitative entanglement score. Evaluation of seven methods on KNOT reveals that (a) all methods degrade severely under high entanglement, (b) the coreset effect reported on standard benchmarks vanishes, and (c) existing metrics miss critical boundary behavior. We propose Boundary Precision and Boundary Recall to fill this gap. Cross-benchmark comparison confirms that KNOT's entanglement, rather than surface difficulty, drives the observed degradation.
Show more
Fine-Grained Benchmark Generation for Comprehensive Evaluation of Foundation Models
Mohammed Saidul Islam ⋅ Arash Afkanpour ⋅ Negin Baghbanzadeh ⋅ Farnaz Kohankhaki ⋅ Afshin Cheraghi ⋅ Ali Kore ⋅ Shayaan Mehdi ⋅ Elham Dolatabadi
Evaluation of foundation models often rely on aggregate scores from benchmarks that lack comprehensive coverage and metadata for a fine-grained evaluation. We introduce a framework for automated benchmark generation. Our framework generates evaluation problems grounded in reference material, such as textbooks, producing benchmarks with broad coverage, rich metadata, and robustness to contamination. The pipeline employs a multi-agent architecture for problem generation and a solution-graph-driven strategy that significantly improves the reliability of ground truth solutions. Using the framework, we generate three benchmarks in Machine Learning, Corporate Finance, and Personal Finance. Expert review finds a significantly lower ground-truth error rate than previous benchmarks such as MMLU and GSM8K. Evaluation of 12 commercial and open-source models shows that our benchmarks achieve near-uniform competency coverage and surface performance differences across models that existing benchmarks fail to capture. We open-source the framework and our curated benchmarks.
Show more
RescueBench: Can Embodied Agents Save Lives in the Wild?
Kui Wu ⋅ Beiyu Guo ⋅ Hao Chen ⋅ Shuhang Xu ⋅ Yuling Li ⋅ Yongdan Zeng ⋅ Zhoujun Li ⋅ Yizhou Wang ⋅ Fangwei Zhong
Search-and-rescue (SAR) requires embodied agents to explore unfamiliar environments under multimodal uncertainty, perform multi-stage interactions, and retrieve spatial memory over long horizons. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear how failures compound when they must be composed in realistic workflows. We introduce RescueBench, a photo-realistic diagnostic benchmark that instantiates SAR as a four-stage pipeline: multimodal exploration, target rescue, memory-guided return, and final handoff. By combining sequential task composition with stage-level evaluation, RescueBench enables analysis of how exploration and memory failures propagate through embodied rescue workflows. It contains five progressive difficulty levels that vary environmental complexity, clue ambiguity, and spatial hierarchy, along with an automatic episode generation and annotation pipeline for scalable evaluation and training. We evaluate seven baselines, an oracle reference, and human players, showing that no baselines completes the full task at the highest difficulty. Stage-level diagnosis identifies autonomous exploration as the dominant failure mode and spatial memory as a second, independent bottleneck, suggesting that these limitations are not resolved by current topological visual-language navigation or map-based methods.
Show more
MCPHallu: Benchmarking Reasoning, Execution, and Memory Hallucinations in MCP Agents
Yizhen Jiang ⋅ 浩伟 郭 ⋅ Tianyi Bai ⋅ Zengjie Hu ⋅ Lu Ma ⋅ Zhengyang Zhao ⋅ Haoze Sun ⋅ Peng Pei ⋅ Wentao Zhang
Large language model agents are increasingly deployed through the Model Context Protocol (MCP), which allows them to discover and invoke tools across multiple servers at runtime. Existing MCP benchmarks primarily measure task completion capability but provide limited insight into why agents fail, reporting aggregate success rates without attributing failures to specific reasoning, execution, or memory breakdowns. We introduce MCPHallu, a benchmark for diagnosing hallucination-prone agent failures in MCP environments along three axes: reasoning, execution, and memory. MCPHallu contains 358 tasks across five capability domains and runs on 36 production MCP servers in isolated containers. Each task is designed with a structural trigger that makes one of four subtypes the dominant failure risk: Branch Collapse, Unreachable Goal, Tool Misuse, or Context Forgetting. This design enables subtype-level analysis rather than aggregate task success alone. We evaluate fourteen frontier LLM agents from eight vendors on 5,012 trajectories. Current MCP agents remain vulnerable to hallucination-inducing tool environments, achieving an average score of 0.642. Failures concentrate on Unreachable Goal tasks, where the average score drops to 0.458 and 22.1\% of runs pass, indicating that agents often continue issuing tool calls or claim successful completion rather than recognizing infeasible goals. Performance across the four subtypes is largely independent: per-model ranks span on average 6.9 of 13 positions, and Branch Collapse is uncorrelated with Unreachable Goal across models ($r = -0.12$). MCPHallu is released at https://anonymous.4open.science/r/mcp_hallu.
Show more
RobustStress: Stress-Testing AI-Generated Text Detectors under Gradual Perturbations
Youcheng Yan ⋅ Jinshuo Liu ⋅ Qiyuan Li ⋅ Xinya Liu ⋅ Jinguang Gu ⋅ Donghong Ji ⋅ Jeff Pan
Detecting text generated by large language models (LLMs) is a critical line of defense for ensuring information content security. Although recent studies have gradually shifted to evaluating the robustness of detectors, existing benchmarks generally suffer from two primary limitations: (1) they lack granularity in controlling perturbation intensity, making it difficult to systematically evaluate the dynamic changes in detector performance as attack intensity increases; and (2) they fail to investigate the impact of perturbation intensity on detector robustness within complex scenarios, such as cross-domain and cross-model settings. To address these challenges, we propose \textbf{RobustStress}, a novel stress-testing framework for intensity-controlled perturbations. We achieve quantitative classification of perturbation intensity by varying the proportion of synonym replacement and employing graded LLM-based paraphrasing attacks. Experiments reveal distinct patterns of performance degradation as perturbation intensity increases, and further capture the synergistic weakening effect between cross-scenario shifts and adversarial perturbations. RobustStress provides an effective stress-testing framework for evaluating detectors in real-world scenarios, encouraging the community to move beyond performance on clean benchmarks to the development of more robust defense mechanisms against adversarial attacks.
Show more
SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?
Kean Shi ⋅ Zihang Li ⋅ Tianyi Ma ⋅ Zengji Tu ⋅ Jialong Wu ⋅ Xinbo Xu ⋅ Qingyao Yang ⋅ Weichu Xie ⋅ Ruoyu Wu ⋅ Ming Wu ⋅ Jason Zeng ⋅ Michael Heinrich ⋅ Liang Chen ⋅ Kuan Li ⋅ Baobao Chang
Computer-Using Agents (CUAs) are rapidly extending large language models (LLMs) beyond text-based reasoning toward action execution in more complex environments, such as web browsers and graphical user interfaces (GUIs). However, existing web and GUI agent benchmarks often rely on simplified settings, isolated tasks, or short-horizon interactions, making it difficult to assess capabilities of agents in realistic professional workflows. Software-as-a-Service (SaaS) environments are a natural choice for CUA evaluation, as they host a large share of modern digital work and naturally involve dynamic system states, cross-application coordination, domain-specific knowledge, and long-horizon dependencies. To this end, we introduce \textbf{SaaS-Bench}, a benchmark built on 23 deployable SaaS systems across six professional domains, containing 106 tasks grounded in realistic work scenarios. These tasks require long-horizon execution, cover both text-only and multimodal settings, and are evaluated with weighted verification checkpoints that measure strict task completion and partial progress. Experiments show that representative LLM-based agents struggle on SaaS-Bench, with even the strongest model completing fewer than 4\% of tasks end-to-end, exposing limitations in planning, state tracking, cross-application context maintenance, and error recovery.
Show more
Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs
Yen-Shan Chen ⋅ Zhi Rui Tam ⋅ Cheng-Kuang Wu ⋅ Yun-Nung (Vivian) Chen
Current evaluations of LLM safety predominantly rely on *severity-based taxonomies* to assess the harmfulness of models' responses to malicious queries. We argue that this formulation requires re-examination as it implicitly equates model output with realized harm while neglecting *Execution Likelihood*—the conditional probability of a threat being realized by a real-world actor given a model's response. In this work, we introduce **Expected Harm**, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of *execution cost*. Through empirical analysis of state-of-the-art models, we reveal a systematic *Inverse Risk Calibration*: models disproportionately exhibit stronger refusal behaviors for low-likelihood (high-cost) threats while remaining vulnerable to high-likelihood (low-cost) queries. Mechanistically, we first provide an analysis using linear probing, revealing that while refusal mechanisms linearly separate queries by severity, they do not use execution likelihood as an indicator for refusal. Then, we use Olmo-3's open-source staged checkpoints to trace the root cause of miscalibration to the Supervised Fine Tuning (SFT) training data distribution (82.8\% of safety examples at cost levels 0–1) and show that cost representations are weak from the SFT stage onward. Finally, we demonstrate that this miscalibration creates a vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to $2\times$.
Show more
Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings
Inês Oliveira e Silva ⋅ Sérgio Jesus ⋅ Iker Perez ⋅ Rita P. Ribeiro ⋅ Carlos Soares ⋅ Hugo Ferreira ⋅ Pedro Bizarro
Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape with little consensus on practical deployment. While theoretical differences are well-documented, evaluation remains reliant on quantitative proxies whose alignment with human utility is unverified. In this work, we use a unified amortized framework to isolate semantic differences between eight Shapley variants under the low-latency constraints of operational risk workflows. We conduct a large-scale empirical evaluation across four risk datasets and a realistic fraud-detection environment involving professional analysts and 3,735 case reviews. Our results reveal a fundamental misalignment: standard quantitative metrics, such as sparsity and faithfulness, are decoupled from human-perceived clarity and decision utility. Furthermore, while no formulation improved objective analyst performance, explanations consistently increased decision confidence, signaling a critical risk of automation bias and overconfidence in high-stakes settings. These findings demonstrate that current evaluation proxies are insufficient for predicting human impact, motivating our release of interaction measurements to support research into behaviorally grounded XAI evaluation.
Show more
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
Parth Asawa ⋅ Christopher M Glaze ⋅ Gabriel Orlanski ⋅ Ramya Ramakrishnan ⋅ Benji Xu ⋅ Asim Biswal ⋅ Vincent Chen ⋅ Frederic Sala ⋅ Matei A Zaharia ⋅ Joseph Gonzalez
Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this---in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.
Show more
Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations
Xinshun Feng ⋅ Ziqi Miao ⋅ Lijun Li ⋅ Jing Shao
Large language model (LLM) agents are increasingly deployed in scientific research, where reliability is critical, and the underlying knowledge is densely interconnected. In such settings, hallucinations are particularly damaging: a single erroneous claim on a foundational concept can propagate through multi-step reasoning and corrupt entire trajectories. Existing hallucination benchmarks largely operate at the surface level, treating facts in isolation and relying on uniform accuracy metrics that ignore this topological structure. We address this gap with \textsc{Schema}, the first evidence-grounded, topology-aware evaluation framework for scientific agents. Built on an automated concept-graph construction pipeline, \textsc{Schema} provides two complementary diagnostic instruments. A trajectory hallucination pipeline audits intermediate reasoning at scale via a topology-weighted severity score ($\mathrm{HS}^w$), while a multi-agent counterfactual attribution module pinpoints the causal mechanism behind selected failures. The framework supports diverse task formats spanning multi-hop reasoning, claim verification, and programmatic experimental execution. Across two biomedical subdomains and eleven contemporary LLM agents, \textsc{Schema} reveals that hallucinations concentrate at a small set of highly connected knowledge hubs, and that final-answer accuracy decouples from trajectory honesty—models often reach correct conclusions through structurally flawed reasoning. These results indicate that for high-stakes scientific applications, terminal accuracy alone is an insufficient signal of agent reliability, motivating mechanism-level evaluation grounded in knowledge topology\footnote{Code is available at \url{https://anonymous.4open.science/r/SCHEMA_NIPS2026}}.
Show more
Attack Selection In Agentic AI Control Evaluations Meaningfully Decreases Safety
Catherine Ge-Wang ⋅ Tyler Crosse ⋅ Benjamin Hadad ⋅ Joachim Schaeffer ⋅ Ram Potham ⋅ Tyler Tracy
An attacker that strategically chooses when to attack is much harder to catch than one that attacks indiscriminately. AI control is a safety framework for deploying capable but untrusted AI agents under oversight from a weaker trusted monitor and a limited human audit budget. Control evaluations stress-test these protocols by pitting a red-team attack policy against the blue-team monitor, but current evaluations typically assume attackers that do not strategically select when to attack. However, a capable attacker would not attack indiscriminately. We study this capability, attack selection, in agentic settings by decomposing attack decisions into a start policy, which decides when an attacker should attack, and a stop policy, which decides when an attacker should abort an ongoing attack. Across two agentic settings, BashArena and LinuxArena, both policies substantially lower measured empirical safety without changing the underlying attack capability. At a 1% audit budget, our start policy reduces safety by 20pp on both BashArena and LinuxArena, and our stop policy reduces safety by 20pp on BashArena and 28pp on LinuxArena. These reductions in safety should be interpreted as upper bounds on the effect of attack selection. Existing control evaluations may therefore yield overly optimistic safety estimates against selective attackers. We recommend that future evaluations, system cards, and safety cases should elicit attack selection to produce more realistic safety estimates.
Show more
PrivacySIM: Evaluating LLM Simulation of User Privacy Behavior
James Flemings ⋅ Qiumin Xu
Large language models (LLMs) are increasingly used to simulate human behavior, but their ability to simulate individual privacy decisions faithfully is not well understood. In this paper, we address the problem of evaluating whether a core set of user persona attributes can drive LLMs to simulate individual-level privacy behavior. We introduce PrivacySIM, an evaluation suite that benchmarks LLM simulation of user privacy behavior against the ground-truth responses of 1,000 users. These users are drawn from five published user studies on privacy spanning LLM healthcare consultations, conversational agents, and chatbots. Drawing on these user studies, we hypothesize three persona facets as plausible predictors of privacy decision-making: demographics, previous experiences with AI, and stated privacy attitudes. We condition nine frontier LLMs on subsets of these three facets and measure how often each model's response to a data-sharing scenario matches the user's actual response. Our findings show that (1) privacy persona conditioning consistently improves simulation quality over no-persona conditioning, but even the strongest model (40.4% accuracy) remains far from faithfully simulating individual privacy decisions. (2) A user's stated privacy attitudes alone may not be the best predictor as they often diverge from the user's actual privacy behavior. (3) Users with high AI experience but low stated privacy attitudes are the most challenging to simulate. PrivacySIM is a first step toward understanding and improving the capabilities of LLMs to simulate user privacy decisions.
Show more
Benchmarking Membership Privacy Risks in Preference-Based LLM Post-Training
Lorenzo Rossi ⋅ Kaif Shaikh ⋅ Franziska Boenisch ⋅ Adam Dziedzic
Modern language models are commonly adapted after pretraining to follow instructions, align with user preferences, and improve deployment behavior. Such post-training often relies on preference data from users, annotators, or model interactions, which may contain sensitive prompts, private responses, proprietary tasks, or confidential judgments. Understanding the privacy implications of this data is therefore essential. Membership inference attacks (MIAs), which test whether a record was used for training, are the de facto standard for empirical privacy auditing in machine learning. However, preference-based post-training changes the unit of membership: a training record contains a prompt, a preferred response, and a dispreferred response, rather than a single input-output sequence. Audits that ignore this structure can under-report leakage. We therefore introduce a benchmark protocol that adapts strong reference-model MIAs to complete preference records. Our protocol asks whether membership is exposed by either response alone, by the model’s preference between them, or by the two responses jointly. We find that auditing the response pair can reveal membership evidence missed when the record is reduced to a single score. Across three preference datasets, two model families, and seven post-training objectives spanning three post-training families, measured leakage depends strongly on both the audited record statistic and the training objective, with imbalanced risk between preferred and dispreferred responses. We further evaluate the impact of parameter-efficient fine-tuning (PEFT) and differential privacy (DP), finding widely varying privacy-utility trade-offs. Overall, preference-based post-training leaks in ways that standard single-response audits can miss, motivating formal privacy methods that protect complete preference records while preserving post-training utility.
Show more
MemPoison: Uncovering Persistent Memory Threats and Structural Blind Spots in LLM Agents
Jifeng Gao ⋅ Kang Xia ⋅ Yi Zhang ⋅ Xiaobin Hong ⋅ Mingkai Lin ⋅ Xingshen Wei ⋅ Wenzhong Li ⋅ Sanglu Lu
Persistent external memory enhances agent continuity but introduces persistent security vulnerabilities: adversarial content can be injected via standard interaction channels, retained across turns, and later distort downstream behavior. To address this challenge, we propose **MemPoison**, a comprehensive benchmark and analysis framework featuring 1227 hand-validated cases across four attack types, three injection channels, and three representative memory substrates, evaluated on seven open-weight and three closed-weight model families. We introduce a three-tier taxonomy: (**L1**) direct single-record corruption, (**L2**) compositional multi-record corruption and (**L3**) context-triggered dormant corruption. Our evaluations reveal a distinct defense frontier: while baseline write-time defenses, such as consistency checks, substantially suppress direct L1 attacks, they fail to reliably suppress L2 and L3 attacks. Through **mechanistic influence decomposition (MID)**, we demonstrate structural blind spots in write-time defenses, which admit seemingly benign records that later become harmful through joint retrieval composition or trigger-conditioned activation. Our findings advocate for shifting from static filtering to adaptive, context-sensitive memory defense strategies. Code, benchmark artifacts, and evaluation scripts are publicly released
Show more
DIS-Bench: Evaluating LLMs on System Testing via Directed Input Synthesis
Siwei Wei ⋅ Yuqi Guo ⋅ Yan Cai
LLM-based coding agents have shown strong capabilities on real-world software testing tasks, yet existing evaluations concentrate on unit test generation, leaving system testing unexplored. System testing is both practically important for software quality assurance and inherently difficult, as it demands repository-level holistic understanding and long-horizon reasoning over program behavior. In this paper, we propose to measure LLM agents' system testing capabilities through its core problem, Directed Input Synthesis (DIS): generating system-level inputs that exercise specified code branches. We introduce DIS-Bench, a benchmark built via an automated pipeline based on random testing that selects target branches that are both reachable and challenging. Because DIS-Bench is constructed from execution behavior rather than human-authored issues or patches, it is scalable and less susceptible to contamination from existing training corpora. DIS-Bench contains 8267 tasks drawn from 37 real-world repositories. We evaluate 9 mainstream LLMs on DIS-Bench-Lite under both BM25 retrieval and coding agents, including OpenHands, Mini-SWE-Agent, and Claude-Code. The results show that DIS is highly challenging: the best open-weight configuration resolves only 28.1\% of targets at pass@1, the best frontier configuration (57.8\%) still leaves over 40\% unsolved, and 23.1\% of targets are unsolved by every configuration in every attempt. We further perform a manual inspection to characterize the dominant failure modes, revealing reasoning bottlenecks of LLMs on system testing tasks.
Show more
PMO-Dock: Benchmarking Docking, Specificity, and Generalization in Molecular Optimization
Gor Simonyan ⋅ Tatevik Abrahamyan ⋅ Narek Abelyan ⋅ Tigran Fahradyan ⋅ Hrant Khachatryan
The Practical Molecular Optimization (PMO) benchmark standardized evaluation in molecular optimization, but it is built on simple property-based oracles that do not capture structure-based drug design. As the field has moved to docking-based objectives, evaluation practice has become unstandardized, and different methods report results on different docking tasks, often without a shared protocol for fair comparison or sample-efficiency constraints. To address this, we introduce Practical Molecular Optimization for Docking (PMO-Dock), a benchmark and protocol for docking-based optimization consisting of 25 tasks covering hit generation, lead optimization, and a new specificity task requiring strong on-target binding while penalizing off-target interactions. The protocol separates model development from final evaluation by assigning validation tasks for hyperparameter tuning and strictly held-out test tasks for reporting, with no test-task use during development. It also enforces strict oracle budgets to reflect realistic drug-discovery settings. We benchmark four diverse high-performing methods spanning different optimization paradigms, Saturn (reinforcement learning), GenMol (discrete diffusion), Genetic-guided GFlowNet, and Chemlactica (LLM-based). Our analysis shows no universally best method across tasks, substantial differences in hyperparameter sensitivity, and method-dependent transferability, where some methods benefit from global hyperparameter selection while others require task-local tuning on sufficiently similar validation tasks. This benchmark provides a concrete evaluation standard for measuring progress on generalizable, sample-efficient molecular optimization.
Show more
MorphoHELM: A Comprehensive Benchmark for Evaluating Representations for Microscopy-Based Morphology Assays
Emre Hayir ⋅ Lorin Crawford ⋅ Alex X Lu
Microscopy images contain rich information about how cells respond to perturbations, making them essential to applications like drug screening. To quantify images, researchers often use representation extraction methods, and recent years have seen a proliferation of deep learning methods. While measuring the quality of these representations is essential, evaluation remains fragmented, with each proposed model evaluated on different tasks and datasets, using custom pipelines and metrics, making it difficult to fairly compare models. Here, we introduce MorphoHELM, a comprehensive open benchmark for evaluating feature extraction methods for cell painting, the most widely-used morphological profiling assay. MorphoHELM consolidates evaluation standards in the field, extends and corrects them to be more robust, and evaluates on the widest range of methods to date. A defining feature of the benchmark is that each task is evaluated at different degrees of batch effects (or technical noise), directly quantifying how the ability of methods to detect biological signal degrades as noise increases. Together, these properties enable MorphoHELM to detect trade-offs between methods, and we demonstrate that models that excel at certain kinds of biological signal are weaker at others. We show that no existing model outperforms classic computer vision analytic strategies across all settings, which remain the strongest general use-case representations. All datasets, code, and evaluation tools are publicly available.
Show more
PaxBench: A Multimodal Sequence Benchmark for Protein Abundance Prediction
Ke Zhai ⋅ Oscar J Charles ⋅ Helena A Saunders ⋅ Conrad Bessant
Protein abundance links genotype to cellular state and is useful for comparing genes, organisms, and engineered systems. PaxDb contains broad abundance mea- surements, but its heterogeneous records have not yet been organized into a shared benchmark for testing protein, CDS, codon, and mRNA representations under the same target. The difficulty is that abundance labels must be recovered from diverse PaxDb sources, matched to species-specific sequences at several molecular levels, and evaluated despite incomplete modality coverage, limited metadata, and variable annotation quality. To address this, we introduce PaxBench, a carefully cleaned and aligned seven-species PaxDb-derived benchmark with matched protein, CDS, codon, and mRNA inputs, and use it to test whether biological foundation models benefit from multimodal fusion. On the cluster-aware split, the full multimodal representation reaches Spearman 0.789, compared with 0.740 for the best single representation, while random evaluation reaches 0.834. Cross-species settings also remain predictive. These results suggest that protein, coding-sequence, codon, and transcript representations provide complementary predictive information under this benchmark. PaxBench provides a compact setting for comparing foundation models, feature baselines, adaptation methods, and interpretability analyses for sequence-based abundance prediction.
Show more
TSB-SEG: A Systematic Time-Series Segmentation Benchmark
Félix Chavelli ⋅ Arik Ermshaus ⋅ Patrick Schäfer ⋅ Fan Yang ⋅ John Paparrizos ⋅ Paul Boniol
Time-series segmentation, studied as either \emph{Change Point Detection} (CPD) or \emph{State Detection} (SD), underpins a broad range of monitoring and diagnostic applications. However, the field suffers from a persistent divide between the statistical community (focusing on CPD) and the machine learning and data mining communities (focusing on SD). Consequently, the literature remains fragmented: evaluations typically cover narrow methodological families, datasets are often small or homogeneous, and CPD and SD are rarely evaluated together. We address these gaps with \texttt{TSB-SEG}, a unified benchmark comprising $592$ univariate and multivariate time series across $8$ heterogeneous domains and surveying $27$ algorithms spanning five decades of progress. To ensure a fair comparison, we introduce a two-pool design: a scalable \emph{main pool} of sub-quadratic detectors used for the aggregate ranking, and an \emph{extended-scope pool} for computationally intensive specialized methods. All methods are unified under tsseg, an open-source library to ensure full reproducibility. Our systematic evaluation yields three insights: (a) no single detector dominates across domains, and per-dataset hyperparameter tuning remains decisive for performance, (b) CPD accuracy is a strong proxy for SD accuracy, and (c) data characteristics should guide method selection. We provide actionable guidance based on data properties, error tolerances, and tuning budgets, identify impactful hyperparameters, and characterize algorithm robustness to these choices.
Show more
VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology
Luca Weishaupt ⋅ Simone de Brot ⋅ Javier Asin ⋅ Llorenç Grau-Roma ⋅ Nic G Reitsam ⋅ Andrew H Song ⋅ Dongmin Bang ⋅ Stefan T Kaluziak ⋅ Long P Le ⋅ Jakob N Kather ⋅ Faisal Mahmood ⋅ Guillaume Jaume
Pathology vision-language models are advancing rapidly, yet existing benchmarks remain focused on human tissue, particularly oncology, leaving non-human pathology largely unaddressed. This gap is especially important in toxicologic pathology, where microscopic tissue examination of laboratory animals is a core component of preclinical drug safety assessment. To address it, we introduce VIPER, the first expert-curated benchmark for vision-language model evaluation in toxicologic pathology. VIPER contains 1251 questions associated with 419 H&E-stained rat histology images across 9 organs, covering multiple-choice, KPrim, and free-text formats. All questions were curated and validated by board-certified veterinary pathologists. In total, we benchmarked 16 models, including two newly introduced veterinary-pathology models, seven human pathology-specialized models, and seven general-purpose frontier models. The results identify a substantial domain gap between veterinary and human pathology, expose the risk of over-diagnosis of normal tissue in frontier models, and show that domain-specific training remains critical for visually grounded predictions. VIPER data and evaluation code are available at https://github.com/mahmoodlab/viper.
Show more
MedFlowBench: Auditing Medical Agents in Full-Study Workflows
Weixiang Shen ⋅ Chengzhi Shen ⋅ Che Liu ⋅ Junde Wu ⋅ Jiayuan Zhu ⋅ Xiao Han ⋅ Zongyue Li ⋅ Jingpei Wu ⋅ Min Xu ⋅ Daguang Xu ⋅ Yueming Jin ⋅ Benedikt Wiestler ⋅ Daniel Rueckert ⋅ Jiazhen Pan
Medical imaging benchmarks often evaluate VLMs on pre-selected 2D images, slices, crops, or patches, making evaluation closer to visual recognition. Real clinical workflows impose a different burden: readers must search through complete studies, operate imaging software, navigate across slices and magnifications, and document visual evidence that can be audited. We argue that this evidence-producing workflow is a critical missing evaluation axis for medical imaging agents. To study it, we introduce MedFlowBench, a full-study benchmark for VLM agents, together with MedOpenClaw, a controlled and replayable runtime in which agents operate medical imaging viewers such as 3D Slicer and QuPath. In each episode, an agent inspects a complete radiology study or whole-slide pathology image, returns a task answer, and submits structured evidence, including key slices, coordinates, regions of interest, or lesion-state fields. This evidence is automatically checked against withheld masks, annotations, and labels. Across evaluated models, final answer-only scoring gives an overly optimistic picture: when answers must also be supported by correct evidence, performance drops substantially on complex workflows. We further find that adding image-analysis tools does not by itself solve the problem. Tools help when they make a complex procedure simple and reliable, but agents still struggle when they must choose inputs, manage viewer state, and verify intermediate outputs over multiple steps. MedFlowBench exposes whether medical imaging agents can produce auditable evidence from complete studies, rather than plausible answers from selected images.
Show more
VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
Zirui Wang ⋅ Junyi Zhang ⋅ Jiaxin Ge ⋅ Long Lian ⋅ Max Fu ⋅ Lisa Dunlap ⋅ Ken Goldberg ⋅ XuDong Wang ⋅ Ion Stoica ⋅ David Chan ⋅ Sewon Min ⋅ Joseph Gonzalez
Despite recent successes in visual reasoning, it is still poorly understood how vision-language models (VLMs) integrate perception, memory, and actions in visual interaction, a key capability that underlies real-world applications such as agents and robotics. Towards understanding these behaviors, we introduce VisGym, a gymnasium of 17 environments spanning symbolic puzzles, image understanding, navigation, and manipulation for the diagnostic evaluation of VLMs across domains. Beyond benchmarking for task performance, VisGym provides flexible controls on difficulty, input representation, planning horizon, and feedback, which enable controlled analyses of how interactive design choices affect model performance. We perform several experiments showing that even the strongest frontier models struggle in visual interaction, achieving low success rates in both the easy (26.8%) and hard (12.6%) configurations. Specifically, our analyses reveal four key vulnerabilities in current models: (1) they fail to effectively leverage long context, performing worse with unbounded history; (2) they struggle to utilize pure visual feedback, degrading when textual feedback is removed; (3) several text-based symbolic tasks become substantially harder once rendered visually; and (4) incorporating explicit goal observations can paradoxically backfire. Finally, each environment includes a heuristic multi-step solver that enables post-training case studies on supervised fine-tuning, including analyses of in-domain improvements, module-level contributions, data curation strategies, and preliminary studies of downstream transfer. We open-source all environments, data generators, evaluation pipeline, and training code at https://anonymous.4open.science/r/VisGym
Show more
MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale
Jiadong Zhang ⋅ Xiaosong Ma
Edge-deployed personal memory assistants must handle private interpersonal conversations on-device with open-weight models. Yet, existing memory benchmarks often under-test the combination of activity-dense interaction, ego-centric perspective, and coherent multi-session worlds. MemArena fills these gaps with a single-world conversational benchmark built with its MASim agent simulator, for $50$ agents over $15$ days ($10.3$M dialog-text tokens, $24.1$K text-only ego-observed tokens/agent/day). With the interaction history, it co-generates ground truth over six recall, reasoning, and trustworthiness evaluation dimensions. We evaluate five open-weight readers with Vanilla context, BM25-RAG, Oracle retrieval, Memobase, and MemSearch as memory backends. Three results stand out: (1) Memory-backend choice matters more for content accuracy: At Qwen3-0.6B, Memobase $\to$ MemSearch gains $+32.5/+19.2$ pp, exceeding MemSearch reader scaling ($+10.6/+6.8$ pp). (2) Permission-aware access fails universally: with Oracle leaking heavily and other backends too timid to disclose. (3) Search latency bites only at very small reader: on a Spark GB10 edge node, memory-search adds a moderate and fixed $87$/$7$/$48$ ms (BM25-RAG/Memobase/MemSearch) that composes a small part of TTFT for most reader-backend combinations. We release code, the MASim simulator, and the MemArena-L benchmark at https://anonymous.4open.science/r/MemArena-NeurIPS2026-anon-3E9D
Show more
RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models
Ruinan Jin ⋅ Xinting Liao ⋅ Hanlin Yu ⋅ Deval Pandya ⋅ Xiaoxiao Li
Modern Voice Cloning (VCL) can synthesize speech that closely matches a target speaker from only seconds of reference audio, enabling applications such as personalized speech interfaces and dubbing. In practical deployments, modern audio generation models inevitably encounter noisy reference audios, imperfect text prompts, multilingual and long-form generation settings, downstream post-processing, and adversarial perturbations, all of which can significantly hurt robustness. Despite rapid progress in VCL driven by autoregressive codec-token language models and diffusion-based models, robustness under realistic deployment shifts remains underexplored. This paper introduces RVCBench, a comprehensive dataset and benchmark that evaluates Robustness in Voice Clone across the full generation pipeline. RVCBench contributes a large-scale, task-aligned robustness dataset that instantiates realistic deployment shifts through controlled text-audio pairing, multilingual and long-form scenarios, expressive prompts, post-processing conditions, and passive or proactive audio perturbations. Covering 18 robustness evaluations, 225 speakers, and 14,370 utterances, RVCBench enables unified evaluation of input sensitivity, generation stability, output resilience, and perturbation robustness. We evaluate 18 representative modern VCL models and reveal systematic vulnerabilities in content consistency, speaker similarity, long-form stability, post-processing resilience, detectability, and adversarial robustness.
Show more
SWE-Marathon: Can AI Agents Autonomously Complete Ultra-Long-Horizon Software Work?
Rishi Desai ⋅ Joan Cabezas ⋅ Neel Harsola ⋅ Adnan E Assadi ⋅ Pramod Srinivasan ⋅ Roey B Chaim ⋅ Fenil Faldu ⋅ Prannay Hebbar ⋅ Jiankai Sun ⋅ Christopher Settles ⋅ Omkaar Kamath ⋅ Pratyush Shukla ⋅ Albert Liu ⋅ Yiyuan Li ⋅ Nevasini Sasikumar ⋅ Xiangyi Li ⋅ Pranav Raja ⋅ Ishan Gupta ⋅ Marek Suppa ⋅ Daniel Wang ⋅ Erik Quintanilla ⋅ Derek Chen ⋅ Chris Kong ⋅ Steven Dillmann ⋅ Ivan Bercovich ⋅ Jesse Hu
AI agents are increasingly expected to complete long-horizon workflows that require sustained progress over hours, millions of tokens, and complex environments. Yet current agent benchmarks largely evaluate short-form tasks, such as single pull requests, small tickets, or 5--10 minute exercises, limiting our ability to measure agents' capabilities in planning, long-context understanding, and memory use. We introduce SWE-Marathon, a benchmark of 20 long-horizon tasks spanning software engineering and adjacent technical domains. Each task consists of a unique executable environment, a human-written reference solution, and a multi-layer verification suite. Logged agent attempts average 27.2M total tokens, making SWE-Marathon substantially longer-horizon than existing SWE and command-line agent benchmarks. Current frontier coding agents solve fewer than 20\% of tasks. Failures often arise from poor self-verification, self-reported infeasibility, and premature termination. We also observe reward-hacking behavior in 18.8\% of rollouts, where agents attempt to exploit the environment or verifier to bypass the intended workflow. SWE-Marathon includes adversarial review of test suites and execution environments, as well as multi-layer checks designed to prevent shortcut solutions. We release SWE-Marathon and its evaluation code to support reproducible measurement of long-horizon agent capabilities.
Show more
WorldSpeech: A Multilingual Speech Corpus from Around the World
Antonis Asonitis ⋅ Luca Lanzendörfer ⋅ Frédéric Berdoz ⋅ Roger Wattenhofer
Automatic speech recognition (ASR) performs well for high-resource languages with abundant paired audio-transcript data, but its accuracy degrades sharply for most languages due to limited publicly available aligned data. To this end, we introduce WorldSpeech, a 24kHz multilingual speech corpus comprising 65k hours of aligned audio-transcript data across 76 languages, collected from diverse public sources including parliamentary proceedings, international broadcasts, and public-domain audiobooks. For 37 languages, WorldSpeech provides more than 200 hours of aligned speech, with 28 exceeding 500 hours and 24 surpassing 1k hours. Fine-tuning existing ASR models on WorldSpeech results in an average relative Word-Error-Rate reduction of 63.5% across 11 typologically diverse languages.
Show more
PitchBench: Measuring Pitch Hearing in Audio-Language Models
Milan Liessens Dujardin ⋅ Song-Ze Yu ⋅ Craver C Thomas-Smith ⋅ David Chan ⋅ Karina Nguyen
Audio-language models (ALMs) are increasingly used in real-world applications that require understanding music, from music tutoring and transcription to captioning, recommendation systems, and music production. More broadly, they are becoming an important component of multimodal AI systems that must reason from sensory input rather than text alone. This makes reliable musical perception a critical prerequisite: if a model cannot accurately hear the structure of sound, it cannot be trusted to reason about, teach, transcribe, or act on audio in the real world. Yet existing benchmarks rarely assess one of the most fundamental musical abilities underlying such perception: pitch hearing. Current evaluations tend to probe pitch hearing only indirectly, through higher-level tasks and often in multiple-choice formats, leaving open how reliably ALMs identify fine-grained pitch across instruments, acoustic conditions, and response formats. We introduce PitchBench, an evaluation suite that systematically measures pitch hearing in ALMs. PitchBench comprises 28 experiments spanning absolute and relative pitch perception within sequences and chords, while varying loudness, note duration, sound source, time stretching, background noise, and other acoustic conditions. Tasks range from identifying individual pitches in isolation to tracking a melodic line within a four-part musical texture. Evaluating frontier ALMs, we find that pitch hearing remains highly unreliable: models perform consistently poorly across settings, with accuracy varying sharply by sound source, note duration, and notation format. Current ALMs do not yet possess stable pitch perception, even for controlled synthetic and instrumental stimuli. Alongside the benchmark, we release PitchBench as a Python package containing the evaluation data and data generation tools to support future work on pitch-aware audio-language modeling.
Show more
CruxBench: A Benchmark of Information Discovery
Lina Piao ⋅ Hui Dai ⋅ Nick Merrill ⋅ Nadja Flechner ⋅ Ezra Karger ⋅ Haifeng Xu
Benchmarks for large language models typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first place: decomposing a difficult problem into subquestions whose answers provide key steps on the path toward solving the larger problem. To evaluate this capability of information discovery, we introduce CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed subquestion updates beliefs about a target outcome. To automate this measurement at scale, we target as outcomes the prices in continuously updating forward-looking prediction markets. Our benchmark has three rare properties: it is contamination-resistant by construction, since ground truth is generated by future world events; it is open-ended, admitting unbounded and complex text-based submissions rather than one correct numeric answer; and it is grounded, with informativeness measured against quantified changes in real-world beliefs (prices). We evaluate eight frontier and open-weight models on 336 target outcomes and find that VOI correlates highly with independent measures of model quality ($r=0.87$) and predicts downstream usefulness when the generated questions are passed to a separate forecasting agent.
Show more
Prefix Executability: Evaluating Tool-Using Agents Beyond Final Success
Amir H Rezaeian ⋅ Weiyi Sun ⋅ Yassine Benajiba ⋅ Dan Roth
Tool-use benchmarks often score agents by final task success, but terminal outcomes can obscure whether the intermediate tool-call trajectory was executable. We propose prefix executability as a trajectory-level evaluation protocol for tool-using agents. The protocol measures whether every prefix of a predicted tool-call sequence remains executable under tool schemas, state preconditions, and data dependencies, and summarizes reliability using Prefix Executability Curves (PEC), First-Error Depth (FED), Trajectory Executability Rate (TER), horizon-normalized AUC-PEC, and Recovery Rate (RR). We instantiate the protocol on stratified samples from BFCL-v3 multi-turn and ToolSandbox. Across both benchmarks, outcome-only and prefix-level rankings diverge: agents whose execution is conditioned on an unverified plan often achieve higher point-estimate final success than non-plan-conditioned baselines while substantially lowering TER and AUC-PEC. These divergences are precisely the failure modes that prefix executability is designed to expose: they reveal whether apparent success rests on an executable trajectory, where invalidity first enters, and whether success follows recovery from an earlier failure. As a validation intervention, explicit pre-execution plan checking reverses much of this prefix-reliability loss, consistently improving executable-prefix survival while maintaining competitive terminal success. These results show that final success alone is insufficient for evaluating tool-use agents and that prefix-level executability provides an actionable reliability signal for comparing agents, diagnosing failures, and assessing validation mechanisms.
Show more
How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants
Xueyang Feng ⋅ Weinan Gan ⋅ Xu Chen ⋅ Quanyu Dai ⋅ Yong Liu
Large language model (LLM)-powered assistants have recently integrated memory mechanisms that record user preferences, leading to more personalized and user-aligned responses. However, irrelevant personalized memories are often introduced into the context, interfering with the LLM's intent understanding. To comprehensively investigate the dual effects of personalization, we develop \textsc{RPEval}, a benchmark comprising a personalized intent reasoning dataset and a multi-granularity evaluation protocol. \textsc{RPEval} reveals the widespread phenomenon of irrational personalization in existing LLMs and, through error pattern analysis, illustrates its negative impact on user experience. Finally, we introduce \textsc{RP-Reasoner}, which treats memory utilization as a pragmatic reasoning process, enabling the selective integration of personalized information. Experimental results demonstrate that our method significantly outperforms carefully designed baselines on \textsc{RPEval}, and resolves 80\% of the bad cases observed in a large-scale commercial personalized assistant, highlighting the potential of pragmatic reasoning to mitigate irrational personalization. Our data is available at \url{https://anonymous.4open.science/r/RPEval-E4B0}.
Show more
SoDeArena: A Socially-Situated Reasoning Benchmark for Large Language Models
Senhao Yang ⋅ Ziling Yuan ⋅ Xiaoran Yang ⋅ Yaqing Wang ⋅ Runpeng Xie ⋅ Chen Huang ⋅ Lexiang Wang ⋅ Shuang Xu ⋅ Yiqin Yang ⋅ Bo Xu
In recent years, Large Language Models (LLMs) have demonstrated remarkable performance in tasks such as mathematical reasoning and code generation. However, as LLMs are increasingly deployed as autonomous agents, their reasoning capabilities in dynamic, human-like social interactions remain largely unexplored. To bridge this gap, we propose the benchmark SoDeArena—Social Deduction Arena, designed to evaluate the socially-situated reasoning capabilities of LLMs through a diverse set of interactive games. Through large-scale multi-agent simulations and human interaction experiments, we have identified three key insights for improving current LLMs: (1) the overall proficiency of models in social deduction is fundamentally bottlenecked by model scale and reasoning capabilities; (2) models lack proactive intent management, defaulting instead to strategic passivity and output conformity during multi-agent interactions; and (3) LLMs exhibit dual behavioral biases, manifesting as a rigid safety alignment in adversarial roles and a pathological reliance on shallow positional patterns over robust reasoning processes. These findings point to potential directions for the future development of AI systems with robust socially-situated reasoning capabilities.
Show more
PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language Models
Inpyo Song ⋅ Eunji Chon ⋅ Jangwon Lee
Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, including software development, education, and technical assistance. However, a fundamental question remains unresolved: can LLMs reason about physical constraints—the rules governing how abstract computations must map onto real-world hardware? This capability is distinct from, and arguably more demanding than, software generation alone, as it requires grounding language understanding in spatial, electrical, and topological constraints. To address this gap, we introduce PCEval (Physical Computing Evaluation), a benchmark and evaluation protocol for physical computing education that enables fully automatic, execution-based assessment of LLMs across both logical and physical implementation artifacts, without human grading. Our evaluation framework assesses LLMs in generating circuits and producing compatible code across varying levels of project complexity. Through comprehensive testing of 14 leading models, PCEval provides the first reproducible and automatically validated empirical assessment of LLMs' ability to reason about fundamental hardware implementation constraints within a simulation environment. Our findings reveal that while LLMs perform well in code generation and logical circuit design, they struggle significantly with physical breadboard layout creation, particularly in managing proper pin connections and avoiding circuit errors. PCEval advances our understanding of AI assistance in hardware-dependent computing environments and establishes a foundation for developing more effective tools to support physical computing education.
Show more
DriveSpatial: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving
Anh Hao Vo ⋅ Khoa Vo ⋅ Phu Loc Nguyen ⋅ Sieu Tran ⋅ Duc Nguyen ⋅ Cuong Ngo ⋅ Gladys Gawugah ⋅ Sreevenkata A Godavarthi ⋅ Chase Rainwater ⋅ Nghi Bui ⋅ Anh Nguyen ⋅ Duy M. H. Nguyen ⋅ Ngan Le
Spatiotemporal intelligence in autonomous driving (AD) requires an agent to integrate multi-view observations into a coherent scene representation, maintain object continuity across viewpoints and time, and reason about spatial relations, interactions, and future dynamics. However, existing AD vision-language benchmarks largely focus on single-view, static, ego-centric, or single-source question answering, leaving it unclear whether current Vision-Language Models (VLMs) can truly construct and reason over dynamic driving scenes. We introduce DriveSpatial a benchmark of 15.6K human-verified QA pairs across 20 tasks from five large-scale AD datasets. DriveSpatial evaluates four abilities: Cognitive Scene Construction, Multi-view Relational Understanding, Temporal Reasoning, and Generalization. Unlike prior benchmarks, DriveSpatial is generated from a dynamic multi-relational scene graph that encodes object states, spatial relations, interactions, camera visibility, and temporal correspondences, enabling QA pairs that enforce genuine cross-view and spatiotemporal reasoning. Evaluating 15 representative VLMs reveals a substantial human–model gap: the strongest model trails humans by 28.4 points, with Cognitive Scene Construction emerging as the key bottleneck. Further diagnostics show that language-only prompting is insufficient, while explicit BEV grounding consistently improves performance. These results suggest that current VLMs lack the scene-construction ability needed for reliable spatiotemporal driving intelligence. DriveSpatial and its construction pipeline will be released to support future research.
Show more
Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving
Aron Distelzweig ⋅ Faris Janjos ⋅ Andreas Look ⋅ Anna Rothenhäusler ⋅ Daniel Jost ⋅ Oliver Scheel ⋅ Raghu Rajan ⋅ Daphne Cornelisse ⋅ Eugene Vinitsky ⋅ Joschka Boedecker
Recent Autonomous Driving (AD) works such as GigaFlow and PufferDrive have unlocked Reinforcement Learning (RL) at scale as a training strategy for driving policies. Yet such policies remain disconnected from established benchmarks, leaving the performance of large-scale RL for driving on standardized evaluations unknown. We present BehaviorBench -- a comprehensive test suite that closes this gap along three axes: Evaluation, Complexity, and Behavior Diversity. In terms of Evaluation, we provide an interface connecting PufferDrive to nuPlan, which, for the first time, enables policies trained via RL at scale to be evaluated on an established planning benchmark for autonomous driving. Complementarily, we offer an evaluation framework that allows planners to be benchmarked directly inside the PufferDrive simulation, at a fraction of the time. Regarding Complexity, we observe that today's standardized benchmarks are so simple that near-perfect scores are achievable by straight lane following with collision checking. We extract a meaningful, interaction-rich split from the Waymo Open Motion Dataset (WOMD) on which strong performance is impossible without multi-agent reasoning. Lastly, we address Behavior Diversity. Existing benchmarks commonly evaluate planners against a single rule-based traffic model, the Intelligent Driver Model (IDM). We provide a diverse suite of interactive traffic agents to stress-test policies under heterogeneous behaviors, beyond just using IDM. Overall, our benchmarking analysis uncovers the following insight: despite learning interactive behaviors in an emergent manner, policies trained via pure self-play under standard reward functions overfit to their training opponents and fail to generalize to other traffic agent behaviors. Building on this observation, we propose a hybrid planner that combines a PPO policy with a rule-based planner, providing a baseline for our new benchmark. Code is available at https://anonymous.4open.science/r/behavior-bench-707C
Show more
LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation
Bo Miao ⋅ Weijia Liu ⋅ Jun Luo ⋅ Lachlan Shinnick ⋅ Jian Liu ⋅ Thomas Hamilton-Smith ⋅ Yuhe Yang ⋅ Zijie Wu ⋅ Vanja Videnovic ⋅ Feras Dayoub ⋅ Anton van den Hengel
Language-conditioned goal navigation (LGN) requires embodied agents to locate user-specified targets without step-by-step guidance. However, existing benchmarks largely focus on category-level goals or rely on instance descriptions generated by vision-language models, which often contain ambiguities and semantic errors, limiting systematic and reliable evaluation. We introduce HieraNav, an open-vocabulary LGN task with goals at four hierarchical semantic levels: scene, room, region, and instance. We present Language as a Map (LangMap), the first LGN benchmark to enrich real-world indoor 3D scans with human-verified semantic annotations supporting tasks across all four goal levels. Built on HM3D using a contrastive annotation protocol that compares same-scene regions and instances, LangMap provides region labels and discriminative region and instance descriptions covering 414 object categories and contains over 18K tasks. Each target has concise and detailed descriptions, enabling evaluation across instruction styles. Automated and human evaluations validate our annotation quality: our descriptions improve text-to-view matching accuracy over GOAT-Bench's by 23 points on all shared annotated instances, and an independent human audit yields 92.5% unique-and-correct matches. We also propose PlaNaVid, an RGB-only baseline that combines Bounded Diverse Memory with high-level planning to prime a reactive policy for multi-goal navigation, achieving top-tier success rates without depth, 3D scene representations, or object masks. Further analyses reveal that exploration and hierarchical disambiguation failures become more prominent at finer goal levels, while long-tail categories, small objects, distant targets, timely stopping, and multi-goal completion remain challenging.
Show more
NavAble: A Large-Scale Dataset and Synthetic Data Generation Pipeline for Blind Navigation
Hochul Hwang ⋅ Jahir Sadik Monon ⋅ Soowan Yang ⋅ Shiven U Patel ⋅ Anh N Nguyen ⋅ Kien Nguyen ⋅ Keshav Garg ⋅ Dylan Gage ⋅ Duretti Hordofaa ⋅ Anshu Anjna ⋅ Eshed Ohn-Bar ⋅ Donghyun Kim
Reliable recognition of accessibility-critical objects (e.g., audible pedestrian signals, door-activation buttons, and handrails) is essential for assistive navigation technologies that support safe, independent mobility for blind and low-vision (BLV) users. Yet these categories are severely underrepresented in existing large-scale vision datasets, and the few datasets that include them suffer from limited class coverage, inconsistent annotations, and poor diversity, limiting the perception reliability that BLV navigation requires. We introduce NavAble, a large-scale dataset spanning 11 accessibility-critical object classes, combining 41K curated open-source images with 8K newly collected, densely annotated real-world images. To extend scale and diversity, we develop a synthetic data generation pipeline that renders high-fidelity 3D assets across 37 environments under diverse styles and viewpoints, yielding 452K frames with rich ground-truth annotations from 565 assets. The pipeline minimizes manual effort by combining automated web image crawling, VLM-based filtering, SAM 3D-based asset generation, and convenient camera trajectory configuration, enabling large-scale data generation at a diversity unattainable through manual collection. Experiments show that fine-tuning existing segmentation models on the NavAble Dataset achieves reliable accessibility-object segmentation (>80\% mIoU on our held-out test set). We also demonstrate the effectiveness of synthetic data augmentation, which yields up to +3.5 mIoU over real-world-only fine-tuning. Beyond benchmark evaluation, validation on egocentric data from two mobility-assistive robots demonstrates improved perception reliability in real-world navigation scenarios. Together, NavAble establishes a scalable foundation for BLV navigation perception, directly addressing the data scarcity and embodiment gaps that have limited progress in this domain. The dataset and pipeline are publicly available at https://huggingface.co/datasets/NavAble/NeurIPS_2026_BLV.
Show more
CineMME: Benchmarking Fine-Grained Perception and Plot Reasoning in Multimodal Large Language Models
Jin Liu ⋅ Dawei Du ⋅ Yexiang Liu ⋅ Sijie Zhu ⋅ Fan Chen ⋅ Junxian Duan ⋅ Huaibo Huang ⋅ Ran He ⋅ Longyin Wen ⋅ Zhenfang Chen
Cinematic understanding requires more than recognizing visible events: models must acquire dialogue evidence, bind it to speakers, and use it to infer narrative meaning. Existing video benchmarks evaluate pieces of this problem, including video question answering, audio-visual reasoning, and temporal grounding, but rarely test whether these abilities form a coherent cinematic evidence chain within the same clips. As a result, a model may answer a movie question correctly while failing to localize the relevant utterance, identify the speaker, or integrate speech with visual context. We introduce **CineMME**, a human-verified benchmark designed to diagnose speech-grounded cinematic reasoning, supported by approximately $1,600$ expert person-hours of annotation. CineMME contains $975$ curated movie clips and two linked tracks: *CineMME-Grounding*, with $13,554$ timestamped speech segments and $33,083$ actor face boxes for temporal dialogue localization, transcription, and active-speaker grounding; and *CineMME-Reasoning*, with $1,600$ multiple-choice and $316$ open-ended questions spanning perception, audio, narrative, social, and cinematic understanding. We evaluate $20$ MLLMs on reasoning and $10$ omni-modal models on grounding. The results reveal a pronounced who-what-when decoupling: Gemini-3 Pro localizes dialogue in time 68.33% tIoU) and partially transcribes it (WER 29.00%), yet reaches only 13.24% sIoU for active-speaker grounding while achieving 65.88% reasoning accuracy. Modality ablations further show that vision is not uniformly beneficial: it improves perception-heavy questions but can hurt dialogue-centric reasoning. CineMME provides a compact diagnostic testbed for measuring whether multimodal video models truly connect dialogue, speakers, visual evidence, and narrative meaning, rather than succeeding through disconnected cues.
Show more
IndustryCode: A Benchmark for Industry Code Generation
Puyu Zeng ⋅ Zhaoxi Wang ⋅ Zhixu Duan ⋅ Liang Feng ⋅ Shaobo Wang ⋅ Cunxiang Wang ⋅ jinghang Wang ⋅ 必应 赵 ⋅ HU WEI ⋅ Qibing Ren ⋅ Linfeng Zhang
Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of NAICS sectors, including Aerospace Product Mfg, Chemical Mfg, and Information, and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.7 Opus, achieved an overall accuracy of 71.9\% on sub-problems and 48.9\% on main problems. The benchmark dataset and automated evaluation code are now publicly available.
Show more
Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering with Generative Optimization
Dapeng Jiang ⋅ Yizhe Chi ⋅ Kaisen Yang ⋅ Tianwei Luo ⋅ Boshi Zhang ⋅ Deyao Hong ⋅ Dianqiao Lei ⋅ Yifan Zhou ⋅ Weiyang Jin ⋅ Xiaoyan Fan ⋅ Han Hao ⋅ Zhe Cao ⋅ Qian Houde ⋅ Qingle Liu ⋅ Bohan Lyu ⋅ Bowen Wang ⋅ Situ Wang ⋅ Youjie Zheng ⋅ Bingxiang He ⋅ Eren Cai ⋅ Calvin Xiao ⋅ Qinhuai Na
Engineering value is often created not by producing a single correct answer, but by improving a feasible artifact under hard constraints. We introduce **Frontier-Eng**, a benchmark for generative optimization agents that repeatedly edit executable engineering artifacts, receive frozen verifier feedback, and maximize the best feasible score found within a fixed budget. Frontier-Eng contains $47$ tasks across five engineering categories, spanning kernels, cryptography, quantum circuits, inventory and scheduling, robotics and control, optics and communication systems, structural design, reactions, and sustainable infrastructure. Each task provides a feasible starting solution, an editable artifact, and a read-only evaluator with verifier-parsed scoring, enabling cross-domain comparison without reducing engineering quality to binary pass/fail completion. We evaluate frontier models under OpenEvolve and compare search frameworks across the full task set. Current agents exhibit clear iterative improvement: **gpt-5.4** obtains the best average rank under OpenEvolve. Long-horizon runs reveal a dual **inverse-law pattern**: improvement events become rarer with iteration, and improvement magnitudes shrink with improvement count. At the same time, improvement is uneven across domains and remains sensitive to search depth, exposing a concrete frontier for engineering-oriented agents under realistic constraints and verifier-grounded feedback.
Show more
The BAMBI Dataset: Multimodal Nadir UAV-Recordings of Forest Wildlife
Christoph Praschl ⋅ Hugo Markoff ⋅ Anna Maschek ⋅ Wolfram Jantsch ⋅ Stephanie Wohlfahrt ⋅ Anton H Jørgensen ⋅ Christian E Mogensen ⋅ Mathias B Skadhauge ⋅ Sara Beery ⋅ Michael Ørsted ⋅ David C Schedl
Large-scale wildlife monitoring in forested environments using drones remains challenging due to occlusion, limited visibility, and scarce annotated data. We present a comprehensive airborne wildlife dataset comprising 386 paired RGB and thermal aerial video sequences recorded across diverse temperate forests and forest-adjacent habitats in Austria. Each frame is geo-referenced with precise global coordinates (longitude, latitude and altitude), enabling learning and evaluation in both image space and geographic space. The dataset includes 10 animal species captured under varying seasonal, environmental, and illumination conditions, with annotations supporting tasks such as object detection, multi-object tracking and fine-grained classification. The multispectral data and spatial metadata enable research on world-coordinate trajectory analysis, spatial population modeling, geo-aware perception, and advanced methods such as Airborne Light Field Sampling. Thermal subsets of this dataset have been used to develop and validate methods for wildlife monitoring, while the RGB data has mainly stayed untouched. To address this, we present a cross-modal pipeline to transfer thermal annotations to RGB frames next to the dataset. With this dataset we aim to provide a blueprint to promote research on multimodal, geo-referenced perception in ecology.
Show more
Robustness of Internal Signals for Hallucination Detection in Vision-Language Models
Allison Chen ⋅ William Yang ⋅ Salma Abdel Magid ⋅ Jonathan Williams ⋅ Olga Russakovsky ⋅ Esin Tureci
When given an image and a text prompt, vision-language models (VLMs) can produce seemingly sophisticated text responses. However, they are also prone to hallucinate objects in their output, which can have detrimental consequences in safety-critical applications. To detect these hallucinations, certain methods interestingly leverage internal signals from the VLM as it generates the text response; potentially, these methods use the same signals that led to a hallucination to detect it. We conduct an in-depth analysis to understand the robustness of these detection methods. First, we show that a naive baseline of a generated word's \emph{token position} in the VLM text response achieves competitive performance. Further, we reveal how some detection methods \emph{rely} on token position and use this heuristic as a shortcut. Second, we explore detection methods' sensitivity to language and vision modalities. Our experiments suggest that strong detection performance is often driven by detecting language-based hallucinations, and when methods rely on token position, they tend to be less visually grounded and exhibit reduced robustness when hallucinations occur earlier in a text response. Lastly, we demonstrate that different detection methods can carry complementary signals; when ensembling methods together, adding weaker methods to an ensemble of higher performing ones still offers performance improvements. Altogether, while there may be useful internal VLM signals to detect hallucinations, our investigation reveals important limitations of the current methods that try to leverage them.
Show more
scShapeBench: Discovering geometry from high dimensional scRNAseq data
Andrew J Steindl ⋅ Joao Felipe Rocha ⋅ Brian T Di Bassinga ⋅ Zachary Warren ⋅ Matthew Scicluna ⋅ César Miguel Valdez Córdova ⋅ Shabarni Gupta ⋅ Leire Torices ⋅ Daniel Neumann ⋅ Timothy J Mann ⋅ Ihuan Gunawan ⋅ Dhananjay Bhaskar ⋅ John G Lock ⋅ Christine L Chaffer ⋅ Guy Wolf ⋅ Smita Krishnaswamy
High-dimensional point cloud data arise across many scientific domains, notably single-cell biology. The "shapes" or topologies of these datasets are informative of types of information that can be extracted from the datasets. For example clustered data admits the extraction of cell types or cell states in a static analysis of the datasets. Continuous trajectory structures admit continuous transition or trajectory analysis, while other shapes such as archetypal shapes admit continuum extraction with a range of cells spanning behaviors. While analysis pipelines exist, they often presuppose shape in data. For example, the standard Seurat pipeline combines UMAP visualization with Louvain clustering. This assumes clustered data. Tools like Monocle and Spade assume a tree-like shape, and flow-models like MIOFlow and Conditional Flow Matching are suitable for trajectories. Deciding which pipeline to apply to which part of the data is often the realm of bioinformaticians who visualize and qualitatively analyze the data before selecting one. However, with the advent of agentic AI scientists, it becomes important to automate data shape detection, particularly into categories that are relevant for downstream analysis pipelines. Towards this end we introduce \textsc{scShapeBench} a benchmark dataset comprising both synthetic and single-cell expert-annotated datasets that are meant for the task of shape detection. Synthetic datasets are sampled from a ground truth “skeleton graph” with variance. Real single-cell datasets are curated from a variety of sources and are annotated by experts, classifying four categories; clusters, single trajectory, multi-branches and archetypes. In addition, we provide a baseline method, scReebTower, to bridge the gap between data visualization and pipeline selection. scReebTower relies on the diffusion geometry to extract Reeb graphs. We provide new topology-aware metrics with which we evaluate scReebTower and existing methods PAGA and Mapper on synthetic data. On single-cell data we curate expert annotations of shapes, and showcase evaluations of methods. Our comparisons indicate scReebTower outperforming other baselines. Overall, our contributions span benchmarks, evaluation metrics, and a novel baseline method for automated shape detection in high-dimensional single-cell data.
Show more
Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences
Benjamin Walker ⋅ Terry Lyons
World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-World-Model, a large-scale state-tracking benchmark built from $10$ million real chess games, where models predict the exact board state reached after a sequence of legal moves. Alongside a held-out real-game split, we include an out-of-distribution split from uniformly random legal play, which tests whether models learn the transition rules rather than shortcuts from common human positions. Prior theoretical and empirical work has shown that Transformers struggle to state-track, while input-dependent linear RNNs require expressive state-transition matrices to do so. We therefore benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues under a matched interface and training protocol. The recurrent models strongly outperform the Transformer at $3$ and $8$ million parameters. Real-game performance saturates above $18$ million parameters, but the random-uniform split remains discriminative up to $38$ million, exposing failures otherwise hidden by scale. Additionally, ablations show that less expressive state-transition mechanisms reduce performance on the out-of-distribution split for all three recurrent models. Together, these results establish Chess-World-Model as a practical large-scale benchmark for state tracking that exposes failures model scale would otherwise conceal.
Show more
MathCD: A Benchmark Dataset for Cognitive Diagnosis with Semantic Information
Xueyi Li ⋅ Youheng Bai ⋅ Tengteng Cheng ⋅ Mingliang Hou ⋅ Teng Guo ⋅ Jiaqi Zheng ⋅ Yongdong WU ⋅ Zitao Liu
Cognitive diagnosis (CD) aims to infer students' knowledge states from historical learning interactions, providing essential support for personalized education. Despite recent progress, most existing CD datasets mainly represent students, exercises, and knowledge concepts as discrete identifiers, offering limited semantic information about exercise text, answer choices, and student responses. This limitation restricts the development of semantic enhanced CD models and makes it difficult to evaluate whether foundation large language models (LLMs) can demonstrate educational understanding in CD scenarios. To address this gap, we introduce \emph{MathCD}, a new K-12 mathematics CD dataset collected from an online learning platform. MathCD contains three subsets with $2,197$ students, $46,431$ exercises, $3,258$ knowledge concepts, and $106,220$ student-exercise interactions. Moreover, MathCD provides rich semantic annotations, including exercise text, KC text, answer text, response text, exercise type, and exercise difficulty. Based on MathCD, we evaluate $9$ existing CD models and $5$ representative foundation LLMs. Experimental results show that semantic information consistently improves both classical and neural CD models, with semantic-enhanced variants achieving comparable or better performance than strong identifier based models. We further find that foundation LLMs exhibit emerging CD-related abilities in student response prediction and exercise difficulty discrimination, but their performance remains unstable and limited without task-specific adaptation. MathCD provides a new benchmark for semantic-enhanced CD and foundation LLMs evaluation in educational modeling.
Show more
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
Xiang Feng ⋅ Jiawei Zhou ⋅ Zhangfeng Huang ⋅ Kewei Wang ⋅ Shanshan Ye ⋅ Jinxin Hu ⋅ Zulong Chen ⋅ Yong Luo ⋅ Jing Zhang
Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, supporting evidence regions, relevant factual statements, and a final answer. We design a four-stage evaluation protocol—Page Localization, Region Grounding, Fact Extraction, and Answer Verification—that audits each level of the trajectory independently through inter-stage decoupling, with all judges selected and calibrated via human alignment studies. DocScope comprises 1,124 questions derived from 273 documents, with all hierarchical evidence annotations completed by human annotators. We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems. Our experiments reveal that answer accuracy cannot substitute for trajectory-level evaluation: even among correct answers, the highest observed rate of complete evidence chains is only 29\%. Across all models, region grounding remains the weakest trajectory stage. Furthermore, the primary difficulty stems from aggregating evidence dispersed across long distances and multiple document clusters, while an oracle study identifies faithful perception and fact extraction as the dominant capability bottleneck. Cross-architecture comparisons further suggest that activated parameter count matters more than total scale. The benchmark and code will be publicly released.
Show more
Strategic Evaluation: Incentivizing AI Capability Coverage with Private Benchmarks
Sang Truong ⋅ Serena Wang ⋅ Nick Haber ⋅ Sanmi Koyejo
Public benchmarks play a significant role in steering LLM development, as market incentives drive model developers to optimize for leaderboard performance. However, fundamental information limitations mean that evaluators are only able to cover a subset of socially relevant capabilities with their evaluation tasks. On the other hand, model developers often have additional private benchmarks that are unavailable to evaluators. The resulting information asymmetry opens the door to strategic task specialization that can lead to suboptimal social welfare. We propose randomized evaluation mechanisms (effectively private benchmarks) as a way to incentivize model developers to train to cover a broader set of socially relevant tasks, including those unknown to the evaluator. We formalize this as a dynamic game with information asymmetry between an evaluator and a model developer. We prove that randomized evaluation aligns incentives to the developer’s prior belief over possible evaluated tasks, and is socially optimal when that belief matches society, but information leakage degrades this over rounds. However, if the evaluator can continually update their known task set, alignment can be asymptotically recovered. We illustrate a variance–leakage–correction tradeoff in semisynthetic experiment with a latent factor structure over MMLU-Pro.
Show more
PanoHK360: A Large-Scale 8K Urban Panoramic Dataset and Benchmark for Depth Estimation
Yunxiao Chen ⋅ Honghuan Lin ⋅ Ruisheng Wang ⋅ Yujun Liu ⋅ Kun Zhou ⋅ Shangfeng Huang ⋅ Tsz Nam Chan
Panoramic 360° depth estimation underpins autonomous navigation, 3D scene reconstruction, and scene understanding. However, existing outdoor panoramic datasets remain limited in three aspects. Small-scale collections are insufficient for modern data-hungry architectures and tend to produce models that generalize poorly across cities. Synthetic datasets transfer poorly to real scenes due to persistent domain gaps in illumination, geometry, and material appearance. Pseudo-label datasets distilled from pretrained estimators inherit the errors of the teacher model. Their downstream accuracy is therefore capped at the performance ceiling of the teacher itself. To address these issues, we present PanoHK360, a city-scale outdoor panoramic RGB-D dataset of diverse urban scenes in Hong Kong. The data are collected by a vehicle-mounted rig that pairs 8K panoramic imaging with survey-grade LiDAR. To the best of our knowledge, PanoHK360 is the first panoramic depth dataset that simultaneously combines three defining properties: city-scale geographic coverage, 8K equirectangular resolution, and sensor-based metric depth. The depth is recovered directly from LiDAR rather than estimated by a pretrained network. The release further includes raw point clouds, 6-DoF camera poses, and temporal capture sequences. We further benchmark representative monocular 360° depth methods on PanoHK360 under a unified protocol with a location-disjoint training and validation split. The results reveal a substantial gap between current methods and the level of resolution and scene diversity required for real-world urban deployment. These findings establish PanoHK360 as a challenging testbed for future research.
Show more
EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries
Jiyoun Kim ⋅ Muhan Yeo ⋅ Eunhye Jang ⋅ Jeewon Yang ⋅ Hangyul Yoon ⋅ Su Ji Lee ⋅ Hee J Han ⋅ Hee-Jae Jung ⋅ Doyun Kwon ⋅ Jun y Lee ⋅ Jaehun Lee ⋅ Jung-Oh Lee ⋅ Sunjun Kweon ⋅ Jong-Hak Moon ⋅ Daseul Kim ⋅ Minjae Cho ⋅ Edward Choi
Discharge summaries are crucial clinical documents for patient care, containing clinical context of a patient's overall admission, and are routinely reviewed by medical experts during patient readmission, ongoing care, and diagnostic decision-making. In practice, medical experts often must synthesize information across multiple lengthy discharge summaries by iteratively obtaining information from the documents, while also verifying the evidence supporting each answer. Although large language models (LLMs) are increasingly explored for clinical document question answering, existing benchmarks do not sufficiently capture this clinical setting: they either evaluate exam-style medical knowledge or focus on single-turn question answering with limited evidence-grounding evaluation. We introduce **EHRNote-ChatQA**, the first benchmark for evidence-grounded multi-turn clinical question answering over patients' multiple discharge summaries. Built from de-identified MIMIC-IV discharge summaries, EHRNote-ChatQA contains 967 patient-level multi-turn samples spanning one to five notes and 16,072 medical-expert-verified QA pairs (8,036 content questions, each paired with an evidence-grounding question) across eight clinical categories. We construct the benchmark through an expert-informed pipeline that combines a comprehensive discharge-summary structuring schema, expert-curated multi-turn QA templates, and LLM-based generation. Every single QA sample is then reviewed and revised by 11 medical experts over three weeks. Benchmarking 22 LLMs (both open and closed-sourced) reveals several findings; evidence grounding is consistently harder than content answering, multi-turn errors compound across turns, and strong performance on existing single-turn clinical QA benchmarks does not reliably transfer to this setting. These results establish EHRNote-ChatQA as a rigorous and practical benchmark for evaluating clinical QA systems.
Show more
Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
Amit Roth ⋅ Ankur Samanta ⋅ Matan Halevy ⋅ Yoav Levine ⋅ Yonathan Efroni
Aligning autonomous agents with human intent remains a central challenge in modern AI. A key manifestation of this challenge is reward hacking, whereby agents appear successful under the evaluation signal while violating the intended objective. Reward hacking has been observed across a wide range of settings, yet methods for reliably measuring it at scale remain lacking. In this work, we introduce a new evaluation paradigm for measuring reward hacking. Whereas prior studies have primarily analyzed it post hoc by inspecting agent trajectories, we instead embed detectable reward hacking opportunities directly into environments. This makes their exploitation verifiable by design, enabling deterministic and automated measurement of whether and how agents exploit such vulnerabilities. We instantiate this approach in TextArena and release Hack-Verifiable TextArena, a testbed in which reward hacking can be measured reliably. Using this benchmark, we analyze reward hacking behavior across language models in diverse environments and settings.
Show more
ColorConceptBench: A Benchmark for Probabilistic Color-Concept Understanding in Text-to-Image Models
Chenxi Ruan ⋅ Yihan Hou ⋅ Yu Xiao ⋅ Guosheng Hu ⋅ Wei Zeng
Text-to-image (T2I) models have advanced considerably in generating high-quality images from textual descriptions. However, their ability to associate colors with concepts remains largely constrained to explicit color names or codes, while their capacity to handle implicit concepts, such as emotions and visual states, remains underexplored. To address this gap, we introduce ColorConceptBench, an expert-annotated benchmark that systematically evaluates color-concept associations through probabilistic color distributions. ColorConceptBench moves beyond explicit color specifications by examining how models interpret 1,281 implicit color concepts, grounded in 6,584 human annotations. Our evaluation of nine leading T2I models reveals that performance varies substantially across semantic categories, and models exhibit a significant lack of sensitivity to abstract semantics. These limitations persist even when applying classifier-free guidance scaling at inference time, suggesting that achieving human-like color understanding demands a shift in how models learn and represent implicit semantic meaning.
Show more
Towards Reliable VLM Judges: State-Conditional Invariance and Presentation-Aware Diagnostics
Qinan Zhang ⋅ Xinyu Wang ⋅ Qihang Jin ⋅ Tianyuan Wang ⋅ Jian Di ⋅ Hongbing Luo ⋅ Pengfei Li ⋅ Wenjun Lv ⋅ Yu Kang
Vision-language models (VLMs) are increasingly used as automated judges for multimodal and embodied tasks, yet aggregate accuracy or consistency alone does not reveal whether a judge changes its verdict for the right reason. We propose State-Conditional Invariance (SCI), a reliability principle requiring a judge to remain stable under semantics-preserving presentation changes while responding to causal interventions that change the task outcome. We instantiate SCI in GridWM-Judge, a simulator-grounded diagnostic benchmark built from deterministic MiniGrid trajectories with aligned Full, evidence-removal, and counterfactual variants, plus presentation and representation probes. Across five diagnostic tasks, GridWM-Judge tests local perception, action-conditioned transitions, presentation stability, representation exchangeability, and Full-CF causal discrimination. Evaluating current VLM judges reveals separable failures, including presentation sensitivity, representation fragility, causal misgrounding, action-insensitive heuristics, and stability traps. These results show that reliable VLM-as-a-judge evaluation requires joint reporting of correctness, presentation invariance, and causal sensitivity rather than a single accuracy or consistency score.
Show more
TraceTriage: A Benchmark for Cost-Aware Stop-or-Continue Decisions in Delayed-Outcome Workflows
Sihang Lei ⋅ Yihang Qiu ⋅ Xueyan Zhao ⋅ Yuwei Wang ⋅ Weiqiang Wang
Current agent benchmarks largely evaluate whether a workflow eventually succeeds, but many real workflows require an earlier decision: whether to continue spending resources or stop and reallocate them before the final outcome is known. In model training, simulation campaigns, and chip implementation, success or failure may be visible only after substantial compute and engineering time. Wrong stops discard runs that would have succeeded; wrong continuations waste downstream compute, tool time, and engineering effort. This asymmetric, risk-constrained intervention problem is not captured by benchmark scores that only measure eventual success. We present TraceTriage, a benchmark for evaluating LLM agents and decision models on cost-aware stop-or-continue policies from checkpoint-local process records. TraceTriage is not a final-outcome prediction benchmark: it evaluates deployable interventions using validation-only thresholding under a false-stop cap and remaining-cost utility. We instantiate the protocol with OpenROAD, an open-source electronic-design-automation workflow that turns hardware designs into placed-and-routed chip layouts through staged tools. OpenROAD is a reproducible substrate, not the benchmark's scope: it provides staged observability, heterogeneous logs and reports, measurable remaining runtime, reproducibility, and verifiable outcomes. The released benchmark uses no-leak run-level splits and instantiates this protocol through 10 OpenROAD design-optimization campaigns with a 50,000-trial budget, retaining 31,572 runs and expanding them to 81,041 checkpoint prefixes with fixed train/validation/test splits. Each example contains only evidence available at a workflow checkpoint, while remaining-cost annotations are used only for evaluation. Together, the results make TraceTriage a capability-boundary benchmark for cost-aware workflow intervention: it verifies actionable signal in checkpoint-local records, quantifies transfer sensitivity, and exposes the unresolved cold-start calibration gap in delayed-outcome, risk-constrained settings.
Show more
Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews
Shreyansh Padarha ⋅ Ryan Othniel Kearns ⋅ Tristan M Naidoo ⋅ Lingyi Yang ⋅ Łukasz Borchmann ⋅ Piotr Blaszczyk ⋅ Christian Morgenstern ⋅ Ruth McCabe ⋅ Sangeeta Bhatia ⋅ Philip Torr ⋅ Jakob Foerster ⋅ Scott Hale ⋅ Thomas Rawson ⋅ Anne Cori ⋅ Elizaveta Semenova ⋅ Adam Mahdi
Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. AgentSLR evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.
Show more
Towards Error-Free EHRs: Reasoning-Intensive Consistency Verification Between Clinical Notes and Structured Tables in Electronic Health Records
Yeonsu Kwon ⋅ Jiho Kim ⋅ Junseong Choi ⋅ Paloma Rabaey ⋅ Minseo Kim ⋅ Sujeong Im ⋅ Jeewon Yang ⋅ Jun-Min Lee ⋅ Sangji Lee ⋅ Jiwon Kim ⋅ Hangyul Yoon ⋅ Hyunwook Kwon ⋅ Edward Choi
Data consistency between unstructured clinical notes and structured tables in Electronic Health Records (EHRs) is essential for patient safety and clinical decision-making. However, existing work on note-table consistency verification mainly relies on surface-level matching of numeric values or simple events. Such approaches fail to capture the reasoning underlying real-world EHR documentation, including clinical interpretation, event relations, and temporal changes. To address this gap, we introduce EHR-ReasonCon, a reasoning-intensive benchmark for note-table consistency verification. Built on MIMIC-III with expert-guided annotations, it comprises 8,048 entities derived from clinical notes and provides high-quality ground-truth labels. The annotation protocol is supported by specialized table-exploration tools to ensure systematic evidence retrieval and reliable consistency assessment. We also propose EHR-INSPECTOR, an LLM-based framework that segments notes, extracts anchor entities and temporal references, and uses table-exploration tools to verify consistency against structured tables. Evaluated using expert-validated LLM-as-a-judge metrics under harsh and lenient criteria, EHR-INSPECTOR achieves state-of-the-art performance across multiple model backbones. Analyses further demonstrate the effectiveness of its components and highlight differences from human verification. The code is available at https://anonymous.4open.science/r/EHR-ReasonCon-1A54/.
Show more
RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
Hongyu Jin ⋅ Siyi Wang ⋅ Yang Xiao ⋅ Jiaheng Dong ⋅ Shihong Tan ⋅ Kaiyuan peng ⋅ Georgiana Juravle ⋅ Shanquan Chen ⋅ Gongping Huang ⋅ Hong Jia ⋅ Eun-Jung Holden ⋅ James Bailey ⋅ Ting Dang
Humans process rich auditory environments through tightly integrated cognitive capabilities such as audio perception, audio reasoning, and memory. Despite recent progress in large audio-language models (LALMs) across speech understanding and multimodal audio reasoning, current evaluation paradigms remain largely task- or modality-centric, focusing on end performance while overlooking underlying auditory cognitive behaviours. This reveals a fundamental gap between how auditory cognition is understood in humans and how it is evaluated in LALMs, particularly in the lack of frameworks that operationalise cognitive principles beyond task-level metrics to systematically capture model behaviour. In this work, we introduce RAIL, a human-centric evaluation paradigm grounded in the Cattell-Horn-Carroll (CHC) cognitive framework. RAIL formalises auditory cognition into five core capabilities and develop them into structured evaluation tasks that probe how models process, retain, and integrate auditory information. We further construct a cognitively grounded benchmark with principled data curation and human-aligned evaluation protocols. Evaluating 26 state-of-the-art LALMs, we find that current models exhibit highly uneven performance across cognitive abilities. RAIL establishes a new evaluation paradigm that moves beyond task-centric benchmarking toward cognitively grounded assessment of auditory intelligence.
Show more
DataComp-VLM: Improved open datasets for Vision-Language Models
Matteo Farina ⋅ Vishaal Udandarao ⋅ Thao Nguyen ⋅ Selim Kuzucu ⋅ Maximilian Böther ⋅ Andreas Hochlehnert ⋅ Adhiraj Ghosh ⋅ Marianna Nezhurina ⋅ Karsten Roth ⋅ Joschka Strüber ⋅ Yuhui Zhang ⋅ Sebastian Dziadzio ⋅ Elaine Sui ⋅ Soumya S Jahagirdar ⋅ Dhruba Ghosh ⋅ Hasan Hammoud ⋅ Thomas De Min ⋅ Simone Caldarella ⋅ Muhammad Jehanzeb Mirza ⋅ Sedrick Keh ⋅ Mehdi Cherti ⋅ Hilde Kuehne ⋅ Bernt Schiele ⋅ Serena Yeung-Levy ⋅ Muhammad Ferjad Naeem ⋅ Federico Tombari ⋅ Ana Klimovic ⋅ Elisa Ricci ⋅ Matthias Bethge ⋅ Sewoong Oh ⋅ Ameya Prabhu ⋅ Alessio Tonioni ⋅ Jenia Jitsev ⋅ Massimiliano Mancini ⋅ Ludwig Schmidt ⋅ Nikhil Parthasarathy
Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we standardize 160 datasets spanning four data types—image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data—into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B–8B models and 6.25B–200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-BASELINE, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FINEVISION, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available.
Show more
Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning
Yuyang Wu ⋅ Yue Huang ⋅ Shuaike Shen ⋅ Xujian Wang ⋅ Shuhao Zhang ⋅ Qiyao Xue ⋅ Weichen Liu ⋅ Runtian Gao ⋅ Jian Ma ⋅ Xiangliang Zhang ⋅ Olexandr Isayev
Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous evaluation of scientific tool use remains limited. In chemistry, recent agents can plan syntheses and invoke domain-specific tools, but evaluations often rely on curated demonstrations, expert assessment, or LLM-as-judge scoring rather than exact, judge-free ground truth. We address this gap with chemical procurement cost estimation, a practical task in which an agent must ground chemical identities, retrieve supplier quotes, select valid purchasable packs, normalize quantities, and compute cost from a reaction description. We introduce ChemCost, a benchmark of 1,427 evaluable reactions grounded to a frozen pricing snapshot covering 2,261 chemicals and 230,775 supplier quotes, supporting scalar scoring and stage-level diagnosis of grounding, retrieval, procurement, and arithmetic failures. To evaluate robustness, we further construct controlled noise-injected views that perturb chemical aliases, quantity expressions, missing fields, and input formatting. Experiments with frontier, open-weight, and chemistry-specialized LLM agents show that tool access is necessary but insufficient for solving the task. The strongest agents reach only 50.6% accuracy within 25% relative error on clean inputs and degrade substantially with realistic noise. Stage-level analysis further shows that failures arise from brittle parsing, ineffective evidence integration, invalid pack selection, and non-convergent tool use.
Show more
Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
Eunsu Kim ⋅ Haneul Yoo ⋅ Guijin Son ⋅ Hitesh Patel ⋅ Amit Agarwal ⋅ Alice Oh
LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.
Show more
U-Bench: A Comprehensive Understanding of U-Net through 100-Variant Benchmarking
Fenghe Tang ⋅ Chengqi Dong ⋅ Wenxin (Wendy) Ma ⋅ Zikang Xu ⋅ Heqin Zhu ⋅ Zihang Jiang ⋅ Rongsheng Wang ⋅ Yuhao Wang ⋅ Chenxu Wu ⋅ Yingtai Li ⋅ S. Kevin Zhou
Over the past decade, U-Net has been the dominant architecture in medical image segmentation, leading to the development of thousands of U-shaped variants. Despite its widespread adoption, there is still no comprehensive benchmark to systematically evaluate their performance and utility, largely because of insufficient statistical validation and limited consideration of efficiency and generalization across diverse datasets. To bridge this gap, we present U-Bench, the first large-scale, statistically rigorous 2D benchmark that evaluates 100 U-Net variants across 28 datasets and 10 imaging modalities. Our contributions are threefold: (1) Comprehensive Evaluation: U-Bench evaluates models along three key dimensions: statistical robustness, zero-shot generalization, and computational efficiency. We introduce a novel metric, U-Score, which jointly captures the performance-efficiency trade-off, offering a deployment-oriented perspective on model progress. (2) Systematic Analysis and Model Selection Guidance: We summarize key findings from the large-scale evaluation and systematically analyze the impact of dataset characteristics and architectural paradigms on model performance. Based on these insights, we propose a model advisor agent to guide researchers in selecting the most suitable models for specific datasets and tasks. (3) Public Availability: We provide all code, models, protocols, and weights, enabling the community to reproduce our results and extend the benchmark with future methods. In summary, U-Bench not only exposes gaps in previous evaluations but also establishes a foundation for fair, reproducible, and practically relevant benchmarking in the next decade of U-Net-based segmentation models. The weights, datasets, and results will be released after the acceptance.
Show more
BenchRep-T: A Systematic Evaluation of T-Cell Repertoire-Based Disease Diagnostics
Chiho Im ⋅ Liel Cohen-Lavi ⋅ Alejandro Buendia ⋅ Anshul Kundaje ⋅ Scott D Boyd
Adaptive immune receptor repertoire sequencing data has emerged as a promising potential modality for disease diagnosis, relying on computational methods to analyze T-cell receptor (TCR) sequences from an individual’s blood sample. Published methods rely on different cohorts, data preprocessing pipelines, and evaluation metrics, making direct comparison across methods challenging. We have developed BenchRep-T, a unified benchmark that standardizes multiple publicly available TCR repertoire datasets and evaluates nine computational approaches, spanning statistical enrichment of shared sequences, feature-engineered ensembles, deep learning, and graph-based clustering. BenchRep-T evaluates methods on four tasks: disease classification across conditions, performance scaling under restricted sequence-sampling depth, recovery of known antigen-specific driver sequences, and evaluation of sensitivity to demographic confounding. Under controlled evaluation, simple baselines prove competitive, with tree-based models trained on V- and J-gene usage and short sequence motifs approaching the classification performance of more complex methods. Our findings underscore the complexity of modeling TCR repertoire data, and show that no single method dominates across all tasks. BenchRep-T provides a framework for rigorous and reproducible evaluation of TCR repertoire classification methods to accelerate the development of immune repertoire-based diagnostics.
Show more
VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs
Chen Chen ⋅ Zheming Liang ⋅ Jiaqi Wang
Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and fine-grained visual reasoning. However, existing benchmarks fall short along two complementary dimensions. First, category coverage is limited: most benchmarks cover only a few thousand fine-grained entity categories, often within narrow domains, far from the open-ended scale of real-world visual entities. Second, task diversity is limited: most benchmarks rely on single task formats such as closed-set classification or attribute QA, and therefore cannot characterize the multi-dimensional structure of visual world knowledge. We introduce VisualWorldBench, a taxonomy-grounded multi-task diagnostic benchmark built on a hierarchical taxonomy of 21K+ leaf-node classes across 8 visual domains, comprising 32K+ instances across four complementary tasks: closed-set recognition, contrastive selection, precise localization, and open-world naming. To our knowledge, this is the largest category coverage among visual world knowledge benchmarks to date. VisualWorldBench reveals substantial capability gaps hidden under conventional benchmarks: the strongest closed-source model achieves only 80.4% average accuracy, and single-model spread across tasks can exceed 40 percentage points, indicating that visual world knowledge is multi-dimensional rather than monolithic. We further release VisualWorldBench-50K, a training resource covering 50K+ categories. Fine-tuning on VisualWorldBench-50K yields over 21 percentage points improvement on VisualWorldBench, with consistent transfer to public fine-grained recognition and visual knowledge benchmarks.
Show more
GeoG2U-Bench: When Does Generation Help Understanding in Ultra-High-Resolution Remote Sensing?
Fengxiang Wang ⋅ Yueying Li ⋅ Mingshuo Chen ⋅ Boya Miao ⋅ Qiuyang Yu ⋅ Yajie Yang ⋅ Mingzhen Xu ⋅ Luqing Luo ⋅ Haonan Guo ⋅ Hongda Sun ⋅ Yulin Wang ⋅ Jun Song ⋅ Jing Zhang ⋅ Long Lan ⋅ Wenjing Yang
Ultra-high-resolution (UHR) remote-sensing imagery captures fine-grained spatial details essential for city-scale Earth observation, yet it exposes a fundamental limitation of current multimodal large language models (MLLMs): native-resolution inputs exceed model capacity, while aggressive downsampling erases the very small objects that many questions hinge on. Generation-understanding unified multimodal models offer a promising remedy, since they can produce intermediate visual observations before answering. However, whether such capabilities genuinely improve UHR remote-sensing understanding remains an open question. Existing UHR benchmarks cannot adjudicate it: they are mostly built from mature datasets covering only a few regions, their image scales remain below real city-level satellite scenes, and they do not systematically identify which tasks benefit from intermediate visual generation. We introduce GeoG2U-Bench, a global benchmark designed to evaluate Generation-to-Understanding (G2U) in UHR remote sensing. GeoG2U-Bench spans 300 cities worldwide, with every raw satellite scene reaching $20{,}000 \times 20{,}000$ pixels, matching the scale at which UHR analysis actually occurs. It comprises 20 subtasks across five capability dimensions and roughly 3,000 expert-verified samples, each requiring models to produce auditable intermediate artifacts such as zoom-in crops, annotated local views, cross-region relation maps, and temporal rewind images. Beyond data, GeoG2U-Bench contributes a dual-protocol evaluation: Direct (answer from the raw input) versus Generate-then-Answer (generate an intermediate visual artifact, then answer), which isolates and quantifies the contribution of generation to downstream understanding. Experiments across several model families reveal that generation is not uniformly beneficial: unfaithful or off-task artifacts can actively degrade performance, whereas reliable, task-relevant artifacts yield consistent gains in cross-scale localization, multi-region comparison, temporal reasoning, and spatial layout understanding. These results show that visual generation can support UHR remote-sensing understanding, but only when it provides reliable visual evidence for the task.
Show more
A Scientific Claim Stability Framework for Evaluating MRI Morphometry Pipelines
Duy-Cat Can ⋅ Dang-Duat Tran ⋅ Anh-Khang Nguyen ⋅ Bao Tran ⋅ Trung-Hieu Do ⋅ Van T Nguyen ⋅ Duy-Thanh VU ⋅ Oliver Y Chén ⋅ Hoang-Quynh Le
MRI morphometry pipelines are often evaluated by measurement agreement or downstream accuracy, yet their scientific value depends on the conclusions they support. This creates a gap: two pipelines may look similar at the feature level while supporting different conclusions about aging, cognitive decline, or neurodegeneration. To bridge this disparity, here we propose Scientific Claim Stability (SCS), a reusable claim-centric evaluation framework that converts region-of-interest (ROI) morphometry tables into interpretable claim cards and tests whether each claim remains supported under realistic perturbations. To demonstrate the efficacy of SCS, we evaluate it on 10,370 MRI scans from three datasets and five cohorts, generating 2,783 claim cards and evaluating 89 claim-family pairs across dataset, pipeline, atlas, feature-family, and target shifts. SCS identifies claims that are stable, transferable, or fragile, and reveals high-concordance fragile-support cases where evidence vectors remain similar while supporting regions change. To assess claim plausibility, clarity, faithfulness, usefulness, and caution level, we further implement and compare an expert (including geriatrics, neuroscience, and radiology) review workflow and an LLM (including ChatGPT and Gemini) review workflow. Our results suggest claim-level evaluation as a practical standard for auditing MRI morphometry pipelines without requiring a universal gold standard.
Show more
ReasoningShield: Safety Moderation over Reasoning Traces of Large Reasoning Models
Changyi Li ⋅ Jiayi Wang ⋅ Xudong Pan ⋅ Geng Hong ⋅ Min Yang
Large Reasoning Models (LRMs) leverage explicit reasoning traces, known as Chain-of-Thought (CoT) reasoning, to decompose complex problems into intermediate steps before deriving final answers. However, these reasoning traces introduce unique safety challenges: harmful content can be embedded in intermediate steps even when final answers appear benign. Our study reveals that existing state-of-the-art moderation tools experience significant performance degradation on CoT moderation, with F1 scores dropping by up to 35.2% compared to traditional answer moderation. To address these challenges, we present ReasoningShield, a comprehensive benchmark and a suite of strong lightweight models for CoT safety moderation. Our benchmark includes 9.2K annotated Query-CoT pairs with structured stepwise risk analysis, consisting of a 7K-sample training set and a 2.2K human-annotated test set, spanning 10 risk categories, 3 safety levels, and 8 LRMs with diverse reasoning paradigms. To further demonstrate the utility of the benchmark, we develop lightweight models with 1B and 3B parameters using a two-stage training strategy. Our models achieve 91.8% F1 on our benchmark, substantially outperforming leading tools such as LlamaGuard-4 by 35.6% and commercial models such as GPT-4o by 15.8%, while generalizing effectively across diverse reasoning paradigms and unseen scenarios. All resources, including the dataset, code, and models, are released at https://anonymous.4open.science/r/ReasoningShield.
Show more
JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation
Yue Xun ⋅ Junyu Liu ⋅ Qian Niu ⋅ Xinyi Wang ⋅ Zheng Yuan ⋅ Zirui Li ⋅ Zequn Zhang ⋅ Bowen Zhao ⋅ Emma, Shujun Wang ⋅ Zihui Li ⋅ Kan Hatakeyama-Sato ⋅ Yusuke Iwasawa ⋅ Yutaka Matsuo
We introduce JMed48k, a multi-profession Japanese healthcare licensing benchmark for evaluating vision-language models. Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images from 11 national licensing examinations between 2005 and 2025, with visual content annotated under an 8-type taxonomy. From this corpus, we derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images. We evaluate 21 proprietary, open-source, and medical-specific models, reporting text-only and with-image performance separately. Because these subsets contain different questions, we further introduce a paired image-removal audit that evaluates questions with images before and after removing visual content to explore four answer-transition states. The audit shows that proprietary and open-source models gain substantially from images, whereas medical-specific systems show limited observable use of visual evidence, with many correct answers persisting after image removal. Even among proprietary models, the net image-removal effect varies sevenfold across professions, from +5.7 points on Physician questions to +39.8 points on Public Health Nurse questions. We release JMed48k to support reproducible, profession-stratified evaluation of vision-language models in medical licensing settings.
Show more
RepoZero: Can LLMs Generate a Code Repository from Scratch?
Zhaoxi Zhang ⋅ Yiming Xu ⋅ Jiahui Liang ⋅ Weikang Li ⋅ Xiaoshuai Chen ⋅ Liwei Qian ⋅ Xin Pei ⋅ Jizhou Huang ⋅ Rui Sun ⋅ Yunfang Wu
Large Language Models (LLMs) have recently shown remarkable progress in code generation, yet their ability to construct complete software repositories from scratch remains poorly understood. A fundamental bottleneck is the lack of verifiable and scalable evaluation: existing benchmarks either focus on patch-based editing or rely on human or LLM-based judgments, which introduce bias and limit reproducibility. In this work, we present RepoZero, the first benchmark that enables fully automated, execution-based verification of repository-level generation from scratch. Our key idea is to reformulate generation as repository reproduction: given only API specifications, an agent must re-implement an entire repository such that its behavior matches the original implementation. This design allows for strict black-box validation via output equivalence, while naturally supporting large-scale construction by reusing existing open-source repositories. To further mitigate data leakage and shortcut solutions, we introduce cross-language constraints and a sandboxed evaluation protocol. Building on this benchmark, we propose an Agentic Code-Test Evolution (ACE) framework that performs iterative test generation and error-driven refinement, enabling effective test-time scaling for repository-level synthesis. Extensive experiments across multiple state-of-the-art LLMs and agent frameworks reveal that even the strongest models achieve only limited pass rates (30\% - 55\%), exposing a substantial gap between current capabilities and real-world software development requirements. Our results establish RepoZero as a challenging, scalable, and reliable testbed for end-to-end code generation, and highlight self-verification via test generation as a critical direction for advancing LLM-based coding agents.
Show more
MetaLoop: Benchmarking the Full Metacognitive Loop in LLMs
Nora Petrova ⋅ John Burden ⋅ Jerome Wynne
We introduce MetaLoop, a seven-task benchmark that evaluates the full metacognitive loop in large language models: monitoring one's own uncertainty, translating that signal into action (abstaining, switching strategies, correcting errors), and updating from feedback. Existing evaluations test these components in isolation; MetaLoop produces integrated per-model profiles across all three. Evaluating 12 frontier models alongside 460 human participants, we find that (1) accuracy does not predict metacognitive ability (high-accuracy models routinely approve their own errors); (2) models universally produce calibration signals but fail to act on them; (3) forced self-explanation distinguishes genuine self-monitoring from brittle pattern-matching; and (4) humans and LLMs both show monitoring–control gaps but with different failure modes (humans are loss-averse, models overconfident), with clarification detection as the one clear human advantage. We release the benchmark, scoring code, model outputs, and human data.
Show more
Benchmarking Vietnamese Legal Knowledge of Large Language Models
Dong Nguyen Tien ⋅ Minh-Anh Nguyen ⋅ Thanh D Hoang ⋅ Nguyen T Ngoc ⋅ Quang M Xuan ⋅ Phan P Hai ⋅ Nguyen T Anh ⋅ Binh Vu ⋅ Dung Le
The rapid advancement of large language models (LLMs) has expanded their potential in the legal domain. However, existing legal benchmarks remain largely English-centric and oriented toward common law, leaving a critical gap in evaluating LLMs for civil law systems that govern most jurisdictions worldwide. To address this gap, we introduce Vietnamese Legal Benchmark VietLegal, a cognitively grounded benchmark designed for the hierarchical and codified structure of Vietnamese law. Although instantiated in Vietnamese legislation, VietLegal provides a replicable evaluation framework for civil law systems characterized by complex statutory hierarchies and frequent amendments. Inspired by Bloom’s taxonomy, VietLegal assesses multiple levels of legal understanding through tasks that mirror real-world legal assistant use cases, including legal question answering, multi-step reasoning, and scenario-based problem solving. The benchmark contains 10,450 expert-annotated samples, each cross-validated against authoritative legal sources to ensure fidelity to practical legal workflows. By offering the first standardized legal benchmark for Vietnamese, VietLegal enables systematic assessment of LLMs in civil law contexts and supports the development of more reliable and interpretable AI-assisted legal systems.
Show more
When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework
Dylan Steiner ⋅ Gustavo Arango ⋅ Gerald J Sun ⋅ Etai Jacob
Multimodal models in oncology can produce accurate predictions, but accurate prediction does not reveal whether the model has learned biology that is shared across modalities, biology confined to one modality, or spurious correlations that reflect confounders rather than genuine biology. We introduce DECAT, a model-agnostic post-hoc evaluation framework that classifies multimodal representations into four diagnostic scenarios for a given task and modality, using five null-referenced metrics and a rule-based decision procedure. The framework operates on learned representations, requires no knowledge of which specific confounder is present, and returns indeterminate when the evidence is insufficient. We evaluate multimodal architecture properties using DECAT on synthetic data across four multimodal model classes spanning over 2,500 trained representations, and demonstrate its operational use on 8,979 TCGA patients with both multimodal model embeddings and with five pretrained pathology foundation models. Entangled models (e.g., CLIP) achieve near-perfect shared biology detection but falsely claim shared biology in the majority of cases where it is absent on real foundation model embeddings. This false claim rate increases with confound strength so that larger cohorts and stronger representations produce more confident but still incorrect diagnoses. Applied to both multimodal TCGA embeddings and five pathology foundation models without paired RNA, DECAT detects confounding invisible to AUROC without requiring the confounder labels, as confirmed by post-hoc stratification.
Show more
Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
Devina Jain ⋅ David Hartmann ⋅ Chuan Li
LM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. Most safety benchmarks evaluate defenders against fixed attack pools collected before evaluation, single-turn or multi-turn. We present a 21-scenario benchmark for adaptive multi-round attacks against memoryless LLM defenders: an autonomous LLM attacker observes prior defender responses and pivots across rounds, while each defender response is evaluated as a fresh interaction. Holding the 21 scenarios, attackers, defenders, and structured-output scoring fixed, restricting scoring to the first attacker turn yields $0$--$1%$ attack success rate (ASR); allowing 15 rounds of adaptive attack yields $8.3$--$19.0%$. Pooling three frontier attacker LLMs uncovers up to $2.4\times$ as many unique successful attacks as the best single attacker, and the generated attacks have low cosine similarity ($0.02$--$0.14$) to attacks in existing benchmarks. Claude Opus 4.6 and GPT-5.4 are aggregate-equivalent ($8.3%$ vs. $11.1%$ ASR), but their weaknesses differ sharply: on one scenario Opus reaches $60%$ ASR ($95%$ CI $36$--$80%$) while GPT-5.4 and Gemini each stay at $7%$ (CI $1$--$30%$). $15$ of $21$ scenarios distinguish at least one defender pair, yet rankings disagree across scenarios (Kendall's $W = 0.21$). We release the benchmark---21 evaluation scenarios, 10 public development scenarios, the orchestrator, baseline harnesses, and a multi-attacker CLI---plus 945 transcripts from the $3\times3$ frontier matrix, an attack-replay dataset, and $18{,}422$ battles from an open competition's final scoring rounds.
Show more
A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
Yongjin Yi ⋅ JIn Yong Park ⋅ Sejoong Kim
The Objective Structured Clinical Examination (OSCE) is widely used to assess clinical communication skills in medical education, requiring trainees to take histories and reason clinically with standardized patients (SPs). Large language model (LLM)-based virtual standardized patients (VSPs) offer a practical alternative to human SPs. These are typically built using persona-injected prompting, which casts a model into a clinical role via a single system prompt. However, existing evaluations of such VSPs often rely on a single backbone LLM, small human rater panels, or LLM-as-a-judge scoring without an utterance-aligned human-expert reference. Consequently, they are limited in their ability to identify the specific shortcomings of individual models. In this study, we evaluated four LLMs (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B, and HyperCLOVA X), spanning commercial, open-source, and Korean-developed models, against a human-expert reference across three Korean OSCE cases (hematuria, fever in pregnancy, and chronic cough). In a double-blind, fully crossed design, 17 senior medical-student raters scored every model utterance on three sentence-level and six encounter-level Likert metrics, yielding 7,480 data points. We analyzed the scores using a stratified linear mixed-effects model based on an 11-category history-taking and 3-level linguistic-form taxonomy. At the encounter level, three of the four LLMs fell 0.08 to 0.47 Likert points below the human-expert reference on Comprehensibility, Consistency and Hallucination-freeness. The difference was not statistically significant for Hallucination-freeness in Claude 3.5 Sonnet and GPT-4o, Consistency in GPT-4o, and Engagement in HyperCLOVA X. Llama-3.1 70B fell significantly below the reference on all six metrics. Furthermore, we introduce a web-based OSCE platform that supports virtual practice across diverse clinical cases. This platform continuously collects student–VSP dialogues under research and privacy consent; the resulting dataset is available at https://huggingface.co/datasets/neurips-2026-cpx/neurips-2026-cpx
Show more
ChronosAlign: A Large-Scale, Updatable Benchmark for Decomposing Temporal Alignment in LLMs
Sanjay Govindan ⋅ Maurice Pagnucco ⋅ Yang Song
Large Language Models (LLMs) struggle with temporal alignment, often producing outdated time-sensitive facts as real-world knowledge evolves. Yet the field's ability to measure this misalignment is itself fragile. Existing benchmarks are static snapshots, rely on rigid templates, are small, lack provenance, conflate willingness-to-answer with recall, and use phrasings that newer models have likely memorised. We argue that progress on temporal alignment is bottlenecked less by data scarcity than by evaluation design, and we contribute on both fronts. We present ChronosAlign, a natural language, programmatically updatable dataset and evaluation framework for temporal alignment in LLMs; generated from a Wikidata and Wikipedia corpus of ~500,000 question-answer pairs covering ~27,000 unique questions across sport, politics, and culture from 2000 to 2025. Each question carries a Wikipedia URL and a documented update procedure, so the same evaluative claim can be re-tested against future models without re-authoring the benchmark. We couple ChronosAlign with a framework focussed on three prompting modalities, explicit (year-anchored), relative ("current" anchored), and invariant (untimed control), whose disagreement isolates which component of temporal alignment a model fails. We benchmark sixteen LLMs surfacing three findings: (i) a sharp post-2022 decay in explicit recall even for models with later training cutoffs, (ii) systematic misalignment between explicit and relative recall, with some models better aligned to explicit year prompts and others to relative anchors, and (iii) a confounding effect of refusal behaviour. Our generation and update code can be found at: https://github.com/sg-sy/ChronosAlign. Our dataset can be found at: https://huggingface.co/datasets/sg-sy/ChronosAlign.
Show more
$\mathbf{\mathtt{MAD\text{-}Bench}}$: How Do Multimodal Agents Deceive You?
Hao Gu ⋅ Mingli Song ⋅ Jiacong Hu
Multimodal agents increasingly operate computers on behalf of users. This shift from passive question answering to interactive execution creates a new safety problem. When an agent encounters normal, non-attack scenarios where the task it needs to perform is impossible or difficult to complete, the central risk is not only that it fails, but that it deceives the user into believing that it has succeeded. Although prior work has studied truthfulness and deception in language models and language agents, multimodal interaction introduces new sources of deceptive behavior, increasing the implicitness and complexity of deceptive behaviors in multimodal agents. To fill this gap, we introduce $\mathbf{\mathtt{MAD\text{-}Bench}}$, a benchmark for evaluating **M**ultimodal **A**gent **D**eception. $\mathbf{\mathtt{MAD\text{-}Bench}}$ contains 360 sandbox tasks organized around three core elements of multimodal execution: *modality*, *target*, and *tool*, and spans six task types including *modality evidence conflict*, *modality asynchronous mismatch*, *modality ambiguity distortion*, *modality object missing*, *target infeasibility*, and *tool defect*. To our knowledge, $\mathbf{\mathtt{MAD\text{-}Bench}}$ is the first benchmark to formalize deceptive behavior in multimodal agents from an execution-evidence perspective, characterizing deception as a misalignment between the user-facing task-state claim and the multimodal evidence available throughout the execution trajectory. We further propose a behavioral taxonomy covering *evasive deception*, *manipulative deception*, *misleading deception*, and *non-deception*. Evaluating 10 mainstream multimodal agents, we find that agents frequently fabricate, conceal, mislead, frame, and disguise during task execution. These results show that substantial deceptive behaviors are widespread across mainstream model families. This benchmark is available at https://anonymous.4open.science/r/MAD-Bench-A304.
Show more
Online Evaluation of LLMs via Dyadic Designs
Jinglong Zhao ⋅ Zijie Zhou
Online A/B tests are the dominant tool for evaluating new LLM variants in deployment: each user query arrives sequentially, must be routed to one variant before its outcome is observed, and is then scored against the alternative. In practice, these tests almost always use complete randomization. Yet for many queries, simple signals such as predicted difficulty, expected reward, or embedding-based scores are already available, and these are predictive of the outcome. Using them to guide assignment can substantially reduce the variance of the estimated treatment effect. We propose a new dyadic design that uses such covariates to balance treatment and control online across the full covariate distribution, with no hyperparameters to tune. Under stochastic arrivals, our design attains a covariate matching discrepancy of $O(\log^{3} T)$ in one dimension, which is a significant improvement over the $O(T^{1/4})$ rate of existing online stratified designs, and matches the offline minimax rate up to logarithmic factors in higher dimensions, closing a gap left open by prior work. Experiments on a sequential A/B test simulator built from AlpacaEval, in which 805 prompts are scored under two real LLM variants, confirm the theory: our design achieves a covariate matching discrepancy roughly $6\times$ smaller than complete randomization and consistently outperforms the best-tuned online stratified baseline. The method extends to multi-dimensional covariates and to general sequential evaluation pipelines.
Show more
NORA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning
Sichao Li ⋅ Sai Ma ⋅ Daniel Kilov ⋅ Secil Yanik Guyot ⋅ Zhuang Li ⋅ Seth Lazar
LLMs and agentic systems are increasingly deployed in social environments, making normative competence critical for safe and appropriate behavior. However, existing approaches either assess normative judgment in text alone or reduce it to choosing among a fixed set of candidate actions. We argue both are insufficient. In practice, agents are never handed a menu of options; they must identify a reasonable action from scratch, grounded in visible facts and supported by inspectable reasons. We introduce NoRA, a visual first-person video benchmark that requires models to generate candidate next actions and justify each through an explicit fact–reason–action support graph. The benchmark comprises 1420 annotated video clips, including HhumanGold-190 and LLMSilver-1230 splits. Each instance is evaluated through action alignment, factual grounding, and support binding, aggregated into a single grounded reasonableness score. We benchmark 12 multimodal systems under direct, deliberative, and structured prompting regimes, finding that current VLMs frequently recover plausible actions and relevant scene facts, but consistently struggle to construct the full reasonable action space and bind selected actions to the correct local support. NoRA makes this gap measurable, shifting the evaluation question from whether a model can pick an action to whether it can justify an appropriate action for the right visible reasons.
Show more
VL-DocIR: A Benchmark for Vision-Based Long Document Retrieval
Simon Ott ⋅ Yufan Chen ⋅ Ruiping Liu ⋅ Junwei Zheng ⋅ Jiale Wei ⋅ Di Wen ⋅ Kunyu Peng ⋅ Jiaming Zhang ⋅ Rainer Stiefelhagen
Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This leaves an important gap between leaderboard performance and realistic long-document retrieval, where the evidence needed to answer a query may span multiple pages or even multiple documents. We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval. VL-DocIR is built from 29,641 HTML-born documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. It contains 271,760 queries across 23 domains and six evidence structures, covering single-page, within-document multi-page, and cross-document evidence configurations. Evidence annotations are grounded to rendered pages and HTML element identifiers, and queries are filtered with a cleaning pipeline targeting clarity, correctness, closedness, and the absence of explicit layout references. We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles. The strongest evaluated configuration reaches 75.6 nDCGAll@10 and 86.2 RecallAll@10, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries. VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval.
Show more
WiREBench: Evaluating AI Agents' Capabilities in Reverse-Engineering Black-Box Applications in the Real World
Robin Rheem ⋅ Alexandra Maxwell ⋅ Mona Wang ⋅ Benjamin Mixon-Baca ⋅ Tianneng Shi ⋅ Jeffrey Knockel ⋅ Ali Aldakheel ⋅ Basel Alomair ⋅ Prateek Mittal ⋅ Dawn Song
As frontier AI technologies dramatically improve their abilities to perform cybersecurity and reverse-engineering tasks, it is critical to rigorously and realistically track their capabilities in these domains. Increased agent capabilities in reverse-engineering will scale black-box vulnerability discovery, which will have dramatic implications on software security at large. In this work, we present WiREBench, a unique benchmark derived from popular real-world mobile applications, to measure agents' end-to-end vulnerability discovery and reverse-engineering capabilities. This benchmark, containing 22 protocols and 62 subtasks, evaluates agents abilities to identify and exploit historical vulnerabilities in binaries' proprietary network cryptography, via black-box reverse-engineering. To resolve these tasks, agents must chain together static binary analysis, understanding of applied cryptography, and complex tool use via dynamic instrumentation of mobile applications. Beyond benchmarking, we ran WiREBench against real-world applications, identifying and confirming 65 protocol vulnerabilities in applications used by hundreds of millions of users. These results demonstrate WiREBench's utility not just as a benchmark to assess current agent capabilities, but also as a pipeline for auditing cryptographic protocols and reverse-engineering in the real world.
Show more
WirelessMathBench-XL: A Contamination-Audited Benchmark for Wireless Mathematical Reasoning
Xin Li ⋅ Mengbing LIU ⋅ Yiyang Zhu ⋅ WENHE ZHANG ⋅ LI WEI ⋅ Jiancheng An ⋅ Chau Yuen
**WirelessMathBench-XL** is a $4{,}027$-problem benchmark for wireless mathematical reasoning, built from $836$ retained arXiv papers across $20$ wireless subfields. Benchmarks built from arXiv papers risk overlap with the same arXiv text used in LLM pretraining, yet few releases include the provenance and metadata needed to audit this overlap. WirelessMathBench-XL builds contamination auditing into the released artifact through a **reverse-probe $13$-gram audit**: it indexes benchmark prompts, streams public pretraining corpora, and records per-problem prompt-surface lexical overlap with benchmark-scale memory. Against $12.6$\,B streamed $13$-grams from RedPajama-arXiv, the audit identifies a strict zero-hit subset $\mathcal{S}_0$ covering $3{,}853$ problems ($95.7\%$). Filtering to $\mathcal{S}_0$ changes accuracy by less than $1$\,pp for every evaluated model; frontier-model calibration rows form a single high-accuracy cluster between $86.5\%$ and $91.3\%$, not a resolved rank order. Thus, on this fixed prompt-surface lexical audit channel, reported scores are not measurably driven by exact recall of detected RedPajama-arXiv prompt text. The audit does not cover paraphrase, target-answer, post-training, or closed-corpus exposure. The release includes source-paper identifiers, verifier-facing ground truths, contamination metadata, cleaned subsets, a paper-disjoint sensitivity view, evaluation traces, paired-bootstrap scripts, training recipes, Croissant metadata, and a Datasheet for Datasets, so users can rerun, tighten, or replace the cleaned view.
Show more
NARRA-Gym for Evaluating Interactive Narrative Agents
Yue Huang ⋅ Yuchen Ma ⋅ Jiayi Ye ⋅ Wenjie Wang ⋅ Zipeng Ling ⋅ Xingjian Hu ⋅ Yuexing Hao ⋅ Zichen Chen ⋅ Zhangchen Xu ⋅ Yunhong He ⋅ Zhengqing Yuan ⋅ Yujun Zhou ⋅ Kehan Guo ⋅ Chaoran Chen ⋅ Toby Li ⋅ Stefan Feuerriegel ⋅ Xiangliang Zhang
Interactive narrative is becoming a practical frontier for language-model agents, spanning game NPCs, collaborative story production, emotionally responsive companions, and story-grounded interface generation. Yet most current evaluations still rely on static prompts, isolated story outputs, or post-hoc ratings, leaving open whether frontier models can sustain coherent, adaptive, emotionally grounded interaction over time. We introduce NARRA-Gym, an executable evaluation environment that turns a sparse emotional seed into a complete interactive story episode and records the full agent loop: story construction, memory updates, planning, pacing control, and optional artifact generation. The benchmark jointly tests five coupled capabilities---creative story generation, long-context state tracking, character simulation, empathic personalization, and interactive artifact generation---through a fixed LLM-as-judge sweep and a human preference study with user-provided experiences. Across nine generator models and eight benchmark personas, Claude Sonnet 4.6 is the strongest and most robust performer, while Claude Opus 4.6 forms a high-variance second tier and several middle-tier models show different tradeoffs between story quality and user experience. Human rankings recover the same top tier but shift the middle of the ordering, indicating that automated judges are useful for broad screening while human preference remains important for style, tone, and emotional resonance. Finally, failure analysis shows that the key remaining bottleneck is not surface fluency but resistance-sensitive personalization: models can write polished scenes while losing track of what the user is resisting or emotionally unable to accept.
Show more
Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?
Anton Korznikov ⋅ Andrey V Galichin ⋅ Alexey Dontsov ⋅ Oleg Rogov ⋅ Ivan Oseledets ⋅ Elena Tutubalina
Sparse Autoencoders (SAEs) have emerged as a promising tool for interpreting neural networks by decomposing their activations into sparse sets of human-interpretable features. Despite much excitement, a growing number of negative results in downstream tasks casts doubt on whether SAEs recover meaningful features. To directly investigate this, we perform two complementary evaluations. On a synthetic setup with known ground‑truth features, we demonstrate that three out of four state‑of‑the‑art SAE architectures recover only 7–9\% of true features despite achieving around 71\% explained variance, showing that strong reconstruction alone is insufficient to guarantee meaningful feature recovery. To evaluate SAEs on real activations, we introduce three baselines that constrain SAE feature directions or their activation patterns to random values. Through extensive experiments across multiple SAE architectures, we show that our baselines match fully-trained SAEs in interpretability (0.87 vs 0.90), sparse probing (0.69 vs 0.72), and causal editing (0.73 vs 0.72). These results show that current evaluation metrics are insufficient to certify that SAEs have learned meaningful features, and we offer our baselines as a reusable protocol for future SAE evaluation.
Show more
ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation
Jungwoo Oh ⋅ Hyunseung Chung ⋅ Junhee Lee ⋅ Min-Gyu Kim ⋅ Hangyul Yoon ⋅ Ki Seong Lee ⋅ Youngchae Lee ⋅ Muhan Yeo ⋅ Edward Choi
While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce **ECG-Reasoning-Benchmark**, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a systematic shortfall in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit consistently low success rates in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, highlighting a limitation in their visual grounding capabilities and motivating robust, reasoning-centric medical AI. The code and data are available at https://anonymous.4open.science/r/ecg-reasoning-benchmark-anonymized-BB3C.
Show more
GenAI Evaluation Results are Largely Artifacts of Evaluation Design Choices: Evidence from Audit Studies of Resume Screening
Nicole Meister ⋅ Hannah Cha ⋅ Mohammed Alsobay ⋅ Alexandra Chouldechova ⋅ Alex Dow ⋅ Hanna Wallach ⋅ Jennifer Wortman Vaughan ⋅ Carlos Guestrin ⋅ Tatsunori Hashimoto ⋅ Solon Barocas
Generative AI is designed to support many tasks, many ways of framing those tasks, and seemingly endless variation in task inputs. But the same flexibility that makes these systems powerful also makes them difficult to evaluate reliably. This flexibility exacerbates the numerous, but often underrecognized and consequential, \emph{researcher degrees of freedom} where even seemingly minor experiment design decisions can substantially affect findings. We demonstrate this problem through an analysis of 36 papers on gender and racial bias in generative AI-assisted resume screening. Although these studies address the same broad research question, they reach strikingly divergent conclusions. To explain this divergence, we adapt recent work on integrative experiment design from the social sciences, and decompose the dimensions along which the studies vary. We then run 1,134 counterfactual experiments by combinatorially varying the experimental modules (names, job-resume pairings, and models) across three baseline experimental frameworks. We find the results prove remarkably brittle: a different design choice flips the conclusion of the reproduced experiment around 22\% of the time. For example, seemingly inconsequential design choices, such as the phrasing of a job description, can reverse the observed conclusion. However, this instability is not always purely stochastic. In some settings, we can build a model that predicts the result of unseen counterfactual experiments with a mean held-out predictive $R^2$ of 0.807. This suggests that despite sensitivities to difference in evaluation design, there exists a learnable structure.
Show more
RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments
Babak Rahmani ⋅ Sebastian Dziadzio ⋅ Joschka Strüber ⋅ Sergio Hernández-Gutiérrez ⋅ Matthias Bethge
For most of scientific history, researchers studying behavior could only infer hidden mechanisms from outward actions—an inverse problem that becomes more tractable when observation is augmented by targeted intervention. We pose a computational analogue: given only behavioral traces of an agent in a game environment, can a learner reconstruct the underlying decision program as executable code, and how much does this reconstruction improve with the ability to design controlled experiments? We introduce RevengeBench, a benchmark of 75 LLM generated, Elo-calibrated policies across five game environments, drawn from CodeClash tournament trajectories. The learner observes the hidden target policy play against sampled opponents and designs behavioral probes in the form of custom opponent policies that elicit informative behavior. It then submits an executable hypothesis, which is evaluated using continuous action-distance metrics. We further validate that recovered code carries informative signal in downstream player-versus-player tournaments. Across twelve frontier LLMs, recovery quality varies substantially (34–72\% of initial distance closed), with reconstructed policies yielding measurable competitive advantage, particularly for weaker models that otherwise struggle to design effective counter-strategies. Our benchmark positions behavioral recovery of programmatic policies as a tractable inverse problem in code-space, opening a path to opponent modeling, policy interpretability, and the broader question of inferring latent mechanisms from observations.
Show more
B-XAIC Dataset: Benchmarking Explainable AI for Graph Neural Networks Using Chemical Data
Magdalena Proszewska ⋅ Tomasz Danel ⋅ Antoni Antoszek ⋅ Dawid Damian Rymarczyk
Understanding the reasoning behind deep learning model predictions is crucial in cheminformatics and drug discovery, where molecular design determines their properties. However, current evaluation frameworks for Explainable AI (XAI) in this domain often rely on artificial datasets or simplified tasks, employing data-derived metrics that fail to capture the complexity of real-world scenarios and lack a direct link to explanation faithfulness. To address this, we introduce B-XAIC, a novel benchmark constructed from real-world molecular data and diverse tasks with known ground-truth rationales for assigned labels. Through a comprehensive evaluation using B-XAIC, we reveal limitations of existing XAI methods for Graph Neural Networks (GNNs) in the molecular domain. This benchmark provides a valuable resource for gaining deeper insights into the faithfulness of XAI, facilitating the development of more reliable and interpretable models.
Show more
pyCD: A Unified Benchmark for Reliable Evaluation of Cognitive Diagnosis Models
Youheng Bai ⋅ Xueyi Li ⋅ Tengteng Cheng ⋅ Mingliang Hou ⋅ Teng Guo ⋅ Jiaqi Zheng ⋅ Zitao Liu
Cognitive diagnosis (CD) infers students' mastery of knowledge components (KCs) from their responses, supporting personalized feedback and exercise recommendation. Cognitive diagnosis models (CDMs) are often ranked by response-prediction AUC under inconsistent experimental protocols, while their KC-level mastery profiles are not consistently audited under a shared protocol. We introduce \texttt{pyCD}, a unified benchmark of eleven CDMs on six public datasets that fixes preprocessing, splits, hyperparameter search, and stopping rules; the benchmark reports KC-level diagnostic metrics for learned mastery alongside standard prediction metrics. Two findings show that gains in AUC do not translate into gains in diagnostic quality. First, the AUC leaderboard is sharply compressed and sensitive to protocol choices: modern CDMs often cluster within $0.005$--$0.020$ AUC, with the per-dataset best non-IRT CDM improving over IRT by only $0.017$ AUC on average; removing validation-AUC early stopping can shift AUC by up to $0.13$ on individual cells. Second, the AUC gaps that remain do not reliably transfer into diagnosis: the top-1 AUC model is typically outside the top-3 by KC-level diagnostic agreement (DOA). The same mismatch extends to exercise difficulty: against Junyi's human pairwise difficulty ratings, learned CDM difficulty parameters rank worse than both an empirical accuracy-based baseline and zero-shot LLMs given only exercise titles. CDM evaluations should therefore report diagnostic outputs directly rather than infer them from predictive gains; \texttt{pyCD} provides a unified protocol for doing so. Code, splits, and configurations are available at \url{https://anonymous.4open.science/r/pyCD-NeurIPS2026-60B8}.
Show more
MotionHalluc: Diagnosing Kinematic Hallucinations in Fine-Grained Motion Reasoning
Weile Guo ⋅ Shenghong He ⋅ Danying Mo ⋅ Chengdong Xu ⋅ Xuexun Liu ⋅ Chao Yu
Motion instruction generation in cross-video comparison aims to produce corrective feedback that describes the differences between a query and a reference motion. However, existing models often generate instructions that exhibit motion hallucinations, failing to reflect actual kinematic differences between paired videos. To systematically investigate these hallucinations, we introduce MotionHalluc, a dedicated benchmark for evaluating motion hallucinations in paired-video comparison. MotionHalluc comprises 1540 fine-grained questions over 553 video pairs, evaluating hallucinations along three core dimensions: (1) directional hallucination, (2) attributional hallucination, and (3) temporal hallucination. Extensive evaluations of state-of-the-art large multimodal models demonstrate high susceptibility to these hallucinations. Furthermore, we provide Perceive-Parse-Verify (PPV) as a training-free measurements extraction and verification baseline that converts candidate instructions into executable measurement queries and supplies kinematic measurements at inference time. Our results show that this simple measurements injection yields an average 10.6% performance gain across models, suggesting that motion reasoning with explicit quantitative measurements is a key factor in reducing hallucinations in cross-video comparison.
Show more
Code Review Bench: An Automated Benchmark for Evaluating the Software Factory
Ashley Zhang ⋅ Narmeen Oozeer ⋅ Aleksandr Zverianskii ⋅ Daniel Ng ⋅ Tasana Pejovic ⋅ Antía García ⋅ Jacob Clyne ⋅ Ravpreet Setia ⋅ Akshay Utture ⋅ Etan J Ginsberg ⋅ Shriyash Upadhyay ⋅ Fazl Barez
As AI agents ship more code with minimal human oversight, code review becomes the critical quality-control checkpoint — but one without a reliable, objective verifier: review quality is nuanced and ultimately a judgment call by the PR author. Existing benchmarks are structurally misaligned with this: annotators are third-party reviewers, like the bots they evaluate, not the PR authors who actually decide what to fix. They also rely on small, static issue sets that require full re-evaluation as tools update, limiting how frequently progress can be tracked. We introduce **CodeReviewBench**, built on a fundamentally different signal: *revealed developer preferences* — if a developer acts on a bot's suggestion, we treat it as useful; if they ignore it, we do not. This signal is mined daily from the public GitHub event stream, yielding 500K+ judged PRs across 18 tools, and refreshes continuously as tools evolve. We validate the developer preference against an independent offline benchmark: the two agree on precision rankings and independently detect the same product updates within days of vendor releases. Together, these form the benchmark.
Show more
DriveStreamBench: Evaluating User-Conditioned Watch-and-Notify in Streaming Driving Video
Yi Wang ⋅ Xin Zhao ⋅ Tian Meng ⋅ Rui Dai ⋅ Chi Zhang ⋅ Cui Tang ⋅ Bingzheng Liu ⋅ Fucheng Wang ⋅ Haopeng Zhang ⋅ Ruidong Ding ⋅ Yang Li ⋅ Kaikui Liu ⋅ Xiangxiang Chu
Vision-language models (VLMs) have made steady progress as in-vehicle driving assistants, yet existing evaluations largely remain offline, focusing on perception and short-term reasoning over pre-recorded driving scenes. A complementary capability is still underexplored: whether a model can continuously monitor a causal driving video stream and issue a calibrated alert when a user-specified condition is met. Existing driving benchmarks mainly assess reactive understanding, while recent streaming video benchmarks are developed largely outside the driving domain. To this end, we introduce DriveStreamBench, a driving and streaming video benchmark that organizes this capability spectrum into four cognitive levels constructed from a shared pool of driving events, to support level-wise capability profiling and gated evaluation. We evaluate various vision-language models along with DriveStream-SFT, a supervised reference baseline trained on the benchmark's training split, under a unified streaming protocol. DriveStreamBench tests a basic but necessary capability, and all evaluated models fall short of it; current vision-language models are not yet ready for user-conditioned alerting in driving. Resources are available at https://anonymous.4open.science/r/DriveStreamBench/ and https://dataverse.harvard.edu/previewurl.xhtml?token=01f6bc4c-d809-4e40-b025-b6d783c86571.
Show more
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Bowen Ye ⋅ Rang Li ⋅ Qibin Yang ⋅ Yuanxin Liu ⋅ Linli Yao ⋅ Hanglong Lv ⋅ Zhihui Xie ⋅ Chenxin An ⋅ Lei Li ⋅ Lingpeng Kong ⋅ Qi Liu ⋅ Zhifang Sui ⋅ Tong Yang
Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with 300 human-verified tasks spanning 9 categories across three groups: general service orchestration, multimodal perception and interaction, and multi-turn professional dialogue. To enable trajectory-aware grading, each run is recorded through three independent evidence channels: execution traces, audit logs, and environment snapshots, yielding 2,159 fine-grained rubric items. The scoring protocol evaluates Completion, Safety, and Robustness, with Average Score, Pass@k, and Pass^k across three trials to distinguish genuine capability from lucky outcomes. Experiments on 14 frontier models show that: (1) Trajectory-opaque evaluation is systematically unreliable, missing 44% of safety violations and 13% of robustness failures detected by our framework. (2) Capability does not imply consistency, with Pass@3 remaining stable under error injection while Pass^3 dropping by up to 24 percentage points. (3) Agent capability is strongly multi-dimensional, with model rankings varying across task groups and metrics, indicating that our heterogeneous evaluation coverage is essential. Claw-Eval highlights directions for developing agents that are not only capable but reliably deployable.
Show more
Are LLMs Good at Feature Engineering? Evidence from a Controlled Synthetic Benchmark
Viacheslav Surkov ⋅ Yasha Pushak ⋅ Ritesh Ahuja ⋅ Rafael Pires ⋅ Anne-marie Kermarrec ⋅ Damien Hilloulin ⋅ Rhicheek Patra ⋅ Sungpack Hong
Large Language Models (LLMs) are increasingly used for automated feature engineering (FE) on tabular data, but prevailing benchmarks are ill-suited to evaluate this capability. They often assume i.i.d. samples, lack temporal dynamics and distribution shifts common in production, provide no known performance ceiling, and their artifacts (e.g., winning solutions) may enter LLM training corpora—blurring genuine discovery vs. memorization or spurious correlation. We introduce TabGen, a controllable synthetic generator that produces two synchronized views: an observable view for learners and an oracle view with minimal sufficient features, yielding a deterministic upper bound. TabGen composes non-i.i.d. dynamics to control inter-row interactions and distribution shift. Using TabGen, we evaluate several FE strategies, including Analyze&Act—a hypothesis-driven FE workflow proposed here. In no-drift settings, methods that instruct LLMs to iteratively analyze and refine data hypotheses consistently outperform alternatives, though LLM-driven FE shows substantial variance. Under drift, all methods degrade, and many fail to generalize—often underperforming the no-FE baseline—indicating FE-level overfitting to training-period correlations that do not transfer to future dynamics. Traditional i.i.d. benchmarks, by design, cannot reveal this failure mode.
Show more
SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
Kaihua Qin ⋅ Dawn Song ⋅ Arthur Gervais
Smart contract decompilation aims to recover high-level source code from bytecode, but evaluating decompilers remains difficult because existing studies use narrow datasets, inconsistent metrics, and limited semantic consistency checks. This gap is increasingly important as large language models (LLMs) begin to generate source-like Solidity that may compile and appear plausible, even when its semantics diverge from the original contract. We introduce SCDBench, a dataset and benchmark methodology for LLM-based smart contract decompilation. The dataset contains $600$ real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints. SCDBench evaluates decompiler outputs through four cumulative stages: format completeness, compilability, Application Binary Interface (ABI) recovery, and semantic consistency via differential replay. We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning and a zero-shot compilation-repair setting. The results show that frontier LLMs can often produce structured and compilable Solidity, but achieving semantic consistency remains far from solved: the best-performing frontier model perfectly decompiles only $42/600$ contracts. We further show that introducing same-model compilation repair substantially improves performance at modest additional cost. SCDBench establishes a common ground for rigorous, reproducible evaluation and aims to accelerate the development of reliable smart contract decompilers for blockchain security and transparency.
Show more
Can Folding Models Tell Binders from Bluffers? Evidence from POISK: The Patent-Derived Antibody Dataset
Daria Tupikina ⋅ Andrea Roncoli ⋅ Alexander Bujotzek ⋅ Brennan Abanades Kenyon
Structure prediction models are increasingly deployed as zero-shot digital screens in antibody design, operating under the assumption that high folding confidence implies true binding. However, rigorous validation of this premise has been limited by small test sets, narrow target diversity, and the risk of data leakage from training-adjacent structures. Here we introduce POISK: Patent-extracted Organization of Immunoglobulin Sequence Knowledge, a large-scale dataset derived from the US patent literature, comprising 89074 antibody-antigen records extracted from 10110 patents using an LLM-based pipeline. From this resource, we curate a balanced benchmark set of 22494 complexes: 11247 patent-validated binders without a known resolved structure paired with putative non-binders constructed from sequence-similar human repertoire antibodies. Using this dataset, we benchmark six leading structure prediction models (AlphaFold-Multimer, Boltz-2, Intellifold v2, OpenFold3, Protenix-v1, and Protenix-v2) on their ability to discriminate binders from decoys. The best-performing model, Protenix-v2, achieves 87% pairwise accuracy on the most confident half of predictions and correctly identifies the true binder from a pool of 50 candidates in 32% of trials; this represents a substantial enrichment over random selection, yet remains far from reliable single-candidate identification. Structural consensus across models also discriminates binders from decoys, with pairwise accuracy reaching 73.1% when three or more models converge on a pose DockQ $\geq$ 0.23, covering around 54.2% of pairs). Yet epitope localization remains poor: the best model contacts annotated epitope residues in only 45% of cases, indicating that high confidence reflects learned statistical associations rather than accurate physical binding modes. Our results suggest that current folding models provide useful but limited signal for antibody screening and should be complemented by orthogonal methods. We release POISK as a community resource for future benchmarking and model training.
Show more
DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models
Hao Liang ⋅ Zhengyang Zhao ⋅ Mingrui Chen ⋅ Meiyi Qiang ⋅ Lu Ma ⋅ Rongyi YU ⋅ Hengyi Feng ⋅ Shixuan Sun ⋅ Zimo Meng ⋅ Xiaochen Ma ⋅ Xuanlin Yang ⋅ Qifeng Cai ⋅ Ruichuan An ⋅ Bohan Zeng ⋅ Zhen H Wong ⋅ Chengyu Shen ⋅ Runming He ⋅ ZhaoYang Han ⋅ Yaowei Zheng ⋅ Fangcheng Fu ⋅ Conghui He ⋅ Bin CUI ⋅ Zhiyu li ⋅ Weinan E ⋅ Wentao Zhang
Data-centric training—selecting, mixing, and reweighting data during LLM optimization—has produced a rapidly growing body of algorithms, yet every new method ships as a bespoke codebase with its own interface, training protocol, and evaluation setup, making head-to-head comparison and reproducible evaluation nearly impossible. We present DataFlex, the first unified benchmark and evaluation platform for data-centric training of LLMs. DataFlex contributes: (i) a standardized evaluation protocol that runs selection, mixture, and reweighting methods under identical seeds, data splits, hardware, and metrics; (ii) a reference implementation of ten representative algorithms (six data selection, three data mixture, one data reweighting) re-engineered on top of LLaMA-Factory to share model, dataloader, and distributed-training infrastructure (including DeepSpeed ZeRO-3 with full-rank gradient acquisition); and (iii) an empirical benchmark whose head-to-head results yield reproducible findings that no single existing codebase could produce: dynamic selection methods consistently outperform static full-data training on MMLU across Mistral-7B and Llama-3.2-3B, and the gain is larger on smaller models (up to 13.3 points); DoReMi and ODM yield complementary wins on high- vs. low-resource domains when pretraining Qwen2.5-1.5B on SlimPajama at 6B and 30B tokens; and DataFlex's implementation is consistently faster than the original codebases of LESS, TSDS, and MoE-SFT. By unifying fragmented methods under a single evaluation substrate, DataFlex provides the community with a reproducible, extensible basis for future data-centric research.
Show more
How to Interpret Agent Behavior
Sophia Gao ⋅ Kaiser Sun ⋅ Jen-Tse Huang ⋅ Katherine Van Koevering ⋅ Sijie Ji ⋅ Heyuan Huang ⋅ Weiyan Shi ⋅ Zhuoran Lu ⋅ Ziang Xiao ⋅ Daniel Khashabi ⋅ Mark Dredze
Autonomous agents such as Claude Code and Codex now operate for hours or even days. Understanding their runtime behavior has become critical for downstream tasks such as diagnosing inefficiencies, fixing bugs, and ensuring better oversight. A primary way to gain this understanding is analyzing the reasoning trajectories and execution traces these agents generate. Yet such data remains in unstructured natural-language form, making it difficult for humans to interpret at scale. We introduce ACT*ONOMY (a combination of Action and Taxonomy, pronounced /ækˈtɑːnəmi/), a taxonomy for describing and analyzing agent behavior at runtime. has two components: (1) the taxonomy itself, developed through Grounded Theory and structured as a three-level hierarchy of 10 actions, 46 subactions, and 120 leaf categories; and (2) an open repository that hosts the living taxonomy, provides an automated analysis pipeline that applies it to agent trajectories analysis, and defines an extension protocol for community contributions. Our experiments show that ACT*ONOMY can compare behavioral profiles across agents and characterize a single agent's behavior across diverse trajectories, surfacing patterns indicative of failure modes. By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently, enabling better oversight and control.
Show more
Beyond Accuracy: A Diagnostic Benchmark for Hypothesis-Driven Experiment Planning in LLM Agents
Ryo Kuroki ⋅ Amin Mansouri ⋅ Philippe Schwaller
Evaluating large language model (LLM) agents on scientific tasks typically reduces to final-task accuracy, which obscures where reasoning succeeds or breaks down. We introduce a benchmark and diagnostic evaluation framework for hypothesis-driven adaptive experimental planning, instantiated in kinetic mechanism identification. The framework goes beyond final accuracy in two ways. First, we evaluate exploration behavior by quantifying the extent to which agents exhibit space-filling or corner-seeking patterns in the experimental design space. Second, it decomposes the reasoning process into four interpretable steps; belief update, hypothesis discrimination, discriminative experiment design, and predictive reasoning under interventions, each scored independently on reasoning traces. Across ten LLM agents, we find that stronger agents outperform both adaptive (Bayesian optimization) and non-adaptive (Latin hypercube) baselines and exhibit exploration patterns inconsistent with naive heuristics. Decomposing reasoning reveals that the stronger agents score consistently well across all four steps, whereas weaker agents often succeed at belief update and hypothesis discrimination but fail at discriminative experiment design and predictive reasoning. This contrast shows that our framework identifies a specific reasoning bottleneck: weaker models can recognize plausible hypotheses and articulate their differences, but struggle to translate this understanding into concrete predictions and experimental designs. The code and dataset are available at https://github.com/krfdq48k4p-arch/HypothesisDrivenExperimentPlanningBench and https://huggingface.co/datasets/xq8wvm/HypothesisDrivenExperimentPlanningBench, respectively.
Show more
PhyTS: A Benchmark for Scientific Time Series
Benedict Armstrong ⋅ Jeroen Audenaert ⋅ Hannah P Binney ⋅ Alice Cheng ⋅ Maarten De Vos ⋅ Luis F Domingues ⋅ Allison Eto ⋅ Mary Feliz ⋅ Joseph Formaggio ⋅ Jessica Fry ⋅ Philip Harris ⋅ Ilay Kamai ⋅ Erik Katsavounidis ⋅ Mykyta Kliapets ⋅ Konstantinos Kontras ⋅ Aobo Li ⋅ Paul Liang ⋅ Daniel Muthukrishna ⋅ Christina Reissel ⋅ Daniela Rus ⋅ T. Konstantin Rusch ⋅ Avi Shporer ⋅ Penny L Slocum ⋅ Talia E Weiss ⋅ Lindley Winslow ⋅ Kyungseop Yoon
We introduce PhyTS, a benchmark suite of precision scientific time series datasets for machine learning, spanning experiments in gravitational-wave detection, dark matter searches, neutrino mass determination, and exoplanet discovery. Despite their diverse scientific goals, these domains share a common challenge: recovering weak, structured signals and estimating underlying physical parameters from noise-dominated measurements. Unlike standard sequence modeling benchmarks such as audio and speech, these data exhibit non-Gaussian and nonstationary noise, long-range temporal correlations, detector-specific systematics, irregular sampling, and signals that are sparse, weak, or only partially modeled. As a result, they provide a challenging testbed for evaluating whether modern AI methods can support downstream scientific inference. We provide standardized tasks, data splits, and evaluation protocols for denoising, signal recovery, and parameter inference across physics domains, along with baseline results. By unifying diverse weak-signal inference problems under a common framework, this benchmark aims to enable reproducible evaluation and accelerate the development of more robust, interpretable, and physically grounded methods for scientific time series analysis.
Show more
BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories
Fei Lin ⋅ Tengchao Zhang ⋅ Ziyang Gong ⋅ Xiaotong Yu ⋅ Bohan Zhang ⋅ Yifan Zhou ⋅ Qihao Yang ⋅ Ji Dai ⋅ Dong Li ⋅ Yining Jiang ⋅ Qinghua Ni ⋅ Jun Huang ⋅ Qiang Zhang ⋅ Yue Zhou ⋅ Yonglin Tian ⋅ Zhihang Zhong ⋅ Xue Yang ⋅ Fei-Yue Wang
Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated QA samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment and state understanding, apparatus connection, operation process reasoning, and result understanding. All questions and annotations are manually created and reviewed by chemistry and embodied AI experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, scientific, chemical, and embodied MLLMs on BEAKER. Results show that current models can recognize laboratory objects and states, but still struggle to ground experimental knowledge into operable regions, feasible trajectories, and long-horizon procedural reasoning. BEAKER thus reveals a clear gap between multimodal laboratory perception and action-grounded experimental understanding, providing a diagnostic benchmark for future MLLMs in SDCLs.
Show more
Origami as a Real-Image Benchmark for Procedural State-Transition Reasoning
Jun Suzuki ⋅ Reina Akama ⋅ Ikumi Numaya ⋅ Kise Yoshida ⋅ Yuka Saito ⋅ Sumika Tsuda ⋅ Takahiro Shimizu ⋅ Miori Sagara
General-purpose embodied AI systems are increasingly expected to interpret multimodal instructions and perform fine-grained physical tasks. However, many manipulation benchmarks provide limited support for jointly evaluating procedural correctness, local geometric sensitivity, and human-guided correction. We propose origami as a benchmark based on real images for procedural state-transition reasoning in embodied AI. Origami requires precise operations such as tucking, inserting, reversing, and folding overlapping layers, making it a focused domain for studying physical state transitions. We introduce a dataset of step-by-step folding images, textual entries, and difficulty annotations, and define two tasks: autonomous state transitions from multimodal instructions and human-guided collaborative transitions. Unlike crease-pattern- or diagram-based origami benchmarks, our setting exposes agents to actual intermediate folding states. For an initial simulator-based evaluation, we use a multimodal LLM controller to evaluate procedural understanding, action-interface grounding, and retry-based correction.
Show more
HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing
Zhenjie Yang ⋅ Xingyu Jiao ⋅ Guopeng Zhong ⋅ Shuzhe Yang ⋅ Shi Che ⋅ Chao Wu ⋅ Chenyu Jiang ⋅ Dongjie Zhang ⋅ Yideng Zhang ⋅ Zheng Zhang ⋅ Muyun Jiang ⋅ Haisheng Su ⋅ Shuang Jin ⋅ Donghang Zhang ⋅ Chao Yang ⋅ Li Chen ⋅ Hongyang Li ⋅ Zuxuan Wu ⋅ Yu-Gang Jiang ⋅ Xiaosong Jia ⋅ Junchi Yan
Robotic manipulation with dexterous hands is a cornerstone of Embodied AI, yet its progress is stifled by the high cost of collecting embodiment-aware teleoperation data. While abundant egocentric videos of human hands offer a scalable alternative, the profound discrepancies in appearance, articulation, and camera viewpoints between human and robotic data raise significant challenges for co-training. Though existing general image-editing models demonstrate strong capabilities, they lack necessary embodiment-specific priors to fully bridge this gap. In this work, we present HandEdit, a unified large-scale embodiment-aware image-editing dataset and benchmark specifically designed to transform human hands and arms into various dexterous robotic embodiments within egocentric frames. HandEdit comprises over 200M editing instances derived from five diverse source datasets, covering 26 distinct URDFs, including 13 hand-only and 13 hand-arm configurations. Alongside the dataset, we establish a unified benchmark protocol with two tracks: Hand-only and Hand-Arm, supporting URDF-conditioned evaluation. We conduct extensive evaluations of 11 representative image-editing baselines using a multi-dimensional metric suite, including generic similarity metrics, VLM-based judgment, and embodied-aware metrics. HandEdit serves as a critical resource at the intersection of image editing and robotics: it advances embodiment-aware editing models while enabling scalable dexterous robotic learning from abundant human video data, paving the way for more generalizable Embodied AI. Code, datasets, and benchmarks are available at \url{https://anonymous.4open.science/r/HandEdit}.
Show more
Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
Roger Creus Castanyer ⋅ Pablo Samuel Castro ⋅ Glen Berseth
AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present **Agentick**, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3--10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.
Show more
RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
Xinbo Xu ⋅ Ruihan Yang ⋅ Haiyang Shen ⋅ Wendong XU ⋅ Bofei Gao ⋅ Ruoyu Wu ⋅ Kean Shi ⋅ Weichu Xie ⋅ Xuanzhong Chen ⋅ Ming Wu ⋅ Jason Zeng ⋅ Michael Heinrich ⋅ Liang Chen ⋅ Kuan Li ⋅ Baobao Chang
Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present ROADMAPBENCH, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to the high scores these models attain on existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.
Show more
TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders
Wei Pang ⋅ Xiangru Jian ⋅ Hehan Li ⋅ Zhixuan Yu ⋅ Jingxi Xue ⋅ Jinyang Li ⋅ Zhengyuan Dong ⋅ Xinjian Zhao ⋅ Hao Xu ⋅ Chao Zhang ⋅ Reynold Cheng ⋅ M. Tamer Özsu ⋅ Tianshu Yu
Tabular encoders are usually evaluated inside task-specific end-to-end pipelines, so models from different training paradigms are difficult to compare directly even when they operate on similar tabular signals. We introduce TRL-Bench, a multi-granular tabular representation learning (TRL) benchmark that standardizes cross-paradigm representation-level evaluation: each encoder exports row-, column-, or table embeddings through its supported wrapper, and shared lightweight heads probe them across three suites: TRL-CTbench (column/table), TRL-Rbench (row), and TRL-DLTE (compositional Data-Lake Table Enrichment spanning all three granularities). To support this standardized setting, we release curated benchmark assets and task reformulations, including 50 OpenML tables with 123 verified targets, 16 row-pair linkage rewrites, and a 47,772-table DLTE lake derived from 1,379 parent tables. Across 20 models and 16 tasks, TRL-Bench shows that once downstream conditions are standardized, encoder quality is capability-specific rather than captured by a single leaderboard. At the column/table level, generic text encoders often lead on tasks with strong surface-text signal, while tabular specialists win where their pretraining objective aligns with the task. At the row level, within-table prediction and cross-table linkage favor different training regimes, with the row-matching stage of DLTE pipelines correlating strongly with atomic linkage performance. In TRL-DLTE, the strongest pipelines combine capability-matched specialists rather than reuse a single encoder, and top end-to-end quality depends on non-additive compositional fit rather than per-stage marginal rank alone. TRL-Bench provides a common protocol for measuring reusable signal in exported tabular representations under shared downstream conditions.
Show more
Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
David Schmotz ⋅ Luca Beurer-Kellner ⋅ Sahar Abdelnabi ⋅ Maksym Andriushchenko
LLM agents are evolving rapidly, powered by code execution, tools, and the recently introduced agent skills feature. Skills allow users to extend LLM applications with specialized thirdparty code, knowledge, and instructions. Although this can extend agent capabilities to new domains, it creates an increasingly complex agent supply chain, offering new surfaces for prompt injection attacks. We identify skill-based prompt injection as a significant threat and introduce Skill-Inject, a benchmark evaluating the susceptibility of widely-used LLM agents to injections through skill files. Skill-Inject contains 202 injection-task pairs with attacks ranging from obviously malicious injections to subtle, context-dependent attacks hidden in otherwise legitimate instructions. We evaluate frontier LLMs on Skill-Inject, measuring both security in terms of harmful instruction avoidance and utility in terms of legitimate instruction compliance. Our results show that today’s agents are highly vulnerable with up to 80% attack success rate with frontier models, often executing extremely harmful instructions including data exfiltration, destructive action, and ransomwarelike behavior. They furthermore suggest that this problem will not be solved through model scaling or simple input filtering, but that robust agent security will require context-aware authorization frameworks.
Show more
Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
Junjue Wang ⋅ Weihao Xuan ⋅ Heli Qi ⋅ Pengyu Dai ⋅ Kunyi Liu ⋅ Hongruixuan Chen ⋅ Zhuo Zheng ⋅ Junshi Xia ⋅ Stefano Ermon ⋅ Naoto YOKOYA
Operational disaster response goes beyond damage assessment, requiring responders to integrate multi-sensor signals, reason over road networks, populations and key facilities, plan evacuations, and produce actionable reports. However, prior work largely isolates remote-sensing perception or evaluates generic tool use, leaving the end-to-end workflows of emergency operations underexplored. In this paper, we introduce **D**isaster **O**perational **R**esponse **A**gent benchmark (**DORA**), the first agentic benchmark for end-to-end disaster response: 515 expert-authored tasks across 45 real-world disaster events spanning 10 types, paired with expert-verified, replayable gold trajectories totaling 3,500 tool-call steps. Tasks span five dimensions that cover the operational disaster-response pipeline: disaster perception, spatial relational analysis, rescue and evacuation planning, temporal evolution reasoning, and multi-modal report synthesis. Agents compose calls from a 108-tool MCP library over heterogeneous geospatial data: optical, SAR, and multi-spectral imagery across single-, bi-, and multi-temporal sequences (0.015-10m GSD), complemented by elevation and social vector layers. We comprehensively evaluate 13 frontier LLMs on our benchmark, revealing three persistent challenges: 1) disaster-domain grounding exposes unique failure modes (damage-semantic grounding, sensor-modality mismatch, and disaster-pipeline composition); 2) agents are doubly bottlenecked by tool selection and argument grounding, where gold tool-order hints improve accuracy by only 1.08-4.40%, and alternative scaffolds yield at most a 3.24% gain; 3) compositional fragility scales with trajectory length, the agent-to-gold gap widening from 7% to 56% on long pipelines. DORA establishes a rigorous testbed for operationally reliable disaster-response agents.
Show more
CoffeeBench: A Benchmark for Long-Horizon Strategic Decision-Making in Multi-Agent Economies
Issa Sugiura ⋅ Daichi Hattori ⋅ Kazuo Araragi ⋅ Keita Ogawa ⋅ Shota Onose ⋅ Taro Makino ⋅ Teppei Usuki ⋅ Takashi Ishida
As LLM agents advance beyond short-horizon tasks, evaluating their ability to perform long-horizon strategic decision-making in realistic multi-agent settings remains a key challenge. Existing agent benchmarks typically focus on isolated task completion or interactions with rule-based or fixed-policy counterparts, limiting their ability to capture strategic adaptation and emergent behaviors arising from sustained multi-agent interactions. We introduce \textbf{CoffeeBench}, a benchmark for evaluating LLM agents in a dynamic multi-agent economic environment consisting of farmers, roasters, and retailers forming a multi-stage supply chain. The environment consists of autonomous firms interacting over a multi-month horizon with evolving supply, demand, and pricing, creating sustained competition and supply-chain dynamics. In each evaluation, the evaluated model controls one firm and must make sequential decisions on procurement, production, pricing, and negotiation to maximize cumulative net income. CoffeeBench enables the study of emergent behaviors arising from repeated strategic interactions, including negotiation dynamics, pricing discipline, and coordination failures. We evaluate several frontier and pareto-frontier LLMs on CoffeeBench to analyze their economic performance and strategic behavior. Most evaluated models achieve positive net income, with GPT-5.5 achieving the highest mean net income. In contrast, Claude~Haiku~4.5 frequently yields negative net income, exhibiting an idle-drift failure mode in which agents collapse into inactivity despite coherent plans. We further find that stronger models proactively negotiate with counterparties through frequent messaging, whereas weaker models trade reactively with limited communication. CoffeeBench provides a controllable testbed for studying strategic interaction, coordination, and long-horizon behavior in multi-agent economies.
Show more
fev-bench: A Realistic Benchmark for Time Series Forecasting
Oleksandr Shchur ⋅ Abdul Fatir Ansari ⋅ Ali Caner Turkmen ⋅ Lorenzo Stella ⋅ Nick Erickson ⋅ Pablo A Guerron ⋅ Michael Bohlke-Schneider ⋅ Yuyang (Bernie) Wang
Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of time series foundation models (TSFMs). Existing benchmarks often have limited domain coverage or overlook real-world settings such as tasks with covariates. Their aggregation procedures frequently lack statistical rigor, making it unclear whether performance differences reflect true improvements or random variation. Many benchmarks lack consistent evaluation infrastructure or are too rigid for integration into existing pipelines. To address these gaps, we propose fev-bench, a benchmark of 100 forecasting tasks across seven domains, including 46 with covariates. Supporting the benchmark, we introduce fev, a lightweight Python library for forecasting evaluation emphasizing reproducibility and integration with existing workflows. Using fev, fev-bench employs principled aggregation with bootstrapped confidence intervals to report performance along two dimensions: win rates and skill scores. We evaluate various models on fev-bench and find that existing TSFMs often miss accuracy by ignoring covariates, pointing to a clear direction for future work.
Show more
BodyBench: Evaluating Adversarial Image Defenses Against AI Nudification Inpainting
Li Qiwei ⋅ Salma Abdel Magid ⋅ Olga Russakovsky ⋅ Eric Gilbert ⋅ Sarita Schoenebeck
Adversarial perturbation defenses protect images from generative editing by adding imperceptible noise. Yet none of these methods have been evaluated against AI nudification, despite AI-generated non-consensual intimate imagery (AIG-NCII) being among the most documented harms of generative AI and produced largely through inpainting. We argue this is a problem-formulation failure: prior threat models do not correspond to nudification attacks, and the misalignment is structural, spanning mask shape, prompt distribution, and success criterion which causes existing evaluations to systematically overestimate protection effectiveness. We reformulate adversarial inpainting protection around this documented harm. We introduce BodyBench, a benchmark of 648 fully synthetic clothed adult subjects, each paired with 11 mask variants spanning body, face, context, and dilation regions, along with nudification and redressing attack prompts. We propose three harm-aligned metrics: sexualization, naturalness, and recognizability that jointly determine whether an inpainted output constitutes a successful nudification attack. Auditing four recent protection methods (PhotoGuard, DiffusionGuard, AdvPaint, DiffVax) on BodyBench, we find that protection succeeds in only one configuration of one method, and only under exact mask alignment between defender and attacker---a condition no real defender can guarantee and one that we show is easily always bypassed. We also show that standard metrics such as PSNR are essentially uncorrelated with nudification protection. BodyBench makes this misalignment measurable and provides a foundation for defenses grounded in documented harm.
Show more
STRABLE: Benchmarking Tabular Machine Learning with Strings
Gioia Blayer ⋅ Myung Jun Kim ⋅ Félix Lefebvre ⋅ Lennart Purucker ⋅ Alan Arazi ⋅ Eilam Shapira ⋅ Roi Reichart ⋅ Frank Hutter ⋅ Marine Le Morvan ⋅ David Holzmüller ⋅ Gael Varoquaux
Benchmarking tabular learning has revealed the benefit of dedicated architectures, pushing the state of the art. But real-world tables often contain string entries, beyond numbers, and these settings have been understudied due to a lack of a solid benchmarking suite. They lead to new research questions: Are dedicated learners needed, with end-to-end modeling of strings and numbers? Or does it suffice to encode strings as numbers, as with a categorical encoding? And if so, do the resulting tables resemble numerical tabular data, calling for the same learners? To enable these studies, we contribute STRABLE, a benchmarking corpus of 108 tables, all real-world learning problems with strings and numbers across diverse application fields. We run the first large-scale empirical study of tabular learning with strings, evaluating 445 pipelines. These pipelines span end-to-end architectures and modular pipelines, where strings are first encoded, then post-processed, and finally passed to a tabular learner. We find that, because most tables in the wild are categorical-dominant, advanced tabular learners paired with simple string embeddings achieve good predictions at low computational cost. On free-text-dominant tables, large LLM encoders become competitive. Their performance also appears sensitive to post-processing, with differences across LLM families. Finally, we show that STRABLE is a good set of tables to study “string tabular” learning as it leads to generalizable pipeline rankings that are close to the oracle rankings. We thus establish STRABLE as a foundation for research on tabular learning with strings, an important yet understudied area.
Show more
Evaluating Test-Time Scaling of General LLM Agents
Xiaochuan Li ⋅ Tianshi Ming ⋅ Pranav Setlur ⋅ Abhijay S Paladugu ⋅ Andy Tang ⋅ Hao Kang ⋅ Shuai Shao ⋅ Rong Jin ⋅ Chenyan Xiong
LLM agents are increasingly expected to operate as general-purpose systems that resolve real-world user requests, yet their dynamic scaling behavior in realistic environments remains poorly understood. In this paper, we systematically investigate two principal test-time scaling axes of LLM agents: sequential scaling through extended interaction and parallel scaling through trajectory sampling. We first introduce a realistic benchmark that provides one unified framework for evaluating LLM agents across search, coding, reasoning, and tool-use domains, more faithfully reflecting the heterogeneity of real-world deployments. Evaluating ten leading LLM agents reveals substantial performance degradation when transitioning from domain-specific evaluations to this realistic setting. Building on this foundation, we progressively scale test-time compute along fine-grained increments to characterize the performance upper bound. We find that neither scaling axis can consistently yield meaningful gains from additional test-time compute in realistic environments, a phenomenon we attribute to two fundamental limitations: the scaling plateau that bottlenecks sequential scaling and the verification gap that undermines parallel scaling. Code and data will be released publicly.
Show more
An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration
Maja Pavlovic ⋅ Silviu Paun ⋅ Massimo Poesio
Central to human-aligned AI is understanding the benefits of human-elicited labels over synthetic alternatives. While human soft-labels improve calibration by capturing uncertainty, prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. We present a controlled audit of soft-label learning across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. Dataset cartography reveals models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment.
Show more
Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration
Shiqi Zhang ⋅ Weixin Zeng ⋅ Ziheng Zhang ⋅ Jiuyang Tang ⋅ Xiang Zhao
Multimodal benchmarks are central to evaluating whether models can reason over information from multiple modalities. However, existing literature has revealed that some benchmarks may contain shortcut items that can be answered from a single modality or from answer-option regularities alone, thus failing to reflecting models' true cross-modal inference ability. Existing methods for assessing benchmarks typically rely either on intrinsic item structure or extrinsic model response behavior, which are not adequate for a comprehensive diagnosis. To fill in this gap, we propose SRCoD, a structure-response co-calibrated diagnosis framework for item-level diagnosis of modality dependence in multimodal benchmarks. SRCoD estimates sample-wise multimodal information structure from benchmark content and target answers, and uses this signal as a soft structural anchor for modality-conditioned item response modeling. The learned model produces a calibrated item-level diagnostic profile that reflects how strongly an item depends on different forms of modality evidence, enabling fine-grained benchmark analysis beyond aggregate accuracy. SRCoD also provides interpretable attribution of the diagnosis results. Extensive experiments on two complementary protocols, i.e., benchmark-refinement oriented validation and human-labeled item validation, demonstrate that SRCoD outperforms state-of-the-art benchmark diagnosis methods. Our code and data are available at https://anonymous.4open.science/r/SRCoD-61B0.
Show more
Bridging Academia and Industry: A Comprehensive Benchmark for Attributed Graph Clustering
Yunhui Liu ⋅ Pengyu Qiu ⋅ Yu Xing ⋅ Peng Du ⋅ Yongchao Liu ⋅ Chuntao Hong ⋅ Jiajun Zheng ⋅ Tao Zheng ⋅ Tieke He
Attributed Graph Clustering (AGC) is a fundamental unsupervised task that integrates structural topology and node attributes to uncover latent patterns in graph-structured data. Despite its significance in industrial applications such as fraud detection and user segmentation, a significant chasm persists between academic research and real-world deployment. Current evaluation protocols suffer from the small-scale, high-homophily citation datasets, non-scalable full-batch training paradigms, and a reliance on supervised metrics that fail to reflect performance in label-scarce environments. To bridge these gaps, we present `PyAGC`, a comprehensive, production-ready benchmark and library designed to stress-test AGC methods across diverse scales and structural properties. We unify existing methodologies into a modular Encode-Cluster-Optimize framework and, for the first time, provide memory-efficient, mini-batch implementations for a wide array of state-of-the-art AGC algorithms. Our benchmark curates 12 diverse datasets, ranging from $2.7 \times 10^3$ to $1.1 \times 10^8$ nodes, specifically incorporating industrial graphs with complex tabular features and low homophily. Furthermore, we advocate for a holistic evaluation protocol that mandates unsupervised structural metrics and efficiency profiling alongside traditional supervised metrics. Battle-tested in high-stakes industrial workflows at Ant Group, this benchmark offers the community a robust, reproducible, and scalable platform to advance AGC research towards realistic deployment. The code and resources are publicly available via [GitHub](https://github.com/Cloudy1225/PyAGC), [PyPI](https://pypi.org/project/pyagc), and [Docs](https://pyagc.readthedocs.io). We invite readers to our [latest survey](https://arxiv.org/abs/2603.20829) for the unified framework and industrial perspective that complement this benchmark.
Show more
TALES: Text-Adventure Learning Environment Suite
Christopher Z. Cui ⋅ Xingdi Yuan ⋅ Ziang Xiao ⋅ Prithviraj Ammanabrolu ⋅ Marc-Alexandre Côté
While Large Language Models (LLMs) are increasingly deployed as agents to complete complex real world tasks, assessing their baseline agentic capabilities remains challenging. Many existing benchmarks for LLM agents require the model to operate through a structured interface that sit one layer removed from natural language, making it difficult to distinguish failures of interface interpretation from genuine limitations in agentic capability. Because they rely entirely on grounded, natural-language interaction, text-adventure games serve as a prime sandbox for evaluating these capabilities. However, current implementations in popular benchmarks obscure performance through the leakage of privileged information or through scaffolding modifications to the tasks themselves. To solve this, we introduce \ours, a diverse collection of synthetic and human-written text-adventure games in their canonical forms, designed to evaluate baseline agentic competencies in progressively more challenging environments. We present results over a range of LLMs, open- and closed-weights; evaluate when and where current models fail as agents; and perform a qualitative analysis on top agents to identify key failure modes. Despite an impressive showing on synthetic games, even the top LLM-driven agents fail to achieve 25\% on games designed for human enjoyment. Visualization of the experiments can be found at https://github.com/tale-suite/tale-suite-anonymized.
Show more
ScAn-Bench: Evaluating Scaling Analysis Methodology
Artin Sermaxhaj ⋅ Nastaran Alipour ⋅ Donat Sinani ⋅ Johannes Hog ⋅ Neeratyoy Mallik ⋅ Steven Adriaensen ⋅ Jenia Jitsev ⋅ Danny Stoll
Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 2875 and 3853 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.
Show more
On Data Engineering for Scaling LLM Terminal Capabilities
Renjie Pi ⋅ Grace Lam ⋅ Mohammad Shoeybi ⋅ Pooya Jannaty ⋅ Bryan Catanzaro ⋅ Wei Ping
Despite rapid recent progress in the terminal capabilities of large language models, the training data strategies behind state-of-the-art terminal agents remain largely undisclosed. We address this gap through a systematic study of data engineering practices for terminal agents, making two key contributions: (1) Terminal-Task-Gen, a lightweight synthetic task generation pipeline that supports seed-based and skill-based task construction, and (2) a comprehensive analysis of data and training strategies, including filtering, curriculum learning, long context training, and scaling behavior. Our pipeline yields Terminal-Corpus, a large-scale open-source dataset for terminal tasks. Using this dataset, we train Nemotron-Terminal, a family of models initialized from Qwen3 (8B, 14B, 32B) that achieve substantial gains on Terminal-Bench 2.0: Nemotron-Terminal-8B improves from 2.5% to 13.0%, Nemotron-Terminal-14B improves from 4.0% to 20.2%, and Nemotron-Terminal-32B improves from 3.4% to 27.4%, matching the performance of significantly larger models. We release Nemotron-Terminal checkpoints and Terminal-Corpus through the Nemotron-Terminal Hugging Face collection.
Show more
MVVBench: Benchmarking 4D Reasoning in Vision-Language Models
Hyungjin Chung ⋅ Byeongjun Park ⋅ Joonseok Lee ⋅ Hojun Kim ⋅ Jaeho Choi ⋅ Byung-Hoon Kim
Multi-view video understanding requires integrating spatial and temporal evidence across multiple, often non-overlapping camera streams—tracking entities as they transition between viewpoints, aligning events across time, and reasoning about latent 4D continuity rather than any single visible frame. We introduce MVVBench, a benchmark for multi-view video reasoning built from real-world multi-camera datasets and curated to be monocular-ambiguous: each question is unanswerable from any single view alone but becomes uniquely solvable by jointly reasoning over multiple views. MVVBench spans diverse dynamic scenes and probes six capabilities: implicit/explicit attribute identification, implicit/explicit relative distance, relative camera pose, and compositional counting, with human-authored QA and rigorous verification. Beyond benchmarking, we provide an extensive analysis of when and why current vision-language models succeed or fail, characterizing errors due to temporal mis-localization, cross-view identity breaks, and brittle multi-hop reasoning. We then study inference-time elicitation strategies that unlock latent multi-view competence—task-specific chain-of-thought scaffolds and structured cross-view evidence aggregation—yielding substantial gains without retraining. Finally, we present preliminary evidence that reinforcement learning with verifiable rewards can elicit some latent multi-view competence in the base model, pointing to training-time approaches as a promising direction for future work. Together, MVVBench offers a rigorous evaluation of 4D multi-view reasoning and a foundation for future progress toward reliable embodied perception.
Show more
SynthHair: Leveraging MetaHumans for a High-Quality 4K Hair Matting Dataset
Markus Karmann ⋅ Shile Li ⋅ Philip Torr ⋅ Puneet Dokania ⋅ Qi Zhang ⋅ Peng-Tao Jiang ⋅ Bo Li ⋅ Onay Urfalioglu
Recent progress in computer vision enables dense alpha matte predictions at resolutions of 4K and higher. While portrait matting has benefited from large-scale, high-resolution datasets, progress in hair matting remains limited due to the difficulty of annotating sub-pixel fine hair details. In this work, we analyze critical limitations in image resolution and annotation quality of such existing datasets and introduce our own synthetic hair matting dataset, \textbf{SynthHair}, which consists of 16k RGBA portrait images in $4096 \times 4096$ image resolution, containing labels for hair matting, as well as hair categorization, growth direction, and depth estimation. We create this dataset using MetaHuman Creator and Unreal Engine to obtain highly detailed and realistic renders of human faces with complex hair structures. We adapt a generalized version of the isoperimetric inequality quotient (IPQ) that we call the \textbf{soft isoperimetric inequality quotient (SIPQ)}, as a scale-invariant label complexity metric that enables comparison of both segmentation and matte labels. Using this metric, we demonstrate that hair matting complexity increases significantly with resolution, reaching an SIPQ of 325.41 at 4K, in contrast with portrait matting datasets at similar resolutions, which reach an SIPQ of ~10. To validate our data, we train separate matting models on SynthHair and existing real-world datasets. Our results demonstrate strong domain generalization and show clear improvements in high-resolution images. A/B testing further confirms these findings, with over 80\% of users preferring our model's results on high-resolution images.
Show more
IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams
Jinzhao Li ⋅ Yinuo Chen ⋅ Wenxuan Song ⋅ Yijia Lei ⋅ Yichi Zhang ⋅ Honglei Yan ⋅ Panwang Pan ⋅ Miao Liu
Recent multimodal large language models (MLLMs) achieve strong performance on reactive question answering, but real-world streaming assistants require proactive reasoning over continuous visual inputs. Existing benchmarks mainly study reactive or proactive interactions in isolated single-turn settings, overlooking dynamic multi-turn scenarios where users may add, modify, or cancel proactive requests alongside interleaved reactive queries. To address this gap, we introduce IPIBench, the first benchmark for evaluating Interactive Proactive Intelligence of MLLMs under streaming video settings. IPIBench covers proactive monitoring, proactive task management, and interleaved reactive–proactive requests. Evaluations on representative MLLMs reveal two major limitations: unstable proactive triggering and weak coordination between reactive and proactive behaviors. We further propose IPI-Agent, a training-free agentic framework with an interaction-control policy and a temporal-gating mechanism for stabilizing proactive triggering and coordinating multi-turn interactions. Experiments show that IPI-Agent consistently improves existing MLLMs across all benchmark settings.
Show more
EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance
Thomson Yen ⋅ Julian Poeltl ⋅ Harshith S Gear ⋅ Yilin Meng ⋅ Joshua Fan ⋅ Adam Shen ⋅ Yili Liu ⋅ Ali Bauyrzhan ⋅ Siri Du ⋅ Haoyang Liu ⋅ C. Guetta ⋅ Hongseok Namkoong
LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions. To meet enterprise needs, frontier AI labs have developed agents that can construct entire spreadsheets from scratch. This capability is especially relevant in finance, where core workflows such as financial modeling, forecasting, and scenario analysis are commonly conducted through spreadsheets. Yet, existing spreadsheet benchmarks do not measure this advanced capability, focusing instead on question-answering or single-formula edits. To address this gap, we provide one of the first evaluations of agents on end-to-end spreadsheet tasks, focusing on economically critical financial workflows such as modeling and scenario analysis. Since deliverables therein are routinely reviewed and revised by multiple stakeholders, judging their quality necessarily involves high-level criteria such as readability or ease of modification. To reflect the multidimensional nature of solution quality, we develop an evaluation taxonomy comprising three dimensions: Accuracy, Formula, and Format, each comprising fine-grained criteria that reflect professional standards. The Claude family leads the benchmark and produces the most professional-looking outputs in our qualitative review, but even the strongest agents frequently fall short of professional finance standards and degrade sharply as the difficulty increases beyond a few chained calculations. This suggests that current agents are not yet able to reliably produce professional-quality spreadsheets at the level of complexity real-world workflows demand.
Show more
MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
Arushi Goel ⋅ Sreyan Ghosh ⋅ Vatsal Agarwal ⋅ Nishit Anand ⋅ Kaousheik Jayakumar ⋅ Lasha Koroshinadze ⋅ Yao Xu ⋅ Katie Lyons ⋅ James Case ⋅ Siddharth Gururani ⋅ Karan Sapra ⋅ Kevin Shih ⋅ Abhinav Shrivastava ⋅ Ramani Duraiswami ⋅ Dinesh Manocha ⋅ Andrew Tao ⋅ Bryan Catanzaro ⋅ Mohammad Shoeybi ⋅ Wei Ping
Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However, their ability to jointly reason over omni-modal (visual, audio, and textual) signals in long and complex videos remains largely unexplored. We introduce MMOU, a new benchmark designed to systematically evaluate multimodal understanding and reasoning under these challenging, real-world conditions. MMOU consists of 16,500 carefully curated questions paired with 9800 web-collected videos of varying length, spanning diverse domains and exhibiting rich, tightly coupled audio-visual content. The benchmark covers 13 fundamental skill categories, all of which require integrating evidence across modalities and time. All questions are manually annotated across multiple turns by professional annotators, ensuring high quality and reasoning fidelity. We evaluate 20+ state-of-the-art open-source and proprietary multimodal models on MMOU. The results expose substantial performance gaps: the best closed-source model achieves only 64.2% accuracy, while the strongest open-source model reaches just 46.8%. Our results highlight the challenges of long-form omni-modal understanding, revealing that current models frequently fail to apply even fundamental skills in long videos. Through detailed analysis, we further identify systematic failure modes and provide insights into where and why current models break. Project: https://mmou-2026.github.io/
Show more
Can We Trust Item Response Theory for AI Evaluation?
Han Jiang ⋅ Sunbeom Kwon ⋅ Jinwen Luo ⋅ Ziang Xiao ⋅ Susu Zhang
AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regimes encountered in human testing, for which standard IRT estimation tools were originally developed: these benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimators used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation runs, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become computationally infeasible for large benchmarks, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or non-normally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.
Show more
AgenticVBench: Can AI Agents Complete Real-World Video Production Tasks?
Zongheng Cao ⋅ Yi Zheng ⋅ Xinyu Hu
Video production workflows offer a rich and demanding testbed for evaluating multimodal AI agents: they require composite capabilities across text, image, audio, and video understanding, along with long-horizon planning, and tool use. To this end, we introduce AgenticVBench, a benchmark of 100 tasks across 4 task families spanning the real world post-production workflow, constructed from the representative work of 20 industry experts averaging 6 years of experience across video production contexts. Tasks are paired with evaluation specifications that combine programmatic verifiers and expert rubrics, both authored under expert protocol. We evaluate frontier vision-language models (VLMs) with vendor-native and open-source harnesses. The best evaluated model achieves a score below 30\%, substantially lower than frontier performance on existing multimodal benchmarks. We further find that the choice of harness substantially affects model behavior, including but not limited to scores and failure modes. AgenticVBench provides a foundation for understanding and improving both models and harnesses for agentic video production.
Show more
SituRecBench: Evaluating Frontier Models as Situated Recommendation Assistants in 3D Environments
Jingtong Yang ⋅ Dongding LIN ⋅ Jian Wang ⋅ Wenjie Li
Recommender systems are transitioning from static retrieval engines into interactive assistants grounded in physical environments. However, current paradigms largely treat recommendation as a decoupled ranking task, neglecting the situated complexities of spatial navigation, visual grounding, and multi-turn social interaction. To bridge this gap, we introduce **SituRecBench**, a novel interactive benchmark designed to evaluate situated recommendation agents within high-fidelity 3D environments. SituRecBench reformulates the recommendation process as a closed-loop interaction between a recommendation assistant and a user with evolving latent preferences and realistic behaviors. Our benchmark features diverse real-world shopping scenarios (e.g., supermarkets and home furnishings stores), requiring assistants to integrate preference elicitation, spatial reasoning, and long-horizon conversation. We introduce a simulator-in-the-loop evaluation protocol along with a multi-dimensional metric taxonomy that quantifies communication fluency, recommendation utility, and navigation effectiveness. Through extensive evaluation of state-of-the-art vision-language models, we reveal critical challenges in situated recommendation, such as inefficient preference elicitation, coherent conversation transition, and poor spatial grounding. SituRecBench provides a rigorous, reproducible foundation for advancing the frontier of situated recommendation in interactive, visually-grounded environments.
Show more
VistaQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence
Mozhgan Nasr Azadani ⋅ Yimu Wang ⋅ Yongpeng Zhu ⋅ Lihong Chen ⋅ Milan Ganai ⋅ Sean Sedwards ⋅ Marco Pavone ⋅ Krzysztof Czarnecki
Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimodal reasoning, yet current multimodal large language model (MLLM) evaluations do not explicitly enforce this alignment. Existing benchmarks assess either textual answer correctness or pixel-level localization in isolation, leaving the coupling of reasoning and grounding an open challenge. We introduce VistaQA, a comprehensive benchmark for joint evaluation of free-form answer correctness and pixel-level evidence grounding in visual question answering. VistaQA comprises 1,157 expert-curated samples spanning six task types and six visual domains, ranging from direct perception to compositional and relational reasoning. VistaQA requires models to not only answer correctly, but to also provide precise segmentation masks that support their answers. It also includes hallucination-aware examples where no valid visual evidence exists. To support this enhanced evaluation, we introduce GROVE, a unified evaluation metric that enforces joint correctness by combining textual accuracy and grounding quality via a per-sample geometric mean, ensuring neither dimension can compensate for deficiencies in the other. Comprehensive experiments across grounding-aware models and hybrid pipelines with general-purpose MLLMs reveal that even the strongest systems achieve limited performance under GROVE, highlighting a substantial gap between answer accuracy and visual evidence alignment.
Show more
RAM-H1200: A Unified Evaluation and Dataset on Hand Radiographs for Rheumatoid Arthritis
Songxiao Yang ⋅ Haolin Wang ⋅ Yao Fu ⋅ Junmu Peng ⋅ Lin Fan ⋅ Hongruixuan Chen ⋅ JIAN SONG ⋅ Masayuki Ikebe ⋅ Shinya Takamaeda-Yamazaki ⋅ Masatoshi Okutomi ⋅ Tamostu Kamishima ⋅ yafei ou
Rheumatoid arthritis (RA) assessment from hand radiographs requires multi-level analysis and modeling of anatomical structures and fine-grained local pathological changes. However, existing public resources do not support such multi-level analysis, often lacking full-hand coverage, fine-grained annotations, and consistent integration with clinical scoring systems. In particular, annotations that enable quantitative analysis of bone erosion (BE) remain scarce. RAM-H1200 contains 1,200 hand radiographs from a regional, multi-institutional Japanese cohort collected across six medical centers in the Sapporo region, with multi-level annotations including (i) whole-hand bone structure instance segmentation, (ii) pixel-level BE masks, (iii) Sharp/van der Heijde (SvdH)-defined joint regions of interest, and (iv) joint-level SvdH scores for both BE and joint space narrowing (JSN). It provides a common evaluation resource covering anatomical structure modeling, localized erosive pathology analysis, and clinically standardized RA severity assessment from the same radiographs. The proposed BE masks enable, for the first time, quantitative BE analysis beyond coarse categorical grading by providing explicit spatial supervision for lesion extent and morphology. To our knowledge, RAM-H1200 is the first public benchmark resource providing whole-hand bone structure instance segmentation, pixel-level BE delineation, and clinically grounded joint-level SvdH scoring for both BE and JSN. Results across benchmark tasks show that anatomical modeling is substantially more mature than quantitative BE analysis: whole-hand bone segmentation achieves strong performance, whereas BE segmentation remains a major open challenge. By providing multi-level annotations spanning anatomical structure modeling, quantitative lesion analysis, and clinically grounded SvdH scoring, RAM-H1200 provides a comprehensive benchmark resource for RA analysis on hand radiographs. Benchmark \& Code: https://github.com/YSongxiao/RAM-H1200 Dataset Repository: https://huggingface.co/datasets/TokyoTechMagicYang/RAM-H1200-v1
Show more
Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance
Ziyun Zeng ⋅ Yiqi Lin ⋅ Guoqiang Liang ⋅ Mike Zheng Shou
In recent years, open-source efforts like Senorita-2M have propelled video editing toward natural language instruction. However, current publicly available datasets predominantly focus on local editing or style transfer, which largely preserve the original scene structure and are easier to scale. In contrast, *Background Replacement*, a task central to creative applications such as film production and advertising, requires synthesizing entirely new, temporally consistent scenes while maintaining accurate foreground-background interactions, making large-scale data generation significantly more challenging. Consequently, this complex task remains largely underexplored due to a scarcity of high-quality training data. This gap is evident in poorly performing state-of-the-art models, *e.g.,* Kiwi-Edit, because the primary open-source dataset that contains this task, *i.e.,* OpenVE-3M, frequently produces static, unnatural backgrounds. In this paper, we trace this quality degradation to *a lack of precise background guidance* during data synthesis. Accordingly, we design a scalable pipeline that generates foreground and background guidance in a decoupled manner with strict quality filtering. Building on this pipeline, we introduce *Sparkle*, a dataset of ~140K video pairs spanning five common background-change themes, alongside *Sparkle-Bench*, the largest evaluation benchmark tailored for background replacement to date. Experiments demonstrate that our dataset and the model trained on it achieve substantially better performance than all existing baselines on both OpenVE-Bench and Sparkle-Bench. Our proposed dataset, benchmark, and model are fully open-sourced at https://showlab.github.io/Sparkle/.
Show more
OverLay++: Dense-Overlap Layout-to-Image Generation Dataset
Shivansh Aggarwal ⋅ Shresth Grover ⋅ Divyansh Srivastava ⋅ Haiyang Xu ⋅ Bingnan Li ⋅ Xiang Zhang ⋅ Ethan Armand ⋅ Chuan Li ⋅ Jianwen Xie ⋅ Zhuowen Tu
Layout-to-Image generation has made substantial progress in image generation with spatial and object-level control. However, existing methods still struggle with complex scenes containing many overlapping and interacting objects. We argue that training data is a particular bottleneck: existing datasets lack examples with dense, complex, object interactions. To address this gap, we introduce **OverLay++**, a large-scale Layout-to-Image dataset with structurally complex scenes. OverLay++ contains approximately 500K images with an average of 6.6 objects per image, exceeding existing datasets by $1.67\times$ in annotation density. Beyond annotation density, OverLay++ provides rich semantic detail with object captions over six times longer than current datasets. Our dataset generation pipeline is simple and robust, producing accurate overlapping regions with rich per-object captions. Across multiple benchmarks, state-of-the-art Layout-to-Image methods trained on OverLay++ dataset show consistent improvement and faster convergence demonstrating the importance of dense, overlap-aware, and caption-rich supervision for controllable image generation.
Show more
Towards Characterizing Scientific Image Utility and Upgradability
Wenzhe Li ⋅ Qihang Yan ⋅ Liang Chen ⋅ Junying Wang ⋅ Farong Wen ⋅ Yijin Guo ⋅ Chunyi Li ⋅ Zicheng Zhang ⋅ Guangtao Zhai
Scientific images function as critical evidence in research communication, yet their integrity faces unprecedented threats from AI-generated content that introduces subtle but consequential errors. Existing evaluation paradigms prove inadequate: image quality assessment poorly correlate with scientific validity, while language models lack domain-specific verification capabilities. To address this gap, we propose the $\textbf{S}$cientific $\textbf{I}$mage $\textbf{U}$tility and $\textbf{U}$pgradability $\textbf{A}$ssessment ($\textbf{SIU$^2$A}$) framework, which introduces two complementary dimensions for scientific image evaluation. $\textbf{Utility}$ encompasses $\textit{error detection}$ (identifying scientific inaccuracies) and $\textit{correction feasibility}$ (assessing whether errors can be reliably repaired). $\textbf{Upgradability}$ measures the quality of correction We categorize scientific image corruption into four fundamental types: Detail Distortion, Incompleteness, False Content, and Entity Confusion. Based on this taxonomy, we construct SIU$^2$A-Benchmark, a comprehensive dataset featuring expert annotations for both error identification and repair. The framework implements a unified two-stage evaluation protocol: the first stage evaluates error detection and correction instruction generation (Utility), while the second stage assesses the effectiveness of actual corrections (Upgradability). Experiments reveal that current multimodal systems exhibit significant limitations in both scientific error assessment and faithful correction, exposing a fundamental gap between visual perception and scientific usability. Our findings establish that diagnostic quality fundamentally constrains restoration performance, highlighting the critical need for robust error perception mechanisms in scientific multimodal AI.
Show more
Rethinking Dataset Distillation for Classification: Do Distilled Sets Outperform Coresets?
Trisha Mittal ⋅ Akshay Mehra ⋅ Joshua Kimball
Dataset distillation (DD) has emerged as a prominent approach in data centric machine learning, aiming to synthesize compact training sets for efficient training by compressing the information in large datasets into a small number of synthetic samples. However, DD methods are often evaluated under inconsistent evaluation protocols, ranging from standard ERM to single/multi‑teacher supervision, making it difficult to isolate the effectiveness of distilled data from evaluation. Moreover, many prior methods claim that DD outperforms data pruning approaches such as coreset selection (CS), based on the assumption that restricting condensed datasets to subsets of real samples fundamentally limits their expressiveness. In this work, we critically evaluate DD methods through large-scale experiments using standardized datasets and evaluation protocols to assess their intrinsic effectiveness. We benchmark seven state-of-the-art DD methods on ImageNet-1K, ImageNet-100, and ImageNette, using three widely adopted training protocols against three CS strategies. Our results show that while some DD methods fail to outperform even simple random subsets, the SOTA DD approaches are comparable to or worse than coresets on large‑scale datasets and incur a substantially higher cost for construction. Beyond accuracy, we also evaluate the representativeness, diversity, and quality of condensed sets, and find that coresets consistently achieve better coverage of the original data distribution. These findings highlight the limited practical advantages of current DD methods and show that coresets remain competitive and are often a more computationally efficient alternative for data-centric learning.
Show more
VTBench: Disentangled and Human-Aligned Evaluation for Image-Based Virtual Try-on
Donghao Luo ⋅ Yujie Liang ⋅ Caoshuo Li ⋅ Xiaobin Hu ⋅ Zuxuan Wu ⋅ Yanwei Fu
While virtual try-on has achieved significant progress, evaluating these models towards real-world scenarios remains a challenge: unpaired metrics (FID/KID) miss task-specific dimensions such as texture and consistency, and most test sets are limited to indoor scenarios. To address these needs, we introduce the Virtual Try-on Benchmark (VTBench), the first-ever hierarchical try-on benchmark suite that systematically decomposes virtual image try-on into hierarchical, disentangled dimensions, each equipped with tailored test sets and evaluation criteria. VTBench exhibits three key advantages: 1) Multi-Dimensional Evaluation Framework: The benchmark defines three top-level dimensions (General Image Quality, Garment Preservation, Auxiliary Consistency) and six fine-grained dimensions (Distributional Fidelity, Aesthetics, Texture Fidelity, Cross-Category Plausibility, Background Consistency, and Hand Consistency). Granular evaluation metrics and targeted test sets pinpoint model capabilities and limitations across diverse, challenging scenarios. 2) Human Alignment: Human preference annotations are provided for the five image-level dimensions, ensuring the benchmark's alignment with perceptual quality. 3) Valuable Insights: Beyond standard indoor settings, we analyze model performance variations across dimensions and investigate the disparity between indoor and real-world try-on scenarios. To foster the field of virtual try-on towards challenging real-world scenarios, VTBench will be open-sourced, including all datasets and evaluation algorithms.
Show more
LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis
Haoxiang Cao ⋅ Yuxuan Zhang ⋅ Penghui Du ⋅ Bo Li ⋅ Jiajiong Cao ⋅ Yingying Fan ⋅ Zimeng Wu ⋅ Yifu Luo ⋅ Mingzhe Zheng ⋅ Junwen Miao ⋅ Huaisong Zhang ⋅ YANG YAO ⋅ Yu Wang ⋅ Dongfu Jiang ⋅ Ping Nie ⋅ Wenhu Chen ⋅ Changqian Yu ⋅ Kelsey Allen ⋅ Chaoqun Wang
We introduce LongBanana, to our knowledge the \textbf{first expert-designed and expert-verified benchmark} for \emph{long-context multi-reference image synthesis} in the age of \textbf{agent-controlled image generation}. The task requires retrieving dispersed visual evidence, rejecting distractors, binding entities and relations, and composing one coherent final image from multiple references. LongBanana contains 90 carefully designed tasks organized by a 3-by-3 taxonomy; each task includes about 32 reference images, 20 constraints, and 50 evaluation items, yielding \textbf{4{,}670 expert-specified atomic evaluation items} in total. Its graph-grounded evaluation design instantiates each task with reference images and an entity-relation graph, then converts node, edge, and whole-scene obligations into checklist items for Content Fidelity, Relational Binding, and Global Constraint Satisfaction. Reviewed expert labels include a 15\% double-labeled subset with \textbf{89.4\% exact agreement} and \textbf{0.83 quadratic weighted kappa}. We further introduce LongBanana-Gen, an agentic controller framework for long-horizon image generation; experiments show that minimal agent harnesses improve over direct generation, while ReAct/Plan-Exec/Reflect yields controller-dependent gains by strengthening evidence grounding and repair. Together, LongBanana provides an auditable testbed for measuring dense visual evidence use and a shared target for developing stronger agent-controlled image-generation systems.
Show more
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Xin Li ⋅ Mengbing LIU ⋅ Chau Yuen
Multi-agent large language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On $6{,}925$ MMLU-Pro debates, the protocol identifies $253$ collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents $29$ collapses but loses $108$ corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate $8$-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high ($G{=}7$, Spearman $\rho=0.893$, exact two-sided $p=0.0123$), but initial-majority accuracy is a close comparator ($\rho=0.821$; family partial $\rho=0.767$, $p=0.0877$), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model--scaffold rows can be compared under the same denominators and signed utility ledger.
Show more
PICID: A Modular Evaluation Infrastructure for Reproducible PHM Across Tasks and Domains
Lev Telyatnikov ⋅ Raffael Theiler ⋅ Leandro Von Krannichfeldt ⋅ Olga Fink
Progress in Prognostics and Health Management (PHM) is hindered by the lack of standardized and reusable evaluation practices across tasks, datasets, and application domains. Reported results are often difficult to reproduce and compare, as key protocol choices, such as data splits, preprocessing, label alignment, temporal windowing, and metrics, are often implicit or implemented ad hoc. We introduce \picid, a modular evaluation infrastructure that formalizes the PHM evaluation pipeline as an explicit, executable, and reproducible protocol. Through well-defined abstractions, \picid enforces deterministic, leakage-safe dataset construction while remaining flexible across diverse PHM settings. The framework supports fault detection, diagnostics, and prognostics through a unified interface and can be extended to new datasets and model classes without violating protocol invariants. By standardizing data contracts and evaluation boundaries, \picid also enables fair cross-task comparisons across diagnostics (classification) and prognostics (regression), allowing identical model families to be evaluated consistently across heterogeneous settings. We demonstrate \picid through an empirical evaluation of thirteen models on twelve datasets spanning batteries, bearings, turbofan engines, hydraulics, filtration systems, and buildings. This work establishes a reusable foundation for standardized, fair and reproducible evaluation in PHM.
Show more
CausalDriveBench: Evaluating Causal Reasoning in Vision-Language-Action Models for Autonomous Driving
Narendiran Chembu ⋅ Navvrat Rao ⋅ Shreedhar Kodate ⋅ Gayatri S Banda ⋅ Arko Sarkar ⋅ Abhinav Khanna ⋅ Rajarshee Das ⋅ Umesh Kanala ⋅ Siddharth Khandelwal ⋅ Kumar Aman ⋅ Aish Dubey ⋅ Kaustubh Beedkar ⋅ Arjun Jain
Vision-Language-Action (VLA) models for autonomous driving produce natural-language reasoning alongside predicted trajectories, but whether this reasoning reflects the causal structure of the scene remains untested. We introduce CausalDriveBench, an evaluation framework grounded in Pearl's Causal Hierarchy (PCH) that tests causal reasoning in driving-specific VLAs through structured visual question answering (QA) and alternative-trajectory prediction. To this end, we construct causal scene graphs over nuScenes that distinguish causally active, dormant, and distractor entities, separating perceptual salience from causal relevance. The benchmark spans all four rungs of PCH (association, intervention, and counterfactual along with causal discovery) for QA generation. For the higher rungs, we additionally provide reference trajectories under specified scene modifications, enabling action-level verification that complements reasoning-level evaluation. In total, the benchmark contains 7,285 verified causal QA pairs and 1,000 counterfactual trajectories derived from nuScenes. We evaluate 10 driving-specific VLAs and 3 general-purpose VLMs, and report three findings. First, the best model reaches only 70.6\% QA accuracy, and 4 of 13 models score below random chance. Second, comparing each driving VLA to the general-purpose VLM that shares its language backbone, the cost of driving fine-tuning ranges from 2 to 34 percentage points on causal QA, with post-training design explaining the spread. Third, causal QA and trajectory accuracy are statistically uncorrelated across models: under counterfactual prompts, predicted trajectories either over-react or collapse onto the observed-scene baseline. Taken together, these results show that neither fluent rationales nor accurate observed-scene trajectories constitute evidence of causal understanding.
Show more
RedVLA: Physical Red Teaming for Vision-Language-Action Models
Yuhao Zhang ⋅ Borong Zhang ⋅ Jiaming Fan ⋅ Jiachen Shen ⋅ Yishuai Cai ⋅ Yaodong Yang ⋅ Jiaming Ji
The real-world deployment of Vision-Language-Action (VLA) models remains limited by the risk of unpredictable and irreversible physical harm. However, we currently lack effective mechanisms to proactively detect these physical safety risks before deployment. To address this gap, we propose \textbf{RedVLA}, the first red teaming framework for physical safety in VLA models. We systematically uncover unsafe behaviors through a two-stage process: (I) \textbf{Risk Scenario Synthesis} constructs a valid and task-feasible initial risk scene. Specifically, it identifies critical interaction regions from benign trajectories and positions the risk factor within these regions, aiming to entangle it with the VLA's execution flow and elicit a target unsafe behavior. (II) \textbf{Risk Amplification} ensures stable elicitation across heterogeneous models. It iteratively refines the risk factor state through gradient-free optimization guided by trajectory features. Experiments on six representative VLA models show that RedVLA uncovers diverse unsafe behaviors and achieves the ASR up to 95.5\% within 10 optimization iterations. To mitigate these risks, we further propose SimpleVLA-Guard, a lightweight safety guard built from RedVLA-generated data. Our assets and code are available at https://anonymous.4open.science/r/redvla.
Show more
OpenSanctions Pairs: A Large-Scale Dataset for Pairwise Entity Matching
Chandler Smith ⋅ Magnus Sesodia ⋅ Friedrich Lindenberg ⋅ Christian Schroeder de Witt
We release OpenSanctions Pairs, the first large-scale public benchmark for entity matching on sanctions and OSINT data. The dataset includes 755,540 expert-labeled pairs over 1 million entities, aggregated from 293 source datasets across 45 jurisdictions. It captures real-world diversity in compliance data, spanning multiple languages and writing systems (e.g., Latin, Cyrillic, Arabic), inconsistent structure, and time-varying provenance, and is substantially more heterogeneous than prior entity matching benchmarks. As baselines, we evaluate the production rule-based matcher (nomenklatura RegressionV1) alongside open- and closed-source LLMs in both zero- and few-shot settings, each tested with and without MIPROv2 prompt optimization to control for prompt sensitivity. The rule-based baseline reaches 91.3\% F1; GPT-4o achieves the best result at 99.0\% F1, and a locally deployable open-source model (DeepSeek-R1-Distill-Qwen-14B) achieves 98.2\% F1. The rule-based baseline and LLMs fail in complementary ways: rules over-match, while LLMs struggle with cross-script transliteration. These results suggest that pairwise matching performance is approaching a practical ceiling and shift attention toward pipeline components such as blocking, clustering, and uncertainty-aware review.
Show more
InformedXRD: Reproducible Benchmarks and Physics-Informed Evaluation for Powder Diffraction Symmetry Classification
Edward G Friedman ⋅ Elizabeth J Baggett ⋅ Abhishek Shetty ⋅ Derrick Chan-Sew ⋅ Vanellsa Acha ⋅ Harshitha Dwaracherla ⋅ P. A Kienzle ⋅ William Ratcliff
Machine learning for powder X-ray diffraction (PXRD) symmetry classification is being trained at scale, yet the field lacks a common evaluation standard: papers report results on different subsets of the RRUFF mineral diffraction database, under different target taxonomies, with different preprocessing and different summary metrics, and published models are rarely released in a form that permits rescoring. We present InformedXRD, an evaluation framework with four parts: (i) a machine-readable mapping from 230 space groups to 99 extinction groups, the finest classes that powder diffraction can identify from systematic absences alone; (ii) two algorithmically curated real-data benchmarks, RRUFF-473 and RRUFF-325, whose inclusion rules are specified as code and reproduce from a frozen upstream snapshot; (iii) a prior-only frequency baseline and stratified reporting protocol that expose when apparent gains are driven by label imbalance or nuisance-fit severity; and (iv) a hierarchy-aware error metric on the condensed translationengleiche (same-lattice) subgroup graph that measures the crystallographic severity of misclassifications. Using four publicly released ViT checkpoints from Baggett et al. (2026) and an in-house 1D residual CNN as case studies, we show that evaluation design changes model ranking: standard paired tests point in different directions depending on whether one evaluates Top-1 or Top-5, and stratified analysis exposes a label-entropy confound that aggregate accuracy hides. All mapping tables, benchmark code, evaluation scripts, and case-study checkpoints are released as shared infrastructure for future PXRD-ML evaluation.
Show more
DiscoverPhysics: Benchmarking LLMs for out-of-the-box scientific thinking
Lindsay Smith ⋅ Matt Sampson ⋅ Siddharth Mishra-Sharma ⋅ Peter Melchior ⋅ Andrew Wilson ⋅ Pavel Izmailov ⋅ Carolina Cuesta Lazaro
Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of established science. We introduce _DiscoverPhysics_, an interactive benchmark that asks a LLM agent to discover the laws of motion of a simulated world whose physics deliberately deviates from our own. We construct eleven worlds governed by, among others, screened and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, and time-varying interactions. Each world is generated on demand by an N-body simulator, for which the agent proposes several rounds of experiments, observes raw trajectory data, and ultimately submits both a natural-language explanation of the world's physics and a Python implementation of the inferred law. Because solving a world requires the agent to design informative experiments and revise its hypotheses, the benchmark probes long-horizon reasoning over an experimental history. We evaluate submissions along two complementary axes: trajectory MSE on held-out particles and an LLM-judged explanation score following an expert-written rubric assessing conceptual understanding of each world. Across ten frontier models, we find that the strongest agents pass only about half of the worlds and consistently fail on those where latent structure must be uncovered. Open-source models lag substantially behind commercial models, both in their ability to design informative experiments and in extracting conclusions from the data. We further find that good predictive accuracy does not guarantee high explanation quality and that conceptual understanding depends on hypothesis refinement through well-chosen experiments.
Show more
ALETHEIA: A Multi-Frequency Eddy Current Pulsed Thermography Dataset for Neural Operator Learning in Nondestructive Testing
Changbin Sun ⋅ jie Xiao ⋅ Xiaotian Chen ⋅ Yuankai Wu
Learning neural solvers for spatiotemporal partial differential equations (PDEs) under real-world constraints remains a key challenge in scientific machine learning, especially for inverse tasks with sparse and noisy boundary observations. We present the Aletheia dataset, the first 3D benchmark for learning data-driven solvers in the context of nondestructive testing (NDT). The dataset simulates eddy-current-induced heating in conductive solids and models the resulting transient heat propagation governed by the heat equation. Aletheia contains over 4,700 high-resolution samples across 10 excitation frequencies (1-100\,kHz), each providing volumetric heat source and temperature fields over time. It supports both forward prediction of temperature evolution and inverse reconstruction of internal heat sources or defects from surface infrared measurements. Real infrared thermography data from cracked rail specimens are included for calibration and generalization studies. We define three canonical tasks on both regular and irregular grids and benchmark them using various neural operators. Aletheia establishes a unified platform for evaluating neural PDE solvers under realistic NDT conditions, enabling progress in reliable, data-driven inverse modeling.
Show more
A systematic evaluation of vision-language models for observational astronomical reasoning tasks
Wenke Ren ⋅ Hengxiao Guo ⋅ Wenwen Zuo ⋅ Xiaoman Zhang
Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a benchmark of over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. We systematically evaluate six state-of-the-art frontier models and find that performance is strongly modality-dependent and uncorrelated with general benchmark rankings. Mechanistic ablations reveal three concrete bottlenecks. First, prompts that ground attention in physical mechanisms (why a feature matters) yield more balanced classifications than phenomenological prompts (what to look for). Second, presenting one-dimensional measurements as numerical tables rather than rendered plots improves accuracy by up to 13 percentage points, indicating that plot rendering can obscure physically relevant signal. Third, a qualitative reasoning analysis reveals a pervasive “right-answer-wrong-reason” phenomenon: models routinely reach correct predictions through physically invalid logic, showing that accuracy alone is insufficient for trustworthy scientific deployment. AstroVLBench provides the first systematic, multi-modal baselines for VLMs in observational astronomy and identifies the representation, grounding, and reasoning bottlenecks that must be addressed before VLMs can be trusted as autonomous scientific interpreters.
Show more
SwitchLingua V2: Agent-Driven Code-Switching via Digital Clones
Peng Xie ⋅ Yequan Bie ⋅ Jianda Mao ⋅ Hao CHEN ⋅ Yangqiu Song ⋅ Kani Chen
Code-switching (CS) involves shifting between two or more languages during a conversation or utterance, a phenomenon often driven by social context and the identity of the speaker. The scarcity of authentic CS data remains a critical bottleneck in developing robust Automatic Speech Recognition (ASR) systems for bilingual communities. While Large Language Models (LLMs) have been employed to synthesize CS data, direct generation approaches typically produce utterances that are pragmatically unnatural and fail to capture the nuanced stylistic variations of real-world bilingual speakers. In this paper, we propose an archetype-driven code-switching dialogue generation framework via agent-based digital cloning. Moving beyond static prompting, our method facilitates autonomous conversations between multi-agent digital clones imbued with specific bilingual archetypes and established linguistic theories. The generation pipeline integrates constrained persona sampling, real-world topic injection, turn-by-turn accommodation dynamics, and multi-dimensional language checking. Applying this framework, we synthesize highly idiomatic bilingual dialogues across 14 diverse language pairs with 260K textual dialogues (1.04M turns of text samples) and convert them into over 1,558 hours of high-fidelity audio data, yielding the SwitchLingua V2 dataset. To rigorously validate our approach and the generated data, we employ a comprehensive evaluation protocol including both advanced LLM assessments and human bilingual expert evaluations. Extensive experiments demonstrate that SwitchLingua V2 produces CS dialogues that are structurally diverse, pragmatically motivated, and socially situated, holding the potential to offer a highly scalable and reliable pathway for advancing multilingual ASR research.
Show more
CTE-Bench: Audited Counterfactual Trace Evaluation for Stateful Software Simulators
Xinran Zhang
Coding agents built from large language models increasingly inspect and modify stateful services. Before such an agent chooses or explains a patch, it needs a basic engineering skill: predicting how a local code or state change will affect future service behavior. CTE-Bench isolates this simulator-fidelity question from agent policy success. Each item gives the model service source, a factual interaction prefix, a post-prefix source patch or hidden-state overwrite, and a fixed suffix query schedule; an executable oracle supplies the counterfactual response trace. CTE-Bench-Core-v1 contains 255 counterfactual items over six deterministic Python services and 10,200 scored suffix decisions per model configuration. The main score is effect-step value-match (VM): exact response equality restricted to the 2,476 suffix positions where the intervention changes the oracle response. DeepSeek V4-Lite, Kimi K2.5, and Claude Sonnet 4.6 reach 60.7%, 61.5%, and 58.4% effect-step VM under teacher-forced one-step prompting, where each prediction sees earlier oracle suffix responses; hiding those responses while preserving the factual prefix reduces them to 11.4%, 28.9%, and 25.0%. As an additional probe, we run self-conditioned free-rollout on DeepSeek and Kimi; effect-step VM is 27.1% and 33.2%, and exact whole-trace match is at most 1.2%. These probes show that CTE-Bench helps identify dependence on oracle suffix feedback and compounding error in self-conditioned service simulation. We release Core-v1, the executable oracle, metadata, and evaluation scripts, with an evaluation card tying each supported claim to its memory protocol and metric.
Show more
AutoDataBench: How Far Are LLM Agents from Autonomously Engineering Post-Training Data Pipelines?
Qiaoyu Tang ⋅ Hao Xiang ⋅ Le Yu ⋅ Yaojie Lu ⋅ Xianpei Han ⋅ Le Sun ⋅ Bowen Yu ⋅ Jiawei Chen ⋅ Zhenru Zhang ⋅ Peng Wang ⋅ Que Shen ⋅ Hongyu Lin ⋅ Dayiheng Liu
Post-training data pipelines are traditionally hand-designed by researchers who orchestrate generation, verification, and diversity balancing. While Large Language Models (LLMs) can already execute individual subtasks, the architectural composition of these pipelines remains a manual bottleneck, excluding the most substantial engineering step in post-training from automation. We introduce AutoDataBench, a benchmark evaluating whether LLM agents can autonomously design post-training data-synthesis pipelines. Each task gives the agent only a description of a target capability and asks it to produce a complete, executable pipeline whose generated corpus is then used to fine-tune a fixed student model and scored on the evaluation. Each task is delivered as a natural-language instruction and a sandboxed workspace given to the agent, paired with an evaluation protocol. We instantiate AutoDataBench on five capabilities: instruction following, competition math, deep search, repo-level software engineering, and terminal automation. For each task we pair the evaluation with a strong human-designed pipeline whose downstream score serves as a reference point for the agent. Experimental results reveal that frontier agents close most of the gap to the human reference on text-centric tasks (e.g., Opus-4.7 nearly matches humans in instruction following), yet a noticeable performance gap persists in tasks requiring executable-environment construction (SWE and Terminal). Furthermore, we observe a decoupling between downstream task-solving strength and pipeline-construction quality, where strong reasoning models like GPT-5.5 may still produce suboptimal pipelines. These findings suggest that autonomous environment synthesis remains the primary bottleneck for self-improving data loops, distinct from mere task-solving competence.
Show more
When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory
Jiaqi Shao ⋅ Yiyi Lu ⋅ Yunzhen Zhang ⋅ Bing Luo
Memory-agent evaluations report fixed-snapshot accuracy or retrieval quality, but these scores do not show whether evidence remains usable as irrelevant sessions (sessions not annotated as task-relevant evidence for the query) accumulate. We present a scale-conditioned evaluation protocol for agent memory under evidence-preserving growth: for each query, task evidence is held fixed while irrelevant sessions are added. The protocol logs agent--memory trajectories and reports four diagnostics: budget-compliant reliability, tail memory-call burden, failure-regime decomposition, and the usable-scale boundary where reliability falls below the target. Applied to LongMemEval and LoCoMo across flat, planar, and hierarchical memory interfaces, the protocol shows reliability loss is not a single phenomenon. On LongMemEval, HippoRAG stays within the two-call budget but loses 16--20 percentage points in budget-compliant reliability as irrelevant sessions are added; LiCoMemory's observed failures depend strongly on the agent, with Qwen3-8B exceeding the budget while Qwen3-32B and Qwen3-235B remain reliable in the tested range. The result supports a framework for making scalable-memory claims conditional on agent, interface, scale range, and interaction budget. Code and evaluation artifacts: https://github.com/AndyLu666/memory-at-scale.
Show more
CryoAtlas: A Large Curated Dataset and Unified Benchmark for Cryo-EM Atomic Model Building
Mingrui Li ⋅ Minzhang Li ⋅ Weichen Qin ⋅ Yufan Xie ⋅ Zhangzhi Xiong ⋅ Sixian Shen ⋅ Jiakai Zhang ⋅ Yuan Pei ⋅ Jingyi Yu
Deep learning has accelerated automated atomic model building from cryo-EM maps, yet progress remains limited by the lack of large, high-quality datasets and standardized evaluation benchmarks. We address this gap by introducing CryoAtlas, a rigorously curated dataset and unified benchmark for density-guided atomic model building. By integrating and filtering EMDB and PDB entries (up to March 2026), CryoAtlas provides 17,809 map--model--sequence entries, including a curated 16,657-entry training split. From this resource, we construct a 618-target benchmark with a shared evaluation protocol that jointly assesses structural accuracy, model-to-map fit, and stereochemical validity. Using this framework, we benchmark state-of-the-art learning-based methods against Phenix and AlphaFold~3, comprehensively analyzing the impact of map resolution, sequence length, protein category and sequence identity on modeling accuracy. As a case study, retraining the ModelAngelo CNN component on CryoAtlas improves performance on the benchmark. CryoAtlas provides an open, reproducible foundation for future cryo-EM atomic model building research.
Show more
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
Lukas Thede ⋅ Yash Kumar ⋅ David Chen ⋅ Danielle Bitterman ⋅ Matthias Bethge ⋅ Tom Hartvigsen ⋅ Zeynep Akata
Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose safety risks. Existing evaluations of knowledge integration focus on factual recall, offering limited insight into whether newly integrated knowledge is actually usable. Our benchmark __MedKIT__ (Medical Knowledge Integration and Transfer) provides a granular evaluation of how models integrate and apply knowledge under realistic sequences of clinical updates. Each instance corresponds to a factual update derived from clinical evidence, paired with targeted probes that assess transfer across lexical variation, relational transformations, compositional reasoning, and open-ended operationalization, as well as locality tests for knowledge preservation. Using MedKIT, we conduct a large-scale empirical study of 12 knowledge integration strategies across 5 diverse models, including both general-purpose and medical LLMs. Our results reveal a consistent gap between recall and usable knowledge: while most methods achieve strong gains on the original update task and under lexical variation, relational generalization is limited, and no method yields meaningful improvements on compositional or operational tasks. These findings highlight a fundamental challenge in knowledge integration and position MedKIT as a testbed for developing methods that make newly integrated knowledge more consistently usable across tasks and contexts.
Show more
EMERGE: A Benchmark for Updating Knowledge Graphs with Emerging Textual Knowledge
Klim Zaporojets ⋅ Daniel Daza ⋅ Edoardo Barba ⋅ Ira Assent ⋅ Roberto Navigli ⋅ Paul Groth
Knowledge Graphs (KGs) are structured knowledge repositories containing entities and relations between them. In this paper, we study the problem of automatically updating KGs over time in response to evolving knowledge in unstructured textual sources. Addressing this problem requires identifying a wide range of update operations based on the state of an existing KG at a given time and the information extracted from text. This contrasts with traditional information extraction pipelines, which extract knowledge from text independently of the current state of a KG. To address this challenge, we propose a generic and extensible pipeline that pairs textual passages with the KG edit operations they induce on a given KG snapshot. We instantiate this pipeline on the Wikidata knowledge graph and the English Wikipedia corpus to create EMERGE, a dataset of 233K Wikipedia passages associated with a total of 1.19 million KG edits across seven yearly Wikidata snapshots from 2019 to 2025. Our experimental results highlight key challenges in updating KG snapshots based on emerging textual knowledge, particularly in integrating knowledge expressed in text with the existing KG structure. These findings position the dataset as a valuable benchmark for future research. The code and dataset are available at https://github.com/klimzaporojets/emerge.
Show more
IntegrityBench: Can LLMs Be Trusted as Co-Scientists? A Research Integrity Benchmark
Sai Sidhanth Manoharan Jayanthi ⋅ Yash Tripathi ⋅ Silu Sharma ⋅ Shivank Garg ⋅ Lin Li
Language models are increasingly deployed as co-scientists in the real world, yet their ability to uphold research integrity, particularly under institutional pressures, remains unmeasured. We introduce IntegrityBench, a comprehensive benchmark evaluating three facets (misconduct classification, ethical action reasoning and artifact-grounded decision making), using 36 paired misconduct and ethical control tasks under a 5-level implicit-explicit pressure protocol across 3 domains and 4 research stages. Through a large-scale evaluation across 18 frontier model variants, we find that, under the strongest pressures, frontier models fail roughly one in three integrity-critical decisions. Surprisingly, neither scale nor reasoning can reliably enhance research integrity. Explicit pressures reliably induce compliance with misconduct while implicit contextual reframing more often causes over refusal of legitimate research tasks. Across tasks, models treat surface-level cues as evidence of misconduct without recognizing procedural justifications. Further, models failing to classify research requests accurately perform equally or better on artifact-grounded decision making (mean accuracy of 85.7 when classification fails versus 79.4 when it succeeds), demonstrating that the three facets might be structurally dissociated because accurate ethical actions don't require correct classification to precede them. Frontier models can appear helpful while displaying research integrity failures that create two distinct deployment risks: facilitating research misconduct and diminishing trust in AI-assisted research outputs.
Show more
The Fault in Our Metrics: Revisiting Generative Model Evaluation with Chamfer Distance
Chirag Vashist ⋅ Vaibhav Santhosh ⋅ Tristan Engst ⋅ Yanshu Zhang ⋅ Ke Li
Reliable evaluation is essential for progress in image generation, yet widely used metrics rely on assumptions that are misaligned with the geometry of real data. We revisit two dominant families of metrics: Fréchet-based metrics such as FID, and precision/recall-style metrics based on KNN balls. We show that FID reduces complex feature distributions to a Gaussian moment summary, making it blind to differences beyond mean and covariance. We further show that KNN-ball metrics construct inflated isotropic support estimates in high-dimensional feature spaces, causing binary membership queries to accept off-manifold samples and making recall unstable under standard finite-sample evaluations. We propose Chamfer distance as a simple alternative: instead of constructing inflated support estimates, it directly measures cross-set nearest-neighbor distances between real and generated samples. The resulting one-sided distances provide interpretable signals for fidelity and coverage, while reducing sensitivity to the neighborhood hyperparameter and remaining computationally efficient.
Show more
Benchmark Health Index: A Systematic Framework for Benchmarking the Benchmarks of LLMs
Longyuan Zhu ⋅ Hairan Hua ⋅ Linlin Miao ⋅ Wei Wang ⋅ 必应 赵
Large Language Models (LLMs) are advancing rapidly, yet the benchmarks used to measure this progress are becoming increasingly unreliable. Score inflation and selective reporting have eroded the authority of standard benchmarks, leaving the community uncertain about which evaluation results remain trustworthy. We introduce the **Benchmark Health Index (BHI)**, a pure data-driven framework for auditing evaluation sets along three orthogonal and complementary axes: (1) **Capability Discrimination**, measuring how sharply a benchmark separates model performance beyond noise; (2) **Anti-Saturation**, estimating remaining headroom before ceiling effects erode resolution and thus the benchmark's expected longevity; and (3) **Impact**, measuring capability-weighted benchmark adoption across the model ecosystem. By distilling **158** validated benchmarks from the technical reports of **117** representative models released between **2025** and **April 2026**, we systematically characterize the evaluation landscape. BHI is the **first framework** to quantify benchmark health at a macro level, providing a well-grounded quantitative basis for benchmark selection and supporting the dynamic filtering and continuous updating of evaluation sets. Our comprehensive analysis not only identifies benchmarks that remain highly credible but also exposes pervasive structural issues. By establishing a rigorous foundation for selection and management, BHI elevates evaluation practice from heuristic-based judgment to a quantifiable and interpretable scientific process. Our code is available online at
Show more
probly: Uncertainty-Aware Machine Learning
Paul Hofman ⋅ Clemens Damke ⋅ Timo Löhr ⋅ Santo Thies ⋅ Alireza Javanmardi ⋅ Valentin Margraf ⋅ Jakub Paplhám ⋅ Yusuf Sale ⋅ Eyke Hüllermeier ⋅ Maximilian Muschalik
As machine learning systems are increasingly deployed in real-world applications, the question of how to represent and quantify uncertainty has moved from a methodological side issue to a central concern. In practice, however, making a model uncertainty-aware is still surprisingly difficult: relevant tools tend to be scattered across libraries, each tied to a particular framework and a particular approach to modeling uncertainty, and the choice of representation, quantification, and evaluation is typically left to the user without much guidance. In this paper, we present probly, a Python package that addresses theses issues in a single, modular framework. probly offers (i) lightweight transformations that turn existing models into uncertainty-aware ones, currently supporting PyTorch, scikit-learn, and Flax, (ii) several representations, including second-order distributions, credal sets and conformal prediction, (iii) the corresponding quantification measures, and (iv) a number of evaluation protocols, which can be combined more or less arbitrarily. Using probly, we conduct a benchmark study on three standard tasks --- selective prediction, out-of-distribution detection, and active learning. In addition, and unlike existing benchmarks, we evaluate suitable methods on first-order data, i.e., datasets for which the target itself is a distribution over outcomes. We further illustrate the flexibility of the package on a number of less standard case studies, including large language models, graph neural networks, and data streams. The package is publicly available at \url{https://anonymous.4open.science/r/probly}.
Show more
GenAI Evaluation Results are Largely Artifacts of Evaluation Design Choices: Evidence from Audit Studies of Resume Screening
Nicole Meister ⋅ Hannah Cha ⋅ Mohammed Alsobay ⋅ Alexandra Chouldechova ⋅ Alex Dow ⋅ Hanna Wallach ⋅ Jennifer Wortman Vaughan ⋅ Carlos Guestrin ⋅ Tatsunori Hashimoto ⋅ Solon Barocas
Generative AI is designed to support many tasks, many ways of framing those tasks, and seemingly endless variation in task inputs. But the same flexibility that makes these systems powerful also makes them difficult to evaluate reliably. This flexibility exacerbates the numerous, but often underrecognized and consequential, \emph{researcher degrees of freedom} where even seemingly minor experiment design decisions can substantially affect findings. We demonstrate this problem through an analysis of 36 papers on gender and racial bias in generative AI-assisted resume screening. Although these studies address the same broad research question, they reach strikingly divergent conclusions. To explain this divergence, we adapt recent work on integrative experiment design from the social sciences, and decompose the dimensions along which the studies vary. We then run 1,134 counterfactual experiments by combinatorially varying the experimental modules (names, job-resume pairings, and models) across three baseline experimental frameworks. We find the results prove remarkably brittle: a different design choice flips the conclusion of the reproduced experiment around 22\% of the time. For example, seemingly inconsequential design choices, such as the phrasing of a job description, can reverse the observed conclusion. However, this instability is not always purely stochastic. In some settings, we can build a model that predicts the result of unseen counterfactual experiments with a mean held-out predictive $R^2$ of 0.807. This suggests that despite sensitivities to difference in evaluation design, there exists a learnable structure.
Show more
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
Jinhao Jing ⋅ Zheng Ma ⋅ Jinwei Liang ⋅ Qiannian Zhao ⋅ Shuang Chen ⋅ Jing Yang ⋅ Prayag Tiwari ⋅ Jingjing Bai ⋅ Benyou Wang ⋅ Por L Yee ⋅ Zhan Su ⋅ Lewei Lu
Large Multimodal Models (LMMs) often struggle with geometric reasoning due to visual hallucinations and a lack of mathematically precise Chain-of-Thought (CoT) data. To address this, we propose the GeoSym Engine, an automated and scalable neuro-symbolic framework. By leveraging a type-conditional grammar and an analytic SymGT Solver, it derives exact symbolic ground truths and seamlessly integrates with a robust rendering pipeline to produce high-precision geometric diagrams. Using this engine, we construct GeoSym127K, a difficulty-stratified dataset featuring 51K high-resolution images, 127K questions with symbolic ground truths, and 55K answer-verified CoT QA pairs. We also introduce GeoSym-Bench, an expert-curated suite of 511 complex samples for rigorous evaluation. Through extensive supervised fine-tuning (SFT), we demonstrate that GeoSym drives concentrated improvements specifically on diagram-dependent and multi-step geometry tasks. Our Qwen3-VL-8B model gains an absolute +22.21% on the MathVerse Vision-Only subset and reaches 61.52% (+6.19% improvement) on WeMath, mitigating long-horizon logic fragmentation and outperforming advanced closed-source models like Doubao-1.8. Furthermore, applying Reinforcement Learning with Verifiable Rewards (RLVR) via GRPO reveals that initializing from structural SFT checkpoints substantially elevates the performance ceiling over zero-shot RL. Driven by deterministic exact-match signals, this showcases the robust scaling potential of our verifiable reasoning synthesis. Datasets and code are available at https://huggingface.co/datasets/Tomie0506/GeoSym127K and https://github.com/Tomie56/GeoSym127K
Show more
HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation
Jaehwi Song ⋅ Suchae Jeong ⋅ Byeongguk Jeon ⋅ Sungdong Kim ⋅ Hyungmok Son ⋅ Kimin Lee
Large-scale demonstration datasets have been central to recent progress in general-purpose robot policies. However, existing datasets are collected in human-absent settings, and policies trained on such data may perform tasks competently in isolation but fail to exhibit human-aware behaviors. To address this gap, we introduce HABIT, a large-scale robot demonstration dataset for human-present environments. We organize tasks into three roles capturing distinct modes of human-robot interaction: Collaborator, where human and robot jointly accomplish a task; Coworker, where they pursue separate tasks in a shared space; and Supervisor, where the human directs the robot. The dataset comprises over 7K episodes and over 100 hours across 48 tasks. Our experiments show that training on human-present data elicits human-aware behaviors that robot-only data fails to produce: spatiotemporal synchronization in Collaborator tasks, yielding in Coworker tasks, and gesture grounding in Supervisor tasks. Moreover, training on HABIT enables rapid adaptation to new human-robot interaction tasks. By introducing human presence as a new axis of dataset diversity, HABIT extends robot policies to environments shared with humans.
Show more
Can LLM Agents Respond to Disasters? Benchmarking Heterogeneous Geospatial Reasoning in Emergency Operations
Junjue Wang ⋅ Weihao Xuan ⋅ Heli Qi ⋅ Pengyu Dai ⋅ Kunyi Liu ⋅ Hongruixuan Chen ⋅ Zhuo Zheng ⋅ Junshi Xia ⋅ Stefano Ermon ⋅ Naoto YOKOYA
Operational disaster response goes beyond damage assessment, requiring responders to integrate multi-sensor signals, reason over road networks, populations and key facilities, plan evacuations, and produce actionable reports. However, prior work largely isolates remote-sensing perception or evaluates generic tool use, leaving the end-to-end workflows of emergency operations underexplored. In this paper, we introduce **D**isaster **O**perational **R**esponse **A**gent benchmark (**DORA**), the first agentic benchmark for end-to-end disaster response: 515 expert-authored tasks across 45 real-world disaster events spanning 10 types, paired with expert-verified, replayable gold trajectories totaling 3,500 tool-call steps. Tasks span five dimensions that cover the operational disaster-response pipeline: disaster perception, spatial relational analysis, rescue and evacuation planning, temporal evolution reasoning, and multi-modal report synthesis. Agents compose calls from a 108-tool MCP library over heterogeneous geospatial data: optical, SAR, and multi-spectral imagery across single-, bi-, and multi-temporal sequences (0.015-10m GSD), complemented by elevation and social vector layers. We comprehensively evaluate 13 frontier LLMs on our benchmark, revealing three persistent challenges: 1) disaster-domain grounding exposes unique failure modes (damage-semantic grounding, sensor-modality mismatch, and disaster-pipeline composition); 2) agents are doubly bottlenecked by tool selection and argument grounding, where gold tool-order hints improve accuracy by only 1.08-4.40%, and alternative scaffolds yield at most a 3.24% gain; 3) compositional fragility scales with trajectory length, the agent-to-gold gap widening from 7% to 56% on long pipelines. DORA establishes a rigorous testbed for operationally reliable disaster-response agents.
Show more
Offloading Score: Measuring AI Reliance through Counterfactual Workflows
Vishakh Padmakumar ⋅ Lujain Ibrahim ⋅ Zora Wang ⋅ Jennifer Wang ⋅ Q.Vera Liao ⋅ Diyi Yang
AI tools are increasingly integrated into real-world workflows. However, existing measures of reliance on these tools focus on AI output adoption or on self-reported indicators, rather than how task effort is distributed between users and tools. Here, we introduce offloading score, a measure of reliance that quantifies the fraction of cognitive effort offloaded to an AI tool. Offloading score is simulation-based---we construct a counterfactual workflow by estimating how the user would have completed the task without the tool, and then computing the fraction of steps saved by using the tool. We validate offloading score through intrinsic evaluations of metric validity, and a controlled user study ($n=40$) with developers performing programming tasks using AI tools. We vary time pressure to test whether reliance measures capture the known increase in reliance under time pressure. We show that offloading score detects significantly higher reliance in time-constrained settings ($+43\%$, $p=0.018$), while usage-based and self-reported baseline measures do not distinguish the conditions. We complement this with descriptive insights showing that higher reliance manifests as greater delegation and more direct reuse of AI outputs. Finally, we demonstrate an approach of using offloading score in combination with target outcomes of reliance (e.g., task understanding) to identify when reliance may be (in)appropriate. Our framework offers two contributions: an instrument users can apply to measure and reflect on their own reliance, and a quantitative signal that agent designers can utilize to mitigate overreliance.
Show more
Evaluating Neural Data Tokenizers: A Framework for Assessing Learned Representations of Spiking Activity
Federico D'Agostino ⋅ Alex Gilbert ⋅ Susanne Keller ⋅ Jaivardhan Kapoor ⋅ Nicolas Reategui ⋅ Vedang Lad ⋅ Matthias Kümmerer ⋅ Jai Bhagat ⋅ Emin Orhan ⋅ Hasan A Bedel ⋅ Nikos Karantzas ⋅ Fabian Sinz ⋅ Surya Ganguli ⋅ Jakob H Macke ⋅ Sophia Sanborn ⋅ Alexander Ecker ⋅ Katrin Franke ⋅ Matthias Bethge ⋅ Andreas Tolias ⋅ Konstantin Willeke
Foundation models of brain activity aim to learn representations from large-scale neural recordings that generalize across sessions, subjects, species, and downstream tasks. Transformer-based models have emerged as a promising architecture for this goal and begin all with the same first step --- tokenization: transforming sparse binary events into vector representations that downstream networks can consume. In other areas of machine learning, tokenization is its own design problem, distinct from the downstream foundation model. Several mostly implicit tokenization approaches dominate the literature, but their impact on downstream performance has not been systematically evaluated. We propose four explicit desiderata for a neural tokenizer---reconstruction fidelity, downstream utility, cross-instance generalization, and compression---and a framework of metrics spanning seven dimensions, including firing statistics, encoding and decoding, and latent structure and robustness. We assemble a benchmark of ten datasets spanning species, brain areas, recording technologies, and behavioral paradigms, and evaluate a representative set of discrete and continuous tokenizers, both single-neuron and population variants. Across the strategies evaluated, we show how tokenizers trained only on reconstruction can support downstream tasks. Yet reconstruction alone is misleading: reconstruction quality and downstream utility can be sharply dissociated. Our multi-axis framework is a response to this failure mode. These findings highlight tokenization as a critical and underexplored design axis in brain foundation models, and suggest that further progress will require continued systematic study of tokenization strategies.
Show more
ALTER: An Allen's Algebra-Based Evaluation Framework for Temporal Reasoning of LLMs
Hongkai Zhou ⋅ Yanhui Li
Despite the critical role of temporal reasoning in Large Language Models (LLMs), current synthetic benchmarks fall short: they either introduce information leakage or, upon addressing this issue, merely assess questions with explicit time points, failing to rigorously evaluate implicit temporal reasoning capabilities. To bridge this gap, we propose a formally grounded evaluation framework, ALTER, based on Allen’s Interval Algebra. By combining basic relations into complex networks and converting these structures into natural language forms through an automated pipeline, we construct a validated benchmark of 2,500 question-answer pairs, ensuring a controllable and scalable evaluation environment. We conduct experiments on a representative selection of LLMs. Empirical evaluations on diverse LLMs yield three key findings: (1) identifying performance decline of LLMs in highly complex scenarios involving dense events; (2) isolating complex logical interactions—rather than linguistic comprehension—as an unresolved bottleneck; and (3) revealing a critical evaluation bias induced by narrative chronology, with LLMs exploiting textual shortcuts. In summary, these findings reveal the blind spots of current evaluation paradigms, providing a rigorous foundation for the robust evaluation of complex temporal reasoning.
Show more
Successful Page Load