Paris Evaluations & Datasets
AM-Bench: A Unified Taxonomy and Evaluation Suite for Agentic Misalignment
Eric Zhang ⋅ Terry J. C. Zhang ⋅ Chijioke Ugwuanyi ⋅ Jerick Shi ⋅ Bernhard Schölkopf ⋅ Zhijing Jin
As LLM-based agents are deployed with increasing autonomy in real-world environments, a critical failure mode emerges: agentic misalignment, where agents choose harmful actions over failure. However, existing safety benchmarks primarily evaluate whether agents comply with explicitly harmful instructions, leaving the conditions that cause agents to choose harmful actions poorly understood. To address this limitation, we propose AM-Bench, a unified taxonomy and 35 scenario evaluation benchmark spanning seven industries for agentic misalignment grounded in the fraud triangle, a well-established criminological framework explaining goal-directed misconduct. We adapt this framework to agentic environments and construct our evaluation suite and detection systems along three dimensions: perceived threat, perceived opportunity, and reasoning. We operationalize each dimension into structured subcategories rooted in prior theory: five threat categories derived from the instrumental convergence thesis, seven opportunity categories drawn from the Emergent Strategic Reasoning Risks (ESRR) taxonomy, and a novel four-question binary reasoning framework synthesized from prior literature. Our evaluation of 10 frontier models reveals a 62-percentage-point gap in misalignment across scenarios, with Gemini 3 Flash and Gemini 3.1 Pro exhibiting the highest levels of agentic misalignment while Claude Sonnet 4.6 displays near-zero rates across all conditions. Our reasoning framework further reveals that models may exhibit harmful behaviours without explicit situational awareness or explanation, complicating both mitigation and detection. AM-Bench scenarios and code are available at https://github.com/AI4Collaboration/Agentic-Misalignment-Bench
Show more
RASS: Risk-Audited Budget Selection for Compact NeRF Benchmark Subsets
Ahmad AlMughrabi ⋅ Flavià Serret ⋅ Ricardo Marques ⋅ Petia Radeva
Full-dataset NeRF evaluation can require training and evaluating thousands of scenes, making exhaustive comparison expensive even when the benchmark is fixed. We introduce RASS, a risk-audited protocol for constructing compact NeRF benchmark subsets with explicit fidelity criteria. Given full-dataset per-scene logs, RASS repeatedly samples candidate subsets, checks whether their global metric means and score distributions match the full benchmark within user-defined tolerances, and recommends the smallest evaluated subset size whose Wilson lower confidence bound exceeds a target pass probability. We instantiate RASS on 3,522 Nutrition5k-derived food scenes using Zip-NeRF logs for the formal audit and PSNR, SSIM, LPIPS, and Kolmogorov-Smirnov (KS) distributional tests as fidelity criteria. The released 48-scene balanced operating point, RASS-48, is the lowest-cost evaluated screening subset that passes our declared global-fidelity audit and is intended for low-cost early ablations and rapid screening. A stricter target pass probability, $p_{\min}=0.20$, selects RASS-96, which we recommend as the compact reporting subset: it also passes joint fidelity audits over three and four view-synthesis methods. We also evaluate alternative subset generators, including uniform random sampling, farthest-first selection, and facility-location diversity sampling, and find that the RASS audit can validate different generators under the same fidelity contract. On DL3DV, a re-declared contract certifies 64 of 140 scenes for four methods under proportional allocation. We release the RASS-48 and RASS-96 scenes, configurations, descriptors, regime labels, seeds, audit scripts, and permitted logs to enable reproducible, compact NeRF evaluation. Project page: https://amughrabi.github.io/RASS/.
Show more
GT-Free OCR Metrics: A Reference-Free Evaluation Framework for Document OCR Systems
Kshitij Singh
Evaluating document OCR systems traditionally requires ground-truth (GT) text annotations aligned to each recognised element. Producing such annotations demands expert human labour, scales poorly with corpus size, varies between annotators on ambiguous content, ages quickly as document distributions and OCR systems evolve, and is often infeasible for sensitive or proprietary material. These constraints prevent continuous OCR quality monitoring at production scale and force practitioners to rely on small audited samples or proxy heuristics. To overcome these inherent limitations of ground-truth-based evaluations, we present a new class of reference-free OCR metrics based on render-and-compare that do not require any ground-truth annotations: OCR output is rendered back into a page image and the result is compared visually against the masked original using no-reference or learned metrics. The underlying principle is straightforward: the original page image already serves as a pixel-level ground truth for the content the OCR system tries to extract (text, formulas, and tables), so visual agreement between the original and a re-rendering of the OCR output is itself a proxy for recognition fidelity. To our knowledge, no prior work has systematically benchmarked reference-free visual metrics as proxies for whole-document OCR quality at scale. We conduct the first such study, evaluating 147 method implementations spanning 15 broad categories of techniques on 1,355 OmniDocBench pages across 5 OCR-output variants (text-only, formula-only, table-only, combined, combined without image masking). We measure every method by Spearman rank correlation against ground-truth-based metrics (edit distance for text and formulae, TEDS for tables), quantifying whether a reference-free metric correctly orders pages by OCR quality without any annotations. A naïve CLIP-cosine full-page similarity baseline achieves Spearman correlation $\rho = 0.339$; our best composite method reaches $\rho = 0.494$ (mean across the 5 variants; significance $p < 0.001$, computed over $N = 1{,}355$ pages), with a per-variant peak of $\rho = 0.605$ on the formula-only variant. Both numbers indicate that reference-free OCR quality estimation is genuinely viable: a meaningful page-level ranking signal can be recovered without any ground-truth annotations, and the gap to the ground-truth-based ranking is consistent with the inherent noise of a multi-domain document corpus where OCR errors are heterogeneous across text, formula, and table content. We also discuss inherent limitations that bound what any reference-free metric can achieve in this setting. To support reproducibility and for future research in this direction we are releasing three public HuggingFace datasets (rendered page pairs, per-element OCR log-probabilities, document-similarity triplets), and a trained document-similarity (DocSim) LoRA head (a DreamSim-style recipe adapted to document page reconstructions) along with the associated code.
Show more
RESCAST-100K: A Comprehensive Dataset for Cross-Domain Residential Load and Indoor Temperature Forecasting
Jainam Dhruva ⋅ Yousaf Raza ⋅ A.B. Siddique ⋅ Simone Silvestri
Accurate short-term forecasting of residential energy load and indoor temperature is essential for home energy management systems, grid-level demand response participation, and community-level energy efficiency programs. Domain adaptation methods and transfer learning have shown promise in improving forecasting accuracy under data heterogeneity and scarcity scenarios commonly observed at the residential level. However, progress remains limited by the absence of comprehensive residential datasets: existing ones are narrow in target coverage and lack structured support for cross-domain evaluation. To address this, we introduce RESCAST-100K, a large-scale residential forecasting benchmark for studying cross-domain generalization. It exposes a configuration-driven interface that instantiates source and target domains along interpretable axes, such as geography, climate zone, wall construction, and heating equipment, supporting systematic evaluation of transfer learning, domain adaptation, and zero-shot generalization under controlled domain shifts. The benchmark covers approximately 100,000 EnergyPlus-simulated U.S. homes derived from ResStock, providing 15-minute time series of three coupled targets per home, namely total load, HVAC (heating, ventilation, and air conditioning) load, and indoor temperature, together with weather channels, HVAC setpoints, and over 40 static building covariates. Finally, it integrates five real-world residential datasets under a unified schema, supporting sim-to-real evaluation on the same tasks. We benchmark models from recurrent, attention, and MLP-mixer architectures for zero-shot performance across different domains, under missing input data conditions, for each task. Cross-attention and MLP-mixer based models consistently outperform recurrent and classical transformer-based baselines under domain shift. RESCAST-100K is intended to support the machine learning and building analytics communities in advancing cross-domain residential forecasting at the home, community, and grid levels.
Show more
haphazard: A unified library and benchmark for online learning under varying feature availability
Rohit Agarwal ⋅ Arijit Das ⋅ Dilip K. Prasad
Haphazard inputs arise in online learning when feature availability varies across instances and the complete feature space is unknown in advance. However, the landscape of haphazard inputs research has been fragmented due to inconsistent data preprocessing, non-uniform evaluation metrics, limited coverage of streaming regimes and normalization, and model specific experimental code, making direct comparison across existing methods unreliable. We present haphazard, an open source Python library for reproducible benchmarking of haphazard input methods. The library provides an order preserving data pipeline with streaming normalization, unified evaluation, four streaming regimes, and shared interfaces for datasets, models, normalizers, and metrics. The current release supports 22 datasets, 11 model implementations, 5 normalization strategies, and 7 evaluation metrics, enabling systematic benchmarking across datasets and stream settings. By consolidating the experimental pipeline and exposing extensible interfaces for datasets, models, normalizers, metrics, and streaming regimes, haphazard provides a common reference point that can grow with future research on haphazard inputs. The anonymized source code is available at https://anonymous.4open.science/r/haphazard, and the PyPI package and docker container will be released publicly upon acceptance.
Show more
A Reproducible Evaluation Protocol for Quantum Non-Linearity in Variational Quantum Models
Pavel Sulimov ⋅ Claude Lehmann
Variational quantum models are linear maps of the input state: any useful non-linearity must be engineered through the data encoding, entanglement pattern, ancilla tracing, classical readouts, or explicit classical layers. Although many such mechanisms have been proposed, there is no standard protocol that scores them on a common footing. Most published comparisons reduce non-linearity to either the accuracy on a single downstream benchmark or a single abstract expressibility number, and the two are rarely measured under matched training budgets, circuit widths, and noise assumptions. We introduce a reproducible evaluation protocol that decomposes non-linearity quality into four complementary scalar scores covering frequency reach, gradient-variance scaling, trainability efficiency on a chaotic dynamical task, and stability under depolarizing noise. In addition, we define a composite index with published weights, enabling methods to be ranked under a single criterion while preserving full access to the underlying components. We instantiate the protocol with six variational ansatzes spanning the standard taxonomy of quantum non-linearity sources (data reuploading, ancilla tracing, their combination, strongly entangling layers, a quantum convolutional network, and a quantum reservoir), together with a classical multilayer perceptron and a hybrid quantum-classical network as non-quantum reference points. The evaluation surfaces a central mismatch that single-number reporting hides: in our reference run, the composite leader (data reuploading) is near the bottom on held-out Lorenz one-step $R^2$, while the Lorenz leader (ancilla reuploading) is only third on the composite. The main outcome is a practical benchmark that makes these trade-offs visible and testable: with fixed tasks, budgets, and seeds, new methods can be added by registering a circuit builder and rerunning the same protocol, and multi-seed aggregates can be updated without changing metric definitions.
Show more
Evaluating Multimodal Narrative Understanding of Popular Hollywood Films
David Bamman ⋅ Kent K Chang ⋅ Allison Cooper ⋅ Juishan Hsu ⋅ Reina Kushihashi ⋅ Madison Mar ⋅ Arnav Podichetty ⋅ Rachael G Samberg ⋅ Ipek N Sancak ⋅ Yuhan Shao
Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning about film history and the evolution of narrative techniques. But the creation of stable benchmarks built around Hollywood films is complicated by copyright protections. In this work, we address these concerns directly, by building a new collection of Hollywood films defined by two criteria: box office popularity (where we publish the first large-scale, open collection of weekly box office earnings reported by Variety magazine from 1922-1979); and likely public domain status (by researching copyright registrations and renewals in the US Catalog of Copyright Entries). We build a new multimodal MCQ benchmark on top of this collection that focuses on narrative elements that directly evaluate the abilities of models to inform meaningful research on film narrative; we find that many vision-language models struggle on this task (with many performing at near-chance levels of accuracy), while audio-visual models (including those that use audio in captioning scenes) reach a maximum accuracy of 61.1%, well below human-level performance.
Show more
Benchmarking sequence-to-ensemble predictors on UNICORNEdb, a UniProt-grouped database of PDB-derived conformational ensembles
William F Vanderbilt-Fried ⋅ Luka Butskhrikidze ⋅ Benjamin P Brown ⋅ Hassane Mchaourab ⋅ Carlos Oliver
Following the striking success of ML-driven static protein structure prediction, the challenge of protein conformational flexibility and thermodynamic ensemble prediction is rapidly coming to the foreground, and with it the need for rigorous benchmarks. Yet, obtaining ground-truth conformational states at scale remains highly challenging. Past approaches are either small and hand-curated or built by clustering PDB chains, which conflates genuine conformational variation with sequence-induced differences across near-homologs. Here we introduce UNICORNE, a database of 50,000+ protein conformational ensembles centered on UniProt and PDB chain/entity annotations to assemble per-protein ensembles of experimentally resolved conformations without relying on clustering. From our database, we derive UNICORNE-BENCH, a set of 839 multi-conformation proteins on which we extensively benchmark seven recent sequence-conditioned predictors spanning generative ensemble models and MSA-perturbation methods. We find that the ensemble-generation problem is far from solved and identify several shared failure modes: even the strongest tool fully recovers only 8.2\% of ensembles, performance degrades with conformational diversity, and tools fail on overlapping rather than complementary subsets of targets. Across the benchmark, we find that MSA-perturbation methods modestly outperform direct generative approaches. UNICORNE provides a biologically grounded reference needed to guide the next generation protein structure prediction. The full database and leaderboard are available at https://unicornedb.org (anonymized), and source code is available at https://anonymous.4open.science/r/unicorne_bench_neurips_source-2327/.
Show more
Federated Dataset Simulation: Inducing Label-Free Heterogeneity Across Tasks
Vasilis Siomos ⋅ Lam Ngo ⋅ Jonathan Passerat-Palmbach ⋅ Giacomo Tarroni
Federated Learning enables collaborative model training across decentralized clients without sharing raw data. While the field has seen rapid growth, the datasets underpinning FL evaluation have not kept pace: real federated datasets remain scarce, and most studies rely on small-scale centralized classification datasets partitioned via Dirichlet label skew, a one-dimensional proxy for the multi-dimensional heterogeneity present in real federations. Moreover, label-based partitioning is by construction inapplicable to non-classification tasks such as segmentation and detection, for which no principled simulation method exists. We cast the conversion of centralized datasets into federated benchmarks as the task of Federated Dataset Simulation (FDS), and decompose it into a representation function $\Phi$ and an assignment mechanism $\pi$. We provide two label-free instantiations---geometric partitioning via pretrained model embeddings and semantic partitioning via vision-language model-derived criteria---applicable to classification and dense prediction tasks alike. We evaluate across four datasets spanning classification, segmentation, and detection, and show that both methods induce heterogeneity that matches or exceeds that of label-skew partitioning, with measurable impact on downstream FL performance. We release \texttt{fds}, a modular open-source package for end-to-end federated dataset simulation, heterogeneity measurement, and FL training, at \url{https://anonymous.4open.science/r/FDS-neurips/}.
Show more
AmaraSpatial-10K: A Spatially and Semantically Aligned 3D Dataset for Spatial Computing and Embodied AI
Mohammadsadegh Salehi ⋅ Alexander J Perkins ⋅ Igor P Maurell ⋅ Ashkan Dabbagh ⋅ Raymond Wong
Web-scale 3D asset collections are abundant but rarely deployment-ready, suffering from arbitrary metric scaling, incorrect pivots, brittle geometry, and incomplete textures, defects that limit their use in embodied AI, robotics, and spatial computing. We present $\textbf{AmaraSpatial-10K}$, a dataset of over $10{,}000$ synthetic 3D assets optimised for zero-shot deployment. Each asset ships as a metric-scaled, deterministically anchored $\texttt{.glb}$ with separated PBR maps, a convex collision hull, a paired reference image, and multi-sentence text metadata. Alongside the dataset we introduce a reusable evaluation suite for 3D asset banks, a continuous Scale Plausibility Score (SPS), an LLM Concept Density metric, anchor-error auditing, and a cross-modal CLIP coherence protocol, and apply it to AmaraSpatial-10K alongside matched subsets of Objaverse, HSSD, ABO, and GSO. AmaraSpatial-10K improves CLIP Recall@5 by $3.4\times$ over Objaverse ($0.612$ vs.\ $0.181$, median rank $267 \rightarrow 3$), achieves a $99.1\%$ physics-stability rate under Habitat-Sim with $\sim$$20\times$ wall-time speed-up, and produces zero-overlap scenes when used as a drop-in asset bank for Holodeck. Controlled ablations on the same asset bank attribute the retrieval gain to description richness.
Show more
HumanStereo: A Benchmark for Metric Facial Depth Estimation and Its Evaluation
Alexandru-Ion Marinescu ⋅ DIANA-LAURA BORZA ⋅ DARABANT Sergiu Adrian
Stereo depth estimation of human faces is a prerequisite for various computer vision applications, yet existing stereo benchmarks target outdoor, driving, or large object reconstruction scenarios and lack the anatomical precision and evaluation framework these applications require. We introduce HumanStereo, the first large-scale synthetic stereo dataset dedicated to human faces, comprising 417k stereo pairs from 128 photorealistic avatars across 109 unique stereo configurations and four indoor environments, with dense ground-truth annotations including depth maps, surface normals, segmentation masks, and facial landmarks. We argue that standard evaluation metrics such as $\delta_1$ and RMSE are ill-suited for facial depth estimation, and propose two complementary metrics: task-grounded absolute inlier-ratio (IR) and Landmark 3D Distance Error (L3DE) for metric anatomical accuracy. We benchmark five representative stereo methods: geometrical matchers, RAFT-Stereo, IGEV++, MASt3R, and FoundationStereo. A downstream evaluation on real optometric data for inter-pupillary distance (IPD) measurement, a parameter required in all eyeglass prescription, shows that even the best-performing model remains an order of magnitude above the sub-millimeter accuracy required for clinical deployment.
Show more
Auditing Attention Head Masking for Out-of-Distribution Detection: Cross-Architecture Wins, Failures, and Polarity Inversions
Vishnu Bindu Balachandran
Post-hoc scoring functions for out-of-distribution (OOD) detection in transformer models are typically validated on a single architecture and a single modality, leaving cross-architecture behavior largely uncharted. We report a cross-modality empirical study of attention-head-masking-based OOD detection across more than 1,200 AUROC measurements, spanning LayoutLMv3 on document classification, ViT-B/16 on natural images, and RoBERTa-base on text. Each setting is evaluated against seventeen baselines under a uniform protocol with five fixed seeds, including the Attention Head Sensitivity Scoring (AHSS) and AHM-augmented distance methods of Constantinou et al. (2024) and the recent X-Mahalanobis from NeurIPS 2025. Paired statistical tests are run on every head-to-head claim. Within the AHM family, mahAvgAvg_AHM significantly beats X-Mahalanobis on Tobacco-Cross (Wilcoxon $p=0.031$) and on RoBERTa SST-2$\to$20-Newsgroups ($p=0.031$); on standard CIFAR vision benchmarks, X-Mahalanobis remains state-of-the-art. Three architecture-dependent phenomena surface that current evaluation practice tends to miss: Monte Carlo Dropout collapses to chance-level AUROC across all four ViT settings tested; the AHSS displacement score inverts polarity on RoBERTa, yielding a mean AUROC of 0.296 even though it functions on vision and document transformers; and on near-OOD splits where ID and OOD share visual or semantic structure, every method including the strongest baselines degrades by 25 to 42 percentage points relative to standard OOD. We release per-seed measurements, AHM masks, and code to support replication.
Show more
CroissantMiner: Automated Extraction and Validation of Croissant Metadata for ML Datasets
Berke Arda ⋅ Mubashara Akhtar ⋅ Ahmetcan Yavuz ⋅ Paul Gerry ⋅ Sebastian Lobentanzer ⋅ Nobin Sarwar ⋅ Joan Giner-Miguelez ⋅ Kongtao Chen ⋅ Luyao Zhang ⋅ Mrinmaya Sachan
Croissant has emerged as a standard for machine-readable dataset metadata, yet populating its fields remains labor-intensive and requires careful reading of accompanying dataset documentation. The benchmark comprises 602 papers, including 102 with human-validated gold annotations (9,595 ratings) and 500 with LLM-validated silver annotations, covering the full Croissant schema with both core and Responsible AI (RAI) fields. To our knowledge, this is the first benchmark enabling end-to-end evaluation of metadata extraction aligned with a community-standard schema. We evaluate 20 extraction systems, including frontier and open-weight models as well as four agentic architectures, using a two-tier evaluation framework that combines rule-based scoring with an LLM judge selected via human audit. Our results show that single-pass extraction with Claude Sonnet 4.6 and a canonical prompt is the strongest configuration (composite 0.704). Across all backbones, agentic architectures consistently underperform their single-pass counterparts, indicating that task decomposition degrades performance for schema-complete metadata extraction. This effect is most prevalent on long-form RAI fields, where accurate extraction requires integrating information across multiple sections. Claude Sonnet 4.5, used to seed the gold pre-fills, is excluded from the headline ranking to avoid circularity. We release the benchmark, evaluation code, and judge audit anonymously for review at https://anonymous.4open.science/r/croissantminer-F87C, with a live Hugging Face Space demo to be provided at camera-ready.
Show more
ContextShift: A Controlled Benchmark for Context Dependence in Object Detection
Dan Zlotnikov ⋅ Alex Lazarovich ⋅ Ohad Ben-Shahar
Modern object detectors achieve strong performance on standard benchmarks, yet their robustness to contextual variation remains insufficiently investigated. Prior evaluations largely rely on aggregate metrics such as AP on uncontrolled distribution shifts, which conflate multiple factors and can obscure how performance degrades under context change. We introduce \textsc{ContextShift}, a controlled benchmark for evaluating context dependence by systematically manipulating object--context relationships while preserving object appearance. Built on COCO 2017, it isolates context as an independent variable both implicitly, through geometric transformations and explicitly via synthetic and natural background substitutions - with a compatibility axis based on normalized pointwise mutual information (NPMI) for continuous evaluation. Across diverse detector architectures, we observe a consistent degradation pattern: false negatives increase up to 227\% and prediction volume decreases down to -44\%, while false positives remain relatively stable or decline. This observed suppression behavior is not revealed by aggregate metrics such as AP, which can mask substantial recall loss and changes in prediction dynamics. Further analysis suggests that degradation is not primarily explained by reduced confidence, but is instead consistent with a reduced formation of valid detection candidates. Moreover, we find that performance along the statistical compatibility axis is non-monotonic, peaking at intermediate NPMI and degrading toward both extremes, indicating that statistical co-occurrence does not correlate linearly with effective visual context. Finally, we show that a training methodology based on context-aware augmentations rather than the dataset alone, which yields overall better models: every augmented variant outperforms the dataset-only baseline on both original (unmanipulated) test images as well as manipulated images, indicating that data-oriented augmentation can partially recover performance lost to prediction-suppression failures by exposing the model to object--context decoupling during training.
Show more
Martine: Benchmarking Multi-View 3D Surface Reconstruction Across Viewpoint Coverage, Resolution, and Lighting
Robin Bruneau ⋅ Baptiste Brument ⋅ Frederic Giraud ⋅ Bastian Sigrist ⋅ Sascha Jecklin ⋅ Philipp Fürnstahl ⋅ Bjoern Menze ⋅ Lilian Calvet
Controlled acquisition setups are widely used for high-accuracy multi-view 3D surface reconstruction, as they allow sensor configuration, image resolution, viewpoint distribution, and illumination to be specified. Yet benchmarking remains difficult, since reconstruction accuracy depends jointly on the reconstruction paradigm, object properties, and acquisition regime. Existing benchmarks often restrict viewpoint coverage, resolution, or illumination, which can bias comparisons toward methods suited to the chosen setting. Others lack accurate 3D ground truth or quantified image-to-geometry alignment, limiting their use for metric surface evaluation. We introduce Martine, a benchmark for controlled 3D object surface reconstruction. Martine comprises 28 objects, including transparent materials and fine structures, captured at \(9568 x 6376\) pixels from 84 views over a spherical cap under both single-light and multi-light regimes. It provides object-independent camera calibration, accurate 3D ground-truth scans, calibration and image-to-geometry accuracy quantified on test images, and 3D property masks for analysing errors by material properties, visibility, curvature and fine structures. Experiments across photogrammetry, classical multi-view stereo, neural implicit surfaces, Gaussian-splatting-based reconstruction, and multi-view photometric stereo show that Martine enables regime-aware evaluation with explicit control over reference-data accuracy. The benchmark resolves sub-millimetric reconstruction differences, with a best observed mean one-sided Chamfer distance of \(0.063\) mm over the evaluated visible surface.
Show more
ThousandWorlds: A benchmark for emulating the climates of potentially habitable exoplanets
Edward Stevenson ⋅ Mei T Mak ⋅ Eric Wolf ⋅ Denis E Sergeev ⋅ Tobi Hammond ⋅ Nathan Mayne ⋅ Miles Cranmer
The search for life beyond Earth will depend on detecting faint signatures in the atmospheres of potentially habitable exoplanets. Interpreting those signatures requires understanding the host planet's climate: the same molecule may signal life on one planet and abiotic chemistry on another. Global climate models (GCMs) provide this understanding, but individual runs can require up to millions of core-hours and substantial domain expert time. Machine-learning emulators could remove this bottleneck, but progress has been limited by the absence of a curated, multi-model exoclimate dataset. We introduce ThousandWorlds, an ML-ready benchmark for exoclimate emulation and for the broader regime of low-data, multi-simulator, parameter-to-field regression. The dataset contains approximately 1,700 simulations from five GCMs, mapping eight planet parameters to 3D atmospheric fields including temperature, humidity, winds, clouds, and radiation. Three nested subsets define progressively harder challenges: single-simulator regression, multi-simulator regression with complete observations, and multi-simulator regression with structured missingness. We propose two evaluation protocols: one for ranking methods, and one that measures performance relative to the disagreement between GCMs themselves. We evaluate ten baselines spanning simple methods, trees, deep learning, and Gaussian processes. GP-based methods perform best, suggesting that ThousandWorlds exposes a regime where off-the-shelf deep learning does not yet succeed. Data: https://doi.org/10.57967/hf/8695. Code: https://github.com/astroautomata/ThousandWorlds.
Show more
SynthForensics: Benchmarking and Evaluating People-Centric Synthetic Video Deepfakes
Roberto Leotta ⋅ Salvatore A Sambataro ⋅ Claudio V Ragaglia ⋅ Mirko Casu ⋅ Yuri Petralia ⋅ Francesco Guarnera ⋅ Luca Guarnera ⋅ Sebastiano Battiato
Modern T2V/I2V generators synthesize people increasingly hard to distinguish from authentic footage, while current evaluation suites lag: legacy benchmarks target manipulation-based forgeries, and recent synthetic-video benchmarks prioritize scale over realistic human depiction. We introduce SynthForensics, a people-centric benchmark of $20{,}445$ videos from 8 T2V and 7 I2V open-source generators, paired-source from FF++/DFD reals, two-stage human-validated, in four compression versions with full metadata. In our paired-comparison human study, raters prefer SynthForensics in $71$--$77\%$ of head-to-head comparisons against each of nine existing synthetic-video benchmarks, while facial-quality metrics fall within the FF++/DFD baseline range. Across 15 detectors and three protocols, face-based methods drop $13$--$55$ AUC points (mean $27$) from FF++ to SynthForensics and a further $23$ under aggressive compression; fine-tuning closes the gap at a backward cost on legacy benchmarks; re-training shows synthetic and manipulation features largely disjoint for most detectors. We release dataset, pipeline, and code.
Show more
GISA: A Benchmark for General Information-Seeking Assistant
Yutao Zhu ⋅ Xingshuo Zhang ⋅ Maosen Zhang ⋅ Jiajie Jin ⋅ Liancheng Zhang ⋅ Xiaoshuai Song ⋅ Kangzhi Zhao ⋅ Wencong Zeng ⋅ Ruiming Tang ⋅ Han Li ⋅ Ji-Rong Wen ⋅ Zhicheng Dou
The advancement of large language models~(LLMs) has significantly accelerated the development of search agents capable of autonomously gathering information through multi-turn web interactions. However, existing benchmarks often construct queries backward from answers, producing tasks misaligned with real-world needs. Moreover, these benchmarks tend to focus on either locating specific information or aggregating information from multiple sources, while relying on static answer sets prone to data contamination. To bridge these gaps, we introduce GISA, a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries reflecting authentic information-seeking scenarios. GISA features four structured answer formats (item, set, list, and table), enabling deterministic evaluation. It integrates both deep reasoning and broad aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization. Notably, GISA provides complete human search trajectories for every query, serving as valuable references for analyzing and improving search agent behavior. Experiments on mainstream LLMs and commercial search products reveal that even the best model achieves only 19.30% exact match, with performance notably degrading on tasks requiring complex planning and comprehensive information gathering.
Show more
Scaling Genomic Language Modeling with Unified Corpus and Evaluation in Bacteria
Maciej Wiatrak ⋅ Samuel T Horsfield ⋅ Aaron Weimann ⋅ Ramon Viñas Torné ⋅ Maria Ntemourtsidou ⋅ David Abelson ⋅ Adam Dinan ⋅ John Lees ⋅ Maria Brbic ⋅ Andres Floto
Progress in genomic language modeling depends on pretraining resources that are both diverse and comprehensive, as well as benchmarks that evaluate models across biological scales from genes to whole genomes. This need is especially acute in bacteria, which are extremely abundant, diverse, and central to human health, ecology, and biotechnology. Yet resources for large-scale pretraining and evaluation in bacterial genomics remain limited, with existing frameworks focusing largely on eukaryotic settings. Here, we introduce BacCorpus, a genome-level resource for training genomic LMs comprising 20B proteins, including 4.6B unique sequences, from 7M genomes spanning more than 150,000 species, and BacBench, a comprehensive benchmark spanning 10 datasets and 5 tasks across gene-, system-, and genome-level tasks. We evaluate a diverse suite of genomic LMs on BacBench and additionally develop BacLM, a 350M-parameter model trained jointly on protein sequences and intergenic DNA to further investigate how bacterial genomes are best represented. Across tasks, we find that performance depends strongly on biological scale: methods leveraging protein sequence are competitive on gene-level prediction, whereas models that incorporate wider genomic context excel at problems which involve complex multi-gene interactions. Together, BacCorpus and BacBench provide a practical foundation for training and evaluating robust, genome-aware genomic language models in bacteria at an unprecedented scale.
Show more
JAXtari: High-Throughput and Easy-to-Modify Arcade Learning Environment
Quentin Delfosse ⋅ Raban Emunds ⋅ Paul Seitz ⋅ Sebastian Wette ⋅ Jannis Blüml ⋅ Daniel Kirn ⋅ Dominik Mandok ⋅ Kristian Kersting
The Arcade Learning Environment (ALE) has been a cornerstone of Deep Reinforcement Learning research, providing a standardized benchmark for evaluating general agents. However, its lack of native support for object-extraction and environment modification restricts research into agent generalization and reasoning.Furthermore, the ALE's dependence on CPU-based emulation creates a significant bottleneck in modern GPU-accelerated training pipelines, limiting both experimental scale and iteration frequency. In this work, we introduce JAXatari, an open-source, high-performance reimplementation of Atari 2600 environments. By leveraging massive parallelization on GPUs, PPO agents finish training with 100M frames on a single A100-GPU in under an hour. Beyond speed, JAXatari provides object-centric observations and a flexible modification system, allowing researchers to, e.g. develop neuro-symbolic methods or discover novel shortcuts and subsequently train and adapt their agents rapidly, sometimes in only minutes.
Show more
Towards Understanding and Measuring Cognitive Atrophy in LLM Behaviour
Abeer Badawi ⋅ Moyosoreoluwa Olatosi ⋅ Negin Baghbanzadeh ⋅ Laleh Seyyed-Kalantari ⋅ Frank Rudzicz ⋅ R. Shayna Rosenbaum ⋅ Sara Pishdadian ⋅ Elham Dolatabadi
Recent incidents involving LLMs used for mental-health support reveal a critical evaluation gap: surface-level safety scores do not capture how models behave across realistic, emotionally sensitive interactions over time. Existing benchmarks measure knowledge, safety, or static response quality, but miss whether LLM interactions help users keep reflecting, coping, and making decisions themselves. We formalize this missing dimension as Cognitive Atrophy, a process-level behavioural pattern in AI-mediated mental-health support distinct from safety and helpfulness. To measure it, we introduce Cognitive Atrophy Bench, a clinically grounded benchmark built from 1,576 fully human-generated counseling conversations, 15,680 turns, and 42,230 responses from five LLMs. Three clinical psychology experts developed a 20-attribute schema spanning user context, response behaviour, and global risk flags; six trained clinical reviewers applied it with span-grounded evidence, producing 5,324 reviewer judgments. We further introduce the User-Input Risk Index (UIRI), the Cognitive Atrophy Risk Index (ARI), and trajectory summaries. Across five LLMs, models show a consistent moderate-to-high level of atrophy-aligned behaviour across single- and multi-turn settings. While models generally respond to overt safety cues, they adapt less reliably when users seek solutions or decisions. The dominant recurring patterns are directive advice, problem-solving, recommendation-heavy responses, topic shifts, and forms of validation that may reinforce dependence rather than reflection. Our work makes Cognitive Atrophy measurable and provides a foundation for auditing model behaviour in sensitive LLM conversations. All code and data are released.
Show more
Penalize to Verify: A Multi-Domain Benchmark for Trustworthy LLM Evaluation
Jia-chen Zhang ⋅ Zheng Zhou ⋅ Zhuo Wang ⋅ Qi Jia ⋅ Zijian Chen ⋅ Jie Zhou ⋅ Kaiwei Zhang ⋅ Xin Li ⋅ Guangtao Zhai ⋅ Liang He
As Large Language Models are increasingly deployed as automated evaluators (LLM-as-a-Judge), ensuring the reliability of their judgments has become a critical challenge. Existing benchmarks primarily focus on human-model agreement, largely overlooking the evaluators' robustness against deceptive contexts, their inherent self-evaluation biases, and their pervasive tendency toward overconfident hallucination. To comprehensively evaluate the model's verification capability, we propose Penalize-to-Verify Benchmark (PVBench), a comprehensive multi-domain benchmark (encompassing Math, Code, and Knowledge) designed to rigorously stress-test the validation capabilities of LLMs. We propose a novel semi-automated framework to construct paired test cases featuring both correct and highly plausible misleading Chain-of-Thought trajectories, naturally derived from the errors of peer models. Furthermore, to mitigate blind guesses, we introduce a tiered penalty metric that rewards accurate verification, heavily penalizes incorrect judgments, and assigns zero score to abstentions, thereby forcing models to calibrate their true confidence. Extensive experiments reveal that state-of-the-art LLMs suffer catastrophic performance degradation when confronted with misleading CoTs, exhibit strong self-judge biases, and demonstrate distinct verification vulnerabilities across different domains. By shifting the evaluation paradigm from simple accuracy to risk-aware verification, our benchmark provides a scalable and rigorous foundation for developing trustworthy LLM evaluators. Our benchmark is available here: https://anonymous.4open.science/r/PVBench-CC63.
Show more
Aligning Language Model Benchmarks with Pairwise Preferences
Marco Gutierrez ⋅ Xinyi Leng ⋅ Hannah Chen ⋅ Jonathan Richard Schwarz ⋅ Ahmed Alaa ⋅ Tom Hartvigsen
Language model benchmarks are pervasive and computationally-efficient proxies for real-world downstream performance. However, many recent works find that benchmarks often fail to predict downstream utility. While some works have begun diagnosing sources of misalignment, there remain no ways to systematically update benchmarks to align their scores with downstream usage. Towards bridging this gap, we introduce and study \textit{benchmark alignment}, where we use information about downstream model performance to automatically update benchmarks, aiming to produce new static benchmarks that predict model pairwise rankings on downstream tasks. Our experiments involving 4576 language models and 6 benchmarks show that benchmark items can successfully be reweighted to predict downstream performance for unseen models, even generalizing across model scales in most cases. And while naive alignment unsurprisingly requires large numbers of models and benchmark questions, we find that proper use of downstream data can often also enable alignment, even using responses from as few as 20 models. Overall, our work takes a step towards efficiently aligning benchmark development with downstream tasks.
Show more
Can an LLM Reason Like a Lawyer? Benchmarking the ability of LLMs to map the facts of a case to the elements of the applicable legal rule
Shounak Paul ⋅ Seungeon Lee ⋅ Christoph Engel ⋅ Krishna Gummadi
Legal judgment prediction has made substantial progress in recent years, yet existing work evaluates whether models reach correct conclusions rather than whether they reason correctly. The core of legal decision-making is the mapping of case facts to the elements of an applicable legal provision, a step that prior benchmarks neglect. We introduce the first benchmark for evaluating legal reasoning, built on 375 judgments of the European Court of Human Rights (ECtHR) discussing Art. 11 ECHR on freedom of assembly. We exploit that the court routinely splits the decision in a stage reporting the facts of the case, and an evaluation stage in which it uses these facts selectively for justification. We give the (preprocessed) list of facts to 10 downstream LLMs and evaluate their justifications. The pipeline is validated by three expert legal annotators. Downstream models are reasonably accurate in predicting whether Art. 11 has been violated, and which elements of doctrine are critical, but perform substantially worse at attributing specific facts to doctrinal elements. Essentially they make the right decision, but for the wrong reasons. This weakness is invisible under standard evaluation metrics. Our benchmark, pipeline, and evaluation framework are released. They can easily be adapted to any legal rule that follows the Civil law tradition, i.e. to legal reasoning that requires the mapping of facts to an abstract legal rule.
Show more
ASO Atlas 2.0: Evaluating antisense oligonucleotide prediction across the preclinical pipeline
Barney Hill ⋅ Nicola Whiffin ⋅ Carlo Rinaldi ⋅ Stephan J Sanders
Predictive models have the potential to transform drug development by increasing success rates across all stages of development pipelines. Evaluating this potential needs to include the whole pipeline, from initial screens of thousands of compounds to the expensive assays, such as toxicity assessment, focused on a few compounds in the later stages. Genetic medicines, such as antisense oligonucleotides (ASOs), could benefit millions affected by individually rare genetic disorders, if economic limitations imposed by the need to screen hundreds of sequences at considerable expense can be overcome. To enable systematic analysis of this problem, we present ASO Atlas 2.0, the largest public multi-endpoint ASO preclinical dataset, spanning 295,007 data points, including both in vitro efficacy and in vivo toxicology, across 430. Using this resource, we introduce a cost-based evaluation framework that translates per-stage classifier enrichment into projected cost savings under a representative preclinical pipeline model. We find that incorporating model predictions across all stages reduces projected pipeline cost by 71%. This saving is largely attributable to modest enrichment at costly late-stage in vivo studies rather than strong enrichment at lower-cost in vitro assays, reversing the model ranking suggested by accuracy alone and highlighting the need for improved toxicology modelling.
Show more
TrialAgentBench: Evaluating AI Agents for Clinical-Trial Analysis and Long-Horizon Drug-Development Decisions
Bradley M Segal ⋅ William J Bolton ⋅ Philip Torr ⋅ David Clifton ⋅ Lei Clifton
Clinical drug development is high stakes. Small statistical errors propagate into trial design, participant exposure, and stop/advance decisions. As agents are increasingly used to support analysis and decision-making, more robust evaluations of their capabilities in clinical domains are needed. We introduce TrialAgentBench, a public benchmark in which agents must analyse participant-level evidence, choose defensible estimands and methods, quantify uncertainty, diagnose assumption failures, and carry those analysis decisions into downstream drug-development decisions. TrialEvalBench contains 400 patient-level analysis tasks across design, assumption, and context tiers. TrialDevBench contains 20 sequential development programmes across five scenario worlds, each starting from a confounded 10,000-patient observational extract and proceeding through asset nomination, endpoint and sample-size design, deterministic phase trials, phase analysis, and stop/advance decisions. The release spans 420 evaluated tasks or programmes, 528,660 simulated subjects, 77.6 million public tabular records, and 1.46 billion row-variable cells. Agents see only participant-facing data; grader-only files contain cell-appropriate targets, deterministic phase state, and scoring code. On TrialEvalBench, GPT-5.4 completes all 400 items but matches the required estimand/method class (e.g., log-HR versus RMST or risk-difference targets) for only 215/400 and produces an acceptable next-trial sample-size plan for 161/400. On TrialDevBench, GPT-5.4 and Claude Opus 4.7 complete all 60 seeded development programme runs per model, yet frequently nominate the wrong initial asset under confounding and choose evidence-supported stop/advance timing in fewer than one-third of programmes. Together, the two suites make clinical-trial decision quality measurable. Current agents can effectively execute familiar analyses, but fail to make the decisions those analyses exist to support.
Show more
BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors
Jonas Ricker ⋅ Asja Fischer ⋅ Erwin Quiring
Given the surge of harmful AI-generated imagery online, reliably distinguishing authentic images from generated ones has become an urgent research topic. While many proposed detection methods perform well under controlled settings, they often collapse when tested on real-world data. A potential root cause are subtle biases in the detectors' training data. As a result, detectors may rely on spurious correlations instead of learning true forensic artifacts. While a recent line of work has identified the problem, there is not yet an established protocol to evaluate how biased a detector actually is. In this work, we therefore take a step back: First, we discuss what it means for a detector to be biased, and how this differs from a lack of robustness. Second, we propose BIAS-ID, a transparent framework for analyzing and quantifying the presence of transformation biases in AI-generated image detectors. We validate our framework by performing an evaluation of six detectors across two datasets, revealing that several state-of-the-art detection methods are strongly affected by biases. Our results highlight the importance of bias-aware evaluation for developing reliable AI-generated image detectors.
Show more
Understanding the Interplay between Memorization and Learning in Large Language Models
Bishwamittra Ghosh ⋅ Soumi Das ⋅ Qinyuan Wu ⋅ Mohammad Aflah Khan ⋅ Deepak Garg ⋅ Krishna Gummadi ⋅ Evimaria Terzi
We investigate foundational questions about the interplay between memorization and learning in large language models (LLMs). When an LLM generates a string, can we disentangle the roles of rote memorization and contextual learning? Rote memorization results in an LLM regurgitating specific details of a training string, while contextual learning leads to the generation of new strings that follow generalizable patterns of the underlying language. Related questions of interest include: can an LLM avoid memorization when optimally learning a language?, and if not, can we design training schemes to reduce memorization and improve learning? To address these questions, we propose contextual memorization, a new measure that precisely disentangles memorization from learning. Using this measure, we establish that different training strings are memorized at different epochs in training, and that memorization of some training strings is unavoidable when optimally learning a language. Finally, we propose memorization-aware training, a scheme that dynamically down-samples training strings based on their memorization, and in doing so, equalizes learning across strings, reduces memorization, and improves learning. Our conclusions are supported by extensive experiments on multiple LLMs using formal languages as a controlled testbed, enabling precise language specification, exact string sampling, and elimination of data contamination; and further validated on natural languages.
Show more
PASSAGE: A Real-Terrain Benchmark for Trustworthy Constrained Path Planning
Geoffrey Delhomme ⋅ François Trap ⋅ Fateh Kaakai
Machine learning is increasingly used for autonomous navigation, yet safety regulations governing safety-critical AI applications require rigorous datasets and protocols to evaluate learned planners on terrain-aware routing. Existing path-planning benchmarks address this only partially: synthetic grids offer algorithmic control without geographic realism, while Geographic Information System (GIS) resources provide real terrain without controllable obstacles or standardised baselines. To close this gap and steer research toward more trustworthy AI-based planners, especially in line with EASA and FAA guidance on AI, we introduce **PASSAGE**, an open-source benchmark and dataset that reframes constrained terrain path planning as a *controlled trustworthiness-oriented evaluation harness* rather than a static dataset. PASSAGE is sampled from a well-defined Operational Design Domain (ODD) built on JAXA ALOS AW3D30 elevation at its native 30 m/pixel resolution, and is derived into seven input-tensor resolutions from 64×64 to 4096×4096 pixels, totalling *4.4 million samples* for ≈ *2.43 TB* of compressed data. Each sample is a real-terrain cost grid annotated with start and goal markers, procedurally generated super-ellipse obstacles, and exact A* path labels under three physically interpretable cost models. The release pairs this dataset with a deterministic generation pipeline, a geographic hold-out, ODD-coverage diagnostics, and a rigorous evaluation protocol — a shift from *dataset collection* towards *auditable benchmark design* that provides a rare level of experimental control for studying robustness, generalisation, and other trustworthiness properties of safety-critical embedded AI. PASSAGE thereby offers a common surface for comparing AI-based path planners, including two dense-vision approaches we introduce as illustrative examples (path-mask segmentation and depth-estimation-inspired cost-to-go regression), with graph-neural and reinforcement-learning families as natural future extensions. Although motivated by aerospace trustworthiness requirements such as completeness, correctness, and representativeness, PASSAGE broadly supports trustworthy machine learning across robotics, autonomous navigation, and other safety-critical planning domains. Code (anonymous repository) and dataset (anonymous mirror) are released openly under permissive licences.
Show more
LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation
jun wang ⋅ Fengpeng Li ⋅ Hang Dong ⋅ Tianjin Huang ⋅ Wei Han
Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general land-cover recognition, lithology interpretation is a knowledge-intensive task that requires experts to infer rock types from various features, e.g., subtle visual, making reliable automated interpretation highly challenging. Geological knowledge-guided large multimodal models offer new opportunities, yet their evaluation remains constrained by the lack of benchmarks that capture lithological annotations, multi-level geological semantics, and expert-informed assessment. Here, we propose LithoBench, a multi-level benchmark for evaluating geological semantic understanding in remote sensing lithology interpretation. LithoBench contains 10,000 expert-annotated interpretation instances across 12 representative lithological categories, including 4,000 multiple-choice and 6,000 open-ended tasks organized into five cognitive levels: Identification and Description, Comparative Analysis, Mechanism Explanation, Practical Application, and Comprehensive Reasoning. We further develop an expert-in-the-loop, knowledge-grounded semi-automated construction pipeline, coupling multi sub-processes, e.g., structured geological image descriptions, to enhance geological validity and evaluation reliability. Experiments with multiple large vision-language models reveal substantial limitations in geological semantic understanding, particularly on higher-order explanation, application, and reasoning tasks. Taken together, LithoBench establishes a dedicated testbed for advancing geological knowledge-guided multimodal models, supporting systematic model evaluation, domain-specific training and adaptation, and knowledge-enhanced reasoning toward expert-level lithology interpretation from remote sensing imagery.
Show more
EEGFaceSem: An EEG Benchmark with Paired Generative Latents for Semantic Visual Modeling
Jun Ma ⋅ Michiel Spape ⋅ Keith M Davis ⋅ Tuukka Ruotsalo
Electroencephalography (EEG) is widely used to measure brain activity for brain-computer interface applications for its non-invasive nature and high temporal resolution. However, progress in modeling the neural responses to visual semantics has been limited by a reliance on static, real-world image datasets that lack the fine-grained control needed to isolate fine-grained changes in visual features and their relation to perception. In this work, we present EEGFaceSem, a large-scale dataset pairing high-quality EEG with photorealistic face images synthesized from a generative model. We provide the generative latent vector for each stimulus, enabling research into the continuous mapping between a visual latent space and neural representations. Each participant's brain responses were recorded while they attended to specific semantic features (e.g., smiling, hair color) in a rapid serial visual presentation paradigm. The dataset contains EEG signals from 30 participants, totaling 67,200 time-locked EEG recordings. We present benchmarks for single-trial semantic classification under single-subject, cross-subject, and subject-adapted settings, as well as for cross-subject voting that aggregates per-stimulus predictions across participants. We further demonstrate the dataset's unique utility with two complementary applications: deriving semantic directions in the latent space for brain-supervised image editing, and conditioning a generative model on brain-decoded labels for brain-supervised image generation. Our dataset paves the way for advancing brain-computer interface (BCI) devices for semantic-level visual saliency detection and brain-supervised generative modeling. Our dataset and code are openly released at OSF https://osf.io/2guk3/ and at HuggingFace https://huggingface.co/datasets/yefllower/EEGFaceSem.
Show more
SLAyiNG: A Diverse and Community-validated Dataset of Queer Slang
Leonor Veloso ⋅ Lea Hirlimann ⋅ Lucija M Zidar ⋅ Philipp Wicke ⋅ Valentin Hofmann ⋅ Hinrich Schuetze
Queer vernacular is rarely studied in NLP, despite advancements in resources and evaluation for other sociolects and informal language. Because of this, NLP systems often process queer language incorrectly, e.g., they misclassify it as hate speech or generate negative responses. To address this problem, we propose \textsc{Slaying}, the first real-world dataset of English queer slang. \textsc{Slaying} is community-validated, and includes over 400 queer slang terms that pertain to more than 20 queer subcommunities. We argue that queer language data resources have great potential in NLP -- e.g., as components of large pretraining corpora and as the basis for benchmarks -- and can improve queer users' experience of NLP systems. We leverage \textsc{Slaying} for two novel findings in support of this argument: (i) For a number of language models, we show that they are unbiased towards the queer community, but at the same time unable to process its language, i.e., absence of representation bias does not entail the absence of linguistic bias. (ii) Model performance on queer slang varies across queer subcommunities; it is generally worse for slang pertaining to African-American and Latine communities. These findings are relevant for both the queer NLP and the broader ML communities. \textsc{Slaying} is available to the public, and open to future revisions and extensions. \textbf{Warning:} This paper contains profane and potentially offensive language.
Show more
StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs
Pierre Le Jeune ⋅ Etienne Duchesne ⋅ Weixuan Xiao ⋅ Stefano Palminteri ⋅ Bazire Houssin ⋅ Benoît Malézieux ⋅ Matteo Dora
Multilingual studies of social bias in open-ended LLM generation remain limited: most existing benchmarks are English-centric, template-based, or restricted to recognizing pre-specified stereotypes. We introduce StereoTales, a multilingual dataset and evaluation pipeline for systematically studying the emergence of social bias in open-ended LLM generation. The dataset covers 10 languages and 79 socio-demographic attributes, and comprises over 650k stories generated by 23 recent LLMs, each annotated with the socio-demographic profile of the protagonist across 19 dimensions. From these, we apply statistical tests to identify more than 1{,}500 over-represented associations, which we then rate for harmfulness through both a panel of humans (N = 247) and the same LLMs. We report three main findings. \textbf{(i)} Every model we evaluate emits consequential harmful stereotypes in open-ended generation, regardless of size or capabilities, and these associations are largely shared across providers rather than isolated misbehaviors. \textbf{(ii)} Prompt language strongly shapes which stereotypes appear: rather than transferring as a shared set of biases, harmful associations adapt culturally to the prompt language and amplify bias against locally salient protected groups. \textbf{(iii)} Human and LLM harmfulness judgments are broadly aligned (Spearman $\rho=0.62$), with disagreements concentrating on specific attribute classes rather than specific providers. To support further analyses, we release the evaluation code and the dataset, including model generations, attribute annotations, and harmfulness ratings.
Show more
The Ruler and the Judge: Benchmark-Conditional Evaluation of LLM-as-a-Judge
Adrián Ghajari ⋅ Alejandro Benito-Santos ⋅ Víctor Fresno
LLM judges are increasingly used to rank systems and validate claimed improvements, but their validation often reduces to a single question: how well do judge scores correlate with aggregated human ratings? We argue that this question is too fragile. A judge can correlate with human averages while collapsing the rating scale, compressing meaningful margins, or confidently separating systems that humans do not distinguish. Furthermore, a correlation-based workflow always returns a leaderboard, even when the human benchmark cannot support one. We propose a benchmark-conditional measurement framework for LLM-judge validation. Stage 1 audits the human benchmark as a reference ruler using a many-facet ordered Rasch model, estimating rater/item fit, conditional precision, and supported pairwise resolution. Stage 2 calibrates each judge as a linked measurement instrument with judge-specific ordinal thresholds. Stage 3 evaluates out-of-sample system-level fidelity using normalized signed-margin fidelity, false-separation diagnostics, empirical transfer intervals, and distributional EMD. On SummEval and HiTZ-BASSE, the framework yields three outcomes: selection, warning, and refusal, depending on the ruler and judge pool. Calibrated predictions improve observable human-score distribution EMD in 57/81 judge–dimension cases, while the remaining failures are concentrated in the regions flagged by the ruler or judge-instrument audits. Together, these results instantiate benchmark-conditional LLM-judge validation as a protocol for deciding when judge-based claims are warranted, uncertain, or unsupported, addressing a central blind spot of correlation-based judge evaluation.
Show more
BO-Arena: An Evolving Benchmark for High-Dimensional Bayesian Optimisation
Thomas Christie ⋅ Colin Doumont ⋅ Henry Moss ⋅ Philipp Hennig
High-dimensional Bayesian optimisation (HDBO) is an increasingly crowded field, with new methods proposed at every major conference, making it ever harder to answer a deceptively simple question: "Which method is currently state-of-the-art?" In this paper, we question the efficacy of benchmarking practices in the field, and find them to be systematically ineffective. Specifically, multiple published works fail to benchmark against the strongest baselines available at the time of writing, or even include incorrect implementations of key methods. As a remedy, we introduce BO-Arena, an actively maintained and extensible package providing canonical implementations of state-of-the-art algorithms, designed to streamline building new methods and rigorously benchmark against existing approaches. We use the package to ask what currently drives performance in HDBO, and find that the answer is even simpler than previously thought. Following these simple principles, we propose a new algorithm which achieves state-of-the-art performance on problems with large numbers of observations, outperforming considerably more complex recent methods. Finally, we question whether there is currently sufficient evidence to suggest that a simple Gaussian process baseline is outperformed by any more complex methods for HDBO.
Show more
The Smart Buildings Control Suite: A Diverse Open Source Benchmark to Evaluate and Scale HVAC Control Policies for Sustainability
Judah Goldfeder ⋅ Victoria Dean ⋅ Zixin Jiang ⋅ Xuezheng Wang ⋅ Bing Dong ⋅ Hod Lipson ⋅ John Sipple
Commercial buildings account for 17% of U.S. carbon emissions, with roughly half of that from Heating, Ventilation, and Air Conditioning (HVAC). HVAC devices form a complex thermodynamic system, and while model predictive control and reinforcement learning have been used to optimize control policies, scaling to thousands of buildings remains a significant unsolved challenge. Most current approaches are over-optimized for specific buildings and rely on proprietary data or hard-to-configure simulators. We present the Smart Buildings Control Suite, the first open source interactive HVAC control benchmark with a focus on solutions that generalize across building. It has 3 components: real-world data from 11 buildings over 6 years, a lightweight data-driven simulator for each building, and a modular Physically Informed Neural Network (PINN) building model as a simulator alternative. The buildings span multiple climates, management systems, and sizes, and both the simulator and PINN easily transfer to new buildings, ensuring solutions using this benchmark are robust to these factors and only reliant on fully scalable building models. This represents a major step towards scaling HVAC optimization from the lab to buildings everywhere. To facilitate use, our benchmark is compatible with the Gym standard, and our data is part of TensorFlow Datasets.
Show more
Mouse Total Capture: A multi-view Dataset for 3D Motion and Expression Capture of Freely Moving Mouse
Jiachen Zhao ⋅ He Zhang ⋅ Liang An ⋅ Tao Yu
3D motion and expression capture of laboratory mice is a critical task in behavioral neuroscience, yet it remains underexplored in the computer vision community. Existing studies have focused narrowly on either body pose estimation of freely moving mice or facial expression capture under head-fixed conditions, leaving a significant gap: no dataset supports simultaneous capture of whole-body motion and expression in a freely moving mouse. To fill this gap, we present PanoMouse, a multiview system and dataset for Mouse Total Capture (MTC). The PanoMouse system deploys 24 cameras across three horizontal layers, achieving 360-degree photographic coverage of the mouse. Built upon this system, the PanoMouse dataset collects over 3.4 million frames, with 7,293 frames annotated with 92 whole-body keypoints, totaling approximately 670,000 keypoint annotations. To our knowledge, PanoMouse is the first dataset to provide fine-grained annotations of a mouse in natural behavior. We further propose 4D Triangulation (Triang4D), a test-time optimization method for 3D MTC that jointly enforces multi-view consistency, temporal smoothness, and bone length constraints. Applying Triang4D to the entire dataset produces 3D whole-body keypoint sequences across all recording sessions, enabling downstream skeleton-based behavior analysis. We benchmark baseline methods on PanoMouse across three tasks: 2D pose estimation, 3D pose estimation, and behavior classification, opening a new paradigm for fine-grained behavior analysis and bridging computer vision with behavioral neuroscience.
Show more
Cephalonauts One: A deep fMRI dataset for decoding naturalistic speech in the human brain
Antoine Collas ⋅ Louis Jalouzot ⋅ Géraud Ilinca ⋅ Corentin Caris ⋅ Romain Valabregue ⋅ Ahmed Hassayoune ⋅ David Gonçalves ⋅ Madeleine Hueber ⋅ Thaddée Delebarre ⋅ Savatovsky Julien ⋅ Clara Fonteneau ⋅ Charles Maussion ⋅ Bertrand Thirion ⋅ Alexis Thual
Cephalonauts One is a whole-brain 3 Tesla (3T) functional magnetic resonance imaging (fMRI) dataset recorded while subjects listened to audio podcasts. Three healthy subjects underwent multiple scanning sessions, each consisting of five 15-minute runs, while listening to podcasts in their native language. The current release contains 14, 20, and 17 hours of fMRI data per subject, respectively, making it one of the deepest per-subject fMRI datasets with naturalistic speech stimuli available. The dataset pairs brain activity with the corresponding podcast audio, transcript annotations, and derived stimulus embeddings. Furthermore, we introduce a brain decoding benchmark formulated as audio segment retrieval: given fMRI activity from a held-out session, the decoder must identify the corresponding time-aligned podcast audio segment among candidate segments. We provide standardized splits, evaluation metrics, and baseline decoders for this task. Finally, a scaling analysis shows that decoding performance improves continuously with the amount of training data per subject. Data & code: https://huggingface.co/datasets/karavela/cephalonauts_one
Show more
ObjView-Bench: Rethinking Difficulty and Deployment for Object-Centric View Planning
Sicong Pan ⋅ Hao Hu ⋅ Xuying Huang ⋅ Benno Wingender ⋅ Maren Bennewitz
Object-centric view planning is a core component of active geometric 3D reconstruction in robotics, yet existing evaluations often conflate object complexity, planning difficulty, budget assumptions, and physical reachability constraints. As a result, conclusions drawn from idealized view-planning evaluations may not reliably predict performance under realistic reconstruction settings. We introduce ObjView-Bench, an evaluation framework for rethinking difficulty and deployment in object-centric view planning. First, we disentangle three quantities underlying view-planning evaluation: omnidirectional self-occlusion as an object-side attribute, observation saturation difficulty, and protocol-dependent planning difficulty defined through a set-cover formulation. This separation supports controlled dataset construction, analysis of slow-saturation objects, and a case study showing that planning difficulty-aware sampling can improve learned view planners. Second, we design deployment-oriented evaluation protocols that reveal how budget regimes and reachable-view constraints alter method behavior. Across classical, learned, and hybrid planners, ObjView-Bench shows that difficulty, budget, and reachability constraints substantially change method rankings and failure modes.
Show more
PurpleAudit: A Co-Evolutionary Auditing Framework for Task Hijacking in Multi-Agent Systems
Tristan Bilot ⋅ Zhilu Zhang ⋅ Kay Liu ⋅ Mikhail Kuznetsov ⋅ Wei Ding
Multi-agent systems (MASs) are increasingly deployed for complex tasks, yet their security against indirect prompt injection remains poorly understood. As frontier LLMs become more robust to injections with overtly malicious semantics, such as credential exfiltration, unauthorized transactions, or policy bypass, the residual attack surface shifts to task hijacking: an injection that silently redirects an MAS toward an attacker-chosen alternative task plausibly within the agent's domain, such as the wrong patient's record, a competitor's product, or a different CVE. There is no malicious verb to refuse, and existing benchmarks, designed around overtly malicious goals in single-agent settings, do not characterize this surface. We introduce PurpleAudit, an auditing framework that lets a practitioner stress-test their own MAS against task hijacking and derive a tailored system-prompt defense. The framework couples 22 baseline attacks and four prompt/tool-level defense baselines from the literature with a co-evolutionary purple-team procedure, in which red- and blue-team agents evolve stronger attacks and adaptive defenses by reasoning over each other's execution traces. Applied across frontier backends on ten MASs and a real-world MAS with live web access, PurpleAudit surfaces an MAS-specific attack pattern that frames the injection as a prerequisite for completing the user's task, reaching 31.26% ASR on Claude Opus 4.5 under the framework's auditing protocol, where 22 baseline attacks fail entirely. The co-evolved defense suppresses both baseline and evolved attacks at negligible utility cost, transfers across reasoning-capable backends, and operates at the prompt level with low deployment overhead.
Show more
IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs
Florian Carichon ⋅ Soumya Sharma ⋅ Meaghan J. Girard ⋅ Romain Rampa ⋅ Golnoosh Farnadi
Large language models (LLMs) are increasingly used for tasks involving creative problem solving and idea generation. However, However, there is a lack of consensus concerning their creative capabilities:: some studies report superior performances compared to humans, while others highlight structural limitations such as fixation and the homogenization of outputs. Existing evaluation approaches either rely on narrow, decontextualized tasks that do not capture goal-oriented generation or on broader settings that confound multiple aspects of the creative process, making it difficult to isolate the effects of task formulation, prompting, and evaluation design. Significantly, the role of structured prompting strategies in shaping idea generation remains underexplored. Therefore, we introduce IDEAFix, an evaluation framework for analyzing divergent thinking in open-ended idea generation tasks. We prompt models to generate multiple original solutions to controlled variations of short design scenarios, task attributes, and defixation prompting strategies. This design enables systematic analysis of how structured guidance influences LLMs' idea generation. Our results show that both task formulation and attribute selection significantly affect models' performance, and that simple prompting strategies can boost the originality of solutions. However, we also observe persistent output homogenization across models, confirming inherent limits in their ability to generate diverse solutions. Overall, IDEAFix provides a controlled, extensible framework for studying the mechanisms underlying LLMs' creativity.
Show more
Principia: Relational Physics Tests for Video Models
Varun Varma Thozhiyoor ⋅ Shivam Tripathi ⋅ Venkatesh Babu Radhakrishnan ⋅ Anand Bhattad
Evaluating physical reasoning in video models is difficult because absolute motion measurements depend on frame rate, object scale, and camera calibration, all of which are often ambiguous or unavailable in generated video. We propose a different approach. When two objects in the same scene obey the same physical law, their motions must satisfy predictable relationships, and these relationships hold independent of calibration. We introduce Principia, a benchmark that evaluates Newtonian physics through relational consistency between paired objects. Principia spans eight phenomena—gravity, restitution, friction, rotational inertia, projectile motion, momentum, pendulum, and mass-spring oscillation—across translational, rotational, collisional, and oscillatory dynamics, using real-world scenes recorded under controlled protocols. We also introduce a calibration-independent consistency score that quantifies physical violation directly in image space. Across thousands of generations from five state-of-the-art video generators, no model exceeds 0.42 on Principia despite all scoring above 0.7 on VBench. Vision-language models show inverse failure profiles, succeeding where generators fail and vice versa.
Show more
TerraMesh-Masks: Open‑Vocabulary Segmentation for Earth Observation
Benedikt Blumenstiel ⋅ Hugues Devimeux ⋅ Johannes Jakubik ⋅ Konrad Schindler
Open‑vocabulary segmentation (OVS) enables models to segment a variety of concepts specified in natural language, removing the need for fixed label sets and fine-tuning. While recent OVS approaches have shown strong performance on natural images, they transfer poorly to Earth observation (EO) data due to differing semantics, scales, sensing modality, and geographic context. To date, large‑scale EO‑specific datasets and benchmarks for OVS remain scarce. We, therefore, introduce TerraMesh-Masks, a large-scale open-vocabulary segmentation dataset for EO, comprising 16 million binary masks aligned with nine million multimodal TerraMesh samples and annotated using OpenStreetMap and Overture-derived semantics. In addition, we provide an expert-reviewed evaluation set of 523 samples covering over 200 classes to enable robust evaluation. To demonstrate and analyze the impact of TerraMesh-Masks, we build an open-vocabulary version of TerraMind. Our results demonstrate significantly improved performance over general-purpose state-of-the-art models reaching 35.15% IoU versus 16.29% for DINOv3.txt and 11.33% for SAM-3. We release the dataset publicly under a permissive license at https://huggingface.co/datasets/ibm-esa-geospatial/TerraMesh-Masks.
Show more
WayPOP: A Panoramic Open-Set Panoptic Tracking Benchmark
Rohit Mohan ⋅ Maximilian Luz ⋅ Swarnava Chowdhury ⋅ Yao Lu ⋅ Florian Drews ⋅ Thomas Nürnberg ⋅ Yakov Miron ⋅ Federico Tombari ⋅ Stefano Gasperini ⋅ Abhinav Valada
Autonomous driving perception increasingly relies on panoramic multi-camera setups for holistic, spatiotemporal scene understanding. However, existing panoptic tracking benchmarks assume a fixed semantic taxonomy, failing to evaluate whether systems can robustly detect, segment, and track out-of-distribution (OOD) objects across time and multiple viewpoints. Creating a realistic open-set benchmark for this setting is challenging: fully synthetic data suffers from domain gaps, while 2D augmentations lack multi-view and temporal geometric consistency. To address this gap, we introduce WayPOP, the first benchmark for multi-view open-set panoptic tracking (MV-OSPT) in autonomous driving. WayPOP augments real-world sequences from the Waymo PVPS dataset with diverse 3D assets using a geometry-grounded pipeline. By leveraging actual LiDAR geometry, out-of-taxonomy objects are seamlessly integrated with realistic scales, multi-view projections, and depth-aware occlusions across five synchronized cameras and time. We formalize a structured task hierarchy for MV-OSPT and adapt established metrics to separately evaluate known and unknown regions for both segmentation and tracking. Our comprehensive benchmarking of representative baselines reveals that maintaining consistent unknown-object identities during camera transitions, viewpoint changes, and temporal occlusions remains a critical open challenge.
Show more
ReefNet: A Large-Scale Dataset and Benchmark for Fine-Grained Coral Reef Recognition
Abdulwahab Felemban ⋅ Yahia Battach ⋅ Faizan F Khan ⋅ Yuqian Fu ⋅ Xuhui Liu ⋅ Yesmeen Khattab ⋅ Yousef A Radwan ⋅ Xiang Li ⋅ Natalie Dunn ⋅ Susanne Bähr ⋅ Tullia I Terraneo ⋅ Fabio Marchese ⋅ Sara Beery ⋅ Burton H Jones ⋅ Francesca Benzoni ⋅ Mohamed Elhoseiny
Coral reefs are rapidly declining under anthropogenic pressures (e.g., climate change), creating an urgent need for scalable and automated monitoring. Progress in data-driven coral analysis, however, is constrained by the scarcity of large-scale datasets with fine-grained labels that are taxonomically consistent across sites and studies. To address this gap, we introduce ReefNet, a large-scale public coral reef image dataset with point-level annotations mapped to the World Register of Marine Species (WoRMS) taxonomy. ReefNet aggregates imagery from 76 curated CoralNet sources and an additional reef site from Al-Wajh (Red Sea), totaling approximately 925K genus-level hard coral annotations. Through expert-driven verification and targeted filtering, we derive a high-confidence benchmark subset with 92% expert agreement over 39 hard-coral label classes, enabling reliable evaluation under realistic label noise and strong class imbalance. Beyond dataset construction, we establish a comprehensive benchmark spanning zero-shot, cross-domain few-shot adaptation, within-source evaluation, and cross-source transfer to the Al-Wajh dataset. Experiments with state-of-the-art vision–language models (VLMs), multimodal large language models (MLLMs), and vision-only backbones reveal substantial degradation in zero-shot and extremely few-shot regimes, while adaptation with in-domain supervision yields large gains yet still leaves a persistent gap under cross-source shift and on long-tail genera. These results highlight fundamental challenges in applying general-purpose multimodal models to biodiversity monitoring and underscore the importance of large-scale, taxonomically grounded, high-quality datasets. ReefNet serves as both a benchmark and a training resource for advancing fine-grained coral reef understanding. ReefNet data and trained models are provided on Hugging Face, and code for accessing and benchmarking the data is provided on GitHub.
Show more
X-Palm: Paired Multispectral-to-Smartphone Dataset for Cross-Domain Palmprint Authentication
Seyed Jamal Seyedmohammadi ⋅ Pai Chet Ng ⋅ Angelo Genovese ⋅ Zhixiang Chi ⋅ Jeannie Lee ⋅ Konstantinos N Plataniotis
Palmprint modality offers a privacy-preserving biometric solution, yet its deployment is hindered by the domain gap between controlled enrollment and unconstrained authentication. Existing datasets are largely restricted to controlled setups and fail to capture the compound variability of real-world environments. In this paper, we introduce X-Palm, a cross-domain dataset comprising 6,006 palm images from 103 individuals (206 hands). To the best of our knowledge, X-Palm is the first palmprint dataset providing novel paired-identity acquisition specifically designed to bridge the gap between reliably controlled multispectral enrollment and unconstrained mobile authentication while encompassing a broad spectrum of in-the-wild variability. Unlike existing datasets that focus on single to a few variations, X-Palm addresses the massive modality and environmental shifts encountered in practical deployments by capturing paired data for identities across two distinct domains: (1) a controlled Multispectral Palmprint setting using our custom-developed scanner, and (2) an unconstrained smartphone palmprint setting that is participant-driven, incorporating simultaneous variations in hardware, hand pose, illumination, background, camera-to-hand distance, perspective, and palm surface conditions (e.g., moisture and occlusions). Our extensive benchmarks of 12 SOTA models reveal that while existing methods achieve high performance on controlled data, they experience severe performance collapse on X-Palm. Conversely, models trained on X-Palm demonstrate consistent robustness across domains, positioning X-Palm as a valuable resource for training a model towards real-world, cross-domain generalization. Data access instructions and the related benchmarking codes are publicly available at: https://github.com/X-Palm/X-Palm-2026
Show more
FACBench: A Benchmark for Formal-Anchor Collisions in Multilingual Mathematical Grounding
Piotr Kuterba ⋅ Józef Spałek
Mathematical autoformalization can fail before proof generation, when an informal concept is grounded to the wrong Lean/Mathlib object. Multilingual terminology makes this step non-monotonic: aliases can recover missing concepts, but under collision stress they can also pull retrieval toward nearby, non-interchangeable anchors. We introduce FACBench, a targeted dependency-light benchmark and audit protocol for multilingual formal-anchor grounding before proof generation. FACBench combines a five-language concept inventory with controlled collision queries, Mathlib-graph stress tests, no-label ablations, theorem-like probes, and Lean-facing anchor checks. In controlled collision settings, naive multilingual alias union can place the expected anchor in the candidate list while ranking a nearby distractor first. A simple rule-based source/context guard improves top-1 anchor accuracy over naive multilingual retrieval by about 11 percentage points on controlled collisions and 28 percentage points on a Mathlib-graph-mined controlled holdout. Less scaffolded and theorem-like probes show where this guard stops helping and point to the need for role-aware grounding. FACBench provides reusable collision-focused evaluation and regression-test scenarios for retrieval-augmented autoformalization systems.
Show more
Doomed to Re-Annotate, Forever: The ImageNet Story
Illia Volkov ⋅ Nikita Kisel ⋅ Tetiana Mishkina ⋅ Klara Janouskova ⋅ Jiri Matas
Top-1 accuracy on ImageNet-1k remains the most universally reported metric in visual recognition, despite its well-documented label quality issues. In the paper, we describe the effort to obtain accurate and complete ImageNet-1k validation set annotations. The result -- ReImageNet -- is a from-scratch reannotation, which includes multilabel correction, object localisation, revised class definitions, and semantic attributes---text-recognition, rendition, reflection, crowd, and dominant. The reannotation reveals that $\approx13$\% of the original ImageNet-1k labels are incorrect, $32.7$\% of images are multilabel and $4.7$\% contain no object from an ImageNet-1k class. With the new labels, top-1 accuracy increases by $0.7$--$1.4$\% for supervised models and by $6$--$8$\% for MLLMs. We show that annotation errors in ImageNet-1k propagate into its derivative test sets, indicating that the problem is structural rather than specific to any single benchmark. We observed that LLMs outperform unaided crowd annotators, and that human and LLM collaboration with appropriate tooling represents the current quality ceiling for annotation at this scale. All [annotations](https://huggingface.co/datasets/c1rcuslegend/ReImageNet), [class definitions](https://gonikisgo.github.io/imagenet-annotation-preview/), [guidelines](https://gonikisgo.github.io/imagenet-annotation-preview/), and [analysis code](https://github.com/klarajanouskova/ImageNet) have been publicly released.
Show more
Instantiation of Human Values in Image Generation
Maria Teresa De Rosa Palmini ⋅ Piera Riccio ⋅ Noa Garcia ⋅ Eva Cetinic
Text-to-image (T2I) models are increasingly embedded in everyday visual culture, yet little is known about how they translate abstract human values into visual scenes. We address this gap by introducing a framework, grounded in Schwartz’s theory of basic human values, for studying value instantiation: the process by which ideals such as success, care, or freedom are visually realized. Using person-centered prompts from 70 positive and negative value items, we generate 28,000 images across four state-of-the-art T2I models. We identify visual prototypes, recurring combinations of subjects, actions, and settings, and use them to quantify how narrowly models instantiate these values and how this breadth compares with human descriptions of the same items. We then map these prototypes onto a taxonomy of social configurations spanning setting, sociality, mode of engagement, and social register, revealing what is systematically foregrounded or omitted. Our results show that T2I models produce far more concentrated representations than humans do when describing the same values. They default to a hyper-individualistic, adult-centric worldview, reproducing value-specific associations, such as achievement tied to formal institutional settings or hedonism rendered as affluent consumption, while marginalizing everyday labor, relational care, and non-adult perspectives. As a proof of concept, we show that these structural gaps can guide a steering method that diversifies outputs without modifying the original prompt.
Show more
SLVMBench: Skill Learning from Video Memory
Yudong Yang ⋅ Guangzhi Sun ⋅ Yixuan Li ⋅ Wei Li ⋅ Zejun MA ⋅ Chao Zhang
We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video large language models (video-LLMs) can learn skills from long video memory and apply them to real-time tasks. SLVMBench presents models with 2–3 hour video streams that contain a tutorial video embedded in a stream of arbitrary irrelevant videos, resembling real-world human learning practices. Video-LLMs are asked to apply the acquired skill to answer real-time questions about an ongoing video. Unlike long-video understanding benchmarks that emphasize passive comprehension and skill-learning benchmarks that rely on short, immediate demonstrations, SLVMBench tests the full pipeline of memorizing and extracting procedural knowledge, as well as transferring it to real-time tasks. Moreover, rigorous human annotations feature sub-second-level temporal calibration, manually engineered questions eliminating common-sense guessing, and collated tutorials to ensure coverage of the required skills. Evaluations on state-of-the-art proprietary and open-source video LLMs show that video-LLMs struggle substantially with learning and applying skill knowledge from videos. Moreover, performance degrades markedly when the skill knowledge is placed within a long video memory. These results reveal a key limitation of existing video LLMs and position SLVMBench as the first benchmark for studying real-time skill acquisition and application from long-context video memory.
Show more
GeoBiaset: A Counterfactual Benchmark for Demographic Bias in World-Level Geolocalization
Niccolò Niccoli ⋅ Silvia Dani ⋅ Marco Mistretta ⋅ Andrew Bagdanov ⋅ Marco Bertini ⋅ Lorenzo Seidenari
World-level image geolocalization is an increasingly important task, yet the biases of current models remain poorly understood. Prior work has identified coarse dataset imbalances noting, for instance, that some cities are absent from training sets and that models tend to perform better in wealthier regions. However no one has yet examined which visual cues drive these disparities. To do so, we introduce GeoBiaset, the first benchmark designed to measure how geolocalization models are influenced by the apparent race of a person inserted into the image foreground. The dataset consists of 9,541 images, pairing existing geotagged background images with counterfactual edits spanning seven races, holding the ground-truth coordinates fixed so that any shift in model predictions is attributable solely to the inserted subject. Alongside the benchmark, we introduce the Person-Origin Perturbation Indicators (POPI), novel metrics that quantify model resilience to these demographic perturbations by measuring whether predictions are pulled toward the region associated with the inserted group. Our evaluation of state-of-the-art geolocalization and multimodal large language models reveals a 16-47\% drop in continent-level performance when a subject from a demographically mismatched group is inserted. A landmark versus no-landamark analysis and gradient-weighted heatmaps show that models preferentially attend to geographic landmarks when present, but shift attention to facial features in their absence.
Show more
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Pierre Beckmann ⋅ Marco Valentino ⋅ Andre Freitas
Three paradigmatic forms of inference recur across scientific reasoning: deduction, induction, and causal abduction. Reliably evaluating LLMs on these in scientific settings is currently out of reach: scientific benchmarks built on human annotations are costly and lack mechanistic ground truth, while synthetic logical-reasoning benchmarks do not resemble real scientific documents. We introduce SciR, a benchmark that combines multi-paradigm reasoning with controllable scientific rendering, anchored on three paradigmatic scientific problems. Tasks are generated from formal objects (deduction tree, inductive rule hypothesis, causal graph) to guarantee verifiable answers, then rendered into multi-document scientific discourse via per-track domain-tuned genres. The construction lets us independently vary two difficulty axes: how hard it is to extract the key information needed for inference, and how hard the principled inference itself is. We test six models. Both axes hurt every model, and their effects compound. The rendering even hurts neuro-symbolic pipelines, which hand inference to a verified solver. The two axes yield a per-model extraction-vs-inference profile: for instance, reasoning models like deepseek-r1 mostly surpass non-reasoning instruct models on the inference axis. To our knowledge, SciR is the first multi-paradigm scientific-reasoning benchmark with parametric control on both extraction and inference difficulty.
Show more
Benchmarking Attention for Tabular Foundation Models
Maximilian Schambach ⋅ Clemens Biehl ⋅ Sam Thelin
Tabular in-context learners such as TabPFN, Mitra, or ConTextTab rely on alternating row and column attention over 2D sequences of latent embeddings. These attention patterns differ markedly from the one-dimensional case in language models: row attention involves longer sequences while column attention operates on much shorter ones, and the strided memory layout of tabular data makes producing contiguous tensors costly. Moreover, the hidden dimensions used in current models are small compared to recent language models. Yet, efficient attention has been studied mostly for one-dimensional sequences, leaving the two-dimensional tabular setting unexplored. To this end, we create a reproducible benchmarking setup and study the unique characteristics of tabular attention across several backends – Torch SDPA (efficient and cuDNN), FlashAttention-2/3/4, and the inference-only backends vLLM and SageAttention – measuring forward and backward throughput across realistic tabular shapes on three GPU generations (A100, H100, B200). We find that the optimal backend choice differs between column and row attention and varies across hardware as well as model specifics: While the FlashAttention implementations tailored for each GPU generation perform overall best, they are at times outperformed by cuDNN in the case of column attention at longer sequences with cross-over points depending on the head dimension. Among inference-only backends, SageAttention performs well for row attention and large sequences beyond 16k rows. Our reproducible benchmark lays the foundation for future improvements to table-native attention. The self-contained benchmarking and evaluation code is openly available at: https://github.com/SAP-samples/tabular-attention-benchmark
Show more
Honeyval: A Comprehensive Evaluation Framework for LLM-powered HTTP Honeypots
Mark Vero ⋅ Fabian Kaczmarczyck ⋅ Ivan Petrov ⋅ I Shumailov ⋅ Niels Heinen ⋅ Jamie Hayes ⋅ Tianqi Fan ⋅ Luca Invernizzi ⋅ Martin Vechev
LLMs increasingly serve as simulation engines for honeypots. They enable defenders to construct high-interaction honeypots with virtually no system security risks. However, LLM-powered honeypot development lacks a unified evaluation framework and protocol. Most evaluations consist of response similarity measurements against a fixed set of commands, manual testing, or online deployment. These evaluation methods are often not scalable for development, reproducible across evaluations, representative of practical attacks, and adaptable to various attacker and honeypot configurations. In this work, we bridge this gap and propose Honeyval, a comprehensive evaluation framework for LLM-powered HTTP honeypots. We address the limitations of prior evaluations by grounding the honeypots in 16 backend applications, using AI hacking agents as attackers, employing two control tasks to monitor agent and honeypot capabilities across customizations, and defining clear and verifiable exploit goals for the attacker. Using Honeyval, we conduct an extensive evaluation of recent cost-efficient LLMs as HTTP honeypots. Our experiments highlight the promise of LLM-powered honeypots; they lead to substantially longer interactions with the attacker than rule-based baseline honeypots and are far less frequently detected even by frontier models, all while preserving a running cost advantage against agentic attackers. Further, we experiment with different counter-offensive configurations for the honeypots, and observe unique trade-offs, such as longer interactions at the cost of increased detection.
Show more
Evaluating Epistemic Uncertainty: Beyond OOD Detection and Active Learning
Jakub Paplhám ⋅ Vojtech Franc ⋅ Willem Waegeman ⋅ Eyke Hüllermeier
Current evaluation of epistemic uncertainty relies on tasks such as out-of-distribution detection and active learning. However, the Bayes-optimal decision strategies for these tasks do not coincide with the scores commonly used to quantify epistemic uncertainty. Building on the epistemic reject-option framework, we evaluate epistemic uncertainty using its ability to identify regret, the reducible error. Formulating selective prediction as a constrained optimization over coverage, expected risk, and regret, we prove the optimal selector is a thresholded convex combination of the ground-truth aleatoric and epistemic uncertainties. This theoretical unification exposes a weakness in recent uncertainty disentanglement literature: we demonstrate that standard correlation metrics between learned components do not necessarily predict their actual operational utility. We instead propose to evaluate the achievable (risk, regret, coverage) surface of the decomposition as a diagnostic for joint disentanglement and utility. Benchmarking standard methods on datasets with dense human annotations reveals that decision-theoretic rankings can disagree substantially with proxy-task rankings, including pairwise rank inversions between methods that are top-ranked on one criterion and bottom-ranked on other.
Show more
Driving on Memory
Christian Löwens ⋅ Thorben Funke ⋅ Alexandru P P Condurache
End-to-end autonomous driving models learn to plan the vehicle's future trajectory from raw sensor observations. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives of the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but because they are historical, they do not reveal the current actors, signal states, or transient conditions. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should therefore be treated with caution. However, we do not observe the same behavior on Bench2Drive: driving from memory remains far from camera-based performance, suggesting that Bench2Drive better preserves interaction demands through longer closed-loop routes. To ensure reproducibility, we provide our code at anonymous.4open.science/r/MemoryDrivoR-E64C.
Show more
WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis
Shuo Lu ⋅ Yinuo Xu ⋅ Kecheng Yu ⋅ Siru Jiang ⋅ Yongcan Yu ⋅ Yubin Wang ⋅ Haitao Yang ⋅ Yuxiang Zhang ⋅ Bin Wang ⋅ Ran He ⋅ Jian Liang
Large language models (LLMs) are increasingly asked not only to write static interfaces, but to construct executable interactive worlds from natural language. Browser-native 3D, commonly built with Three.js, is a natural next frontier: generated programs must integrate assets, obey spatial and physical constraints, and keep user-facing controls synchronized with hidden runtime state. Existing web-generation benchmarks and evaluators, however, largely observe only pixels or DOM nodes, while the mechanics of a Three.js world unfold inside an opaque ``. We introduce WorldCoder-Bench, a benchmark for autonomous, physically grounded 3D world synthesis. WorldCoder-Bench contains 2,026 expert-curated tasks across Simulation, Rendering, and Application scenarios, with optional `.glb` assets and hidden behavioral contracts. We further propose StateProbe, an execution-based protocol that probes generated programs in a sandboxed browser and verifies hidden, mutation-hardened contracts over runtime states and transitions. Beyond verification coverage, we report Return on Automation and Time Efficiency Multiplier to measure correctness-adjusted cost and time savings. Across nine frontier models, the best system reaches only 27.8% verification coverage on WorldCoder-Core and 19.9% on WorldCoder-Rob, with failures dominated by state-schema drift and broken interaction chains rather than missing scene elements. Utility metrics further show that cheap or fast models can still provide substantial value on easier domains. WorldCoder-Bench is available at https://anonymous.4open.science/r/WorldCoder-Bench/.
Show more
CalArena: A Large Scale Post-Hoc Calibration Benchmark
Eugène Berta ⋅ David Holzmüller ⋅ Francis Bach ⋅ Michael Jordan
Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibration provides a simple and widely used solution, but the large number of proposed methods, combined with small-scale and inconsistent evaluations, makes it difficult to determine which approaches are truly effective in practice. We introduce a large-scale, standardized benchmark for post-hoc calibration, covering nearly 2000 experiments across tabular and computer vision tasks, including binary, multiclass, and large-scale classification settings. Our benchmark aggregates predictions from a diverse set of classical models, modern deep learning architectures, and foundation models, and provides unified, reproducible implementations of dozens of calibration methods within a common evaluation framework. We argue that Post-Hoc Improvement (PHI) in proper scoring rules offers a principled alternative to traditional calibration error estimators for comparing post-hoc methods, capturing both calibration quality and potential degradation to the model's predictive performance. Using this framework, we conduct the most comprehensive empirical study of post-hoc calibration to date. Our results reveal consistent patterns across domains: smooth calibration functions outperform binning-based approaches, dedicated multiclass methods are essential in high-dimensional settings, and generic machine learning models are not competitive without calibration-specific design. To facilitate future research, we release all data, code, and evaluation tools, providing a plug-and-play benchmark for developing and comparing calibration methods.
Show more
When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
Dean Ariel ⋅ Guy Laban
Conversational large language models can respond with warmth and reassurance while failing to advance the user's stated goal. We conceptualize this failure as $\textbf{goal displacement}$, the substitution of persona-maintenance content for goal-relevant substance. To measure it, we introduce $\textbf{GDB}$ (Goal Displacement Benchmark), comprising 230 scenarios across nine domains and $4{,}134$ human-scored responses from 18 models. A $2 \times 2$ taxonomy crosses hard constraints with emotional distress, distinguishing failures to confront infeasible goals or combine empathy with practical guidance from reverse displacement, where unsolicited problem-solving replaces requested emotional support. A deletion-test rubric assesses whether substantive help remains after purely supportive sentences are removed. Independent raters achieve weighted $\kappa = 0.82$ against author scores for displacement severity. Trap displacement ranges from 0.0\% to 37.7\% and is strongly negatively correlated with Chatbot Arena Elo ($\rho = -0.706$, $n = 15$). Reverse displacement reaches 41.9\%, including substantial rates in models with negligible trap displacement. A factorial probe supports a distinction from social sycophancy. Human-scored responses in the high-complexity, low-pressure condition exhibit 35.0\% sycophancy but 7.5\% displacement, with little association between the constructs. Matched OLMo-2 checkpoints show that mixed empathy-with-constraints displacement remains at 52.3\% after supervised fine-tuning and falls to 13.8\% after direct preference optimization. GDB provides a reproducible diagnostic of whether responses advance the operationalized user goal, supporting model comparison, post-training evaluation, and regression testing. We release the dataset, annotations, and rubric under CC-BY-4.0 and the evaluation code under MIT.
Show more
stable-worldmodel: A Platform for Reproducible World Modeling Research and Evaluation
Lucas Maes ⋅ Quentin Le Lidec ⋅ Luiz Facury ⋅ Nassim Massaudi ⋅ Ayush Chaurasia ⋅ Francesco Capuano ⋅ Richard Gao ⋅ Taj Gillin ⋅ Dan Haramati ⋅ Damien Scieur ⋅ Yann LeCun ⋅ Randall Balestriero
Research on world models is currently fragmented, with disparate codebases, data pipelines, and evaluation protocols hindering reproducibility and fair comparison. Current practice is further limited by three key bottlenecks: fragile one-off codebases, slow video data loading, and the lack of standardized generalization benchmarks. We present Stable-WorldModel (SWM), an open-source platform for standardized and reproducible world modeling research and evaluation. It delivers (1) a high-performance Lance-based data layer with native support and conversion tools for MP4, HDF5, and LeRobot datasets, (2) clean, well-tested implementations of modern world model baselines and planning solvers, and (3) a broad suite of environments and tasks extended with controllable visual, geometric, and physical factors of variation for systematic in-silico evaluation of dynamics understanding, control performance, representation quality, and out-of-distribution generalization. By unifying the full pipeline under a single, scalable framework, SWM dramatically reduces research overhead and accelerates trustworthy progress toward reliable world models.
Show more
M4Bench: Evaluating Procedural Specification for Clinical EHR Derivation Agents
Hannes Ill ⋅ Rafi Al Attrach ⋅ Rajna Fani ⋅ Ahram Han ⋅ Ryohei Yamamoto ⋅ Marlene L Moerig ⋅ Dukyong Yoon ⋅ Leo Anthony Celi
Clinical research over electronic health records depends on derived variables whose definitions sit outside the schema, in item codes, time windows, missingness rules, and output-grain choices that change cohorts and downstream conclusions. We introduce M4Bench, 28 clinical derivation tasks across 15 task families on MIMIC-IV and eICU, with controls that decompose the value of procedural context for AI coding agents into specification completeness, delivery format, and schema familiarity. A Docker-isolated Codex campaign isolates specification completeness as the primary driver: an operational spec over the schema-only baseline yields +0.162 [0.072, 0.260] on eight sentinel tasks, while clinician-reviewed skills add only +0.024 [-0.034, 0.099] on top, and the combined targeted-skill effect over no-skill is +0.136 [0.085, 0.191] across all 28 tasks (26/28 improving). Only 87/760 primary runs pass all strict diagnostics, with failures dominated by output-grain and temporal-boundary errors, and key F1 moves only +0.003 task-balanced, so the gain is partial reference agreement on matched keys, not autonomous task completion or row-grain repair. Schema restructuring preserves 0.704 of the with-skill gain on gpt-5.4-mini over the 26 MIMIC-IV tasks, bounding schema-name memorization at ≤ 29.6% of the effect; decoy-skill and raw-SQL controls further rule out generic-context and reference-code-structure-only explanations. Skills are released as a versioned, clinician-reviewable markdown delivery format, one packaging among several, with task-level reference-validity audit and per-run egress logs. M4Bench releases tasks, skills, evaluation code, and per-run audit logs as a fully open-source benchmark object designed for extension and reuse.
Show more
EditJudge-Bench: Auditing VLM Image-Edit Judges with Synthetic Ground Truth
Jens Parslov ⋅ Marco Schouten ⋅ Dim Papadopoulos
Vision–language models (VLMs) are increasingly used as automated judges for image-editing benchmarks, yet their reliability is poorly understood. Auditing these judges is difficult because real edited images lack controlled ground truth and entangle the requested change with model artifacts, unintended side effects, and ambiguous degrees of success. In this paper, we introduce EditJudge-Bench, a diagnostic benchmark of 1,500 synthetic before/after image pairs rendered in Blender, with edits parameterized across nine categories derived from real user requests and exact ground truth from known 3D scene parameters. Because every edit is generated by a known intervention on a controlled scene, EditJudge-Bench evaluates VLM judges as binary verifiers of whether an instruction was correctly executed, isolating judge perception from editor quality. Across 18 judge configurations, we find substantial reliability gaps: (a) the widely adopted VIEScore template reaches only 67.1\% AUROC with GPT-4o; (b) fine-tuned judges score 50.9–65.0\%; spatial edits such as rotation and shape remain near chance; (c) and prompt-formatting choices like image–text ordering shift performance by up to 13\%. Despite using only synthetic data, judge rankings on EditJudge-Bench correlate strongly with two real-image human-preference benchmarks ($\rho$=0.93 and $\rho$=0.77), indicating that controlled audits surface failure modes relevant to real evaluation.
Show more
WildBox: A Dataset and Benchmark for Aerial Monocular 3D Detection of African Savanna Wildlife
Vandita Shukla ⋅ Kilian Meier ⋅ Lucie Laporte-Devylder ⋅ Camille R Saint-Jean ⋅ Jenna M Kline ⋅ Blair R Costelloe ⋅ Devis Tuia ⋅ Fabio Remondino ⋅ Benjamin Risse
We introduce WildBox, a dataset and benchmark for monocular 3D detection of wildlife from drone video, comprising 237,505 3D bounding box annotations across seven African savanna species grouped into six benchmark classes. Annotations follow a KITTI/Omni3D-compatible format in a per-segment scale-normalised camera frame, with instance identities maintained across each segment. We evaluate two open-vocabulary monocular 3D architectures, OVMono3D-LIFT and DetAny3D, under zero-shot, ground-truth 2D box prompt, and supervised fine-tuning protocols. Open-vocabulary 2D foundation models provide usable zero-shot wildlife localisation (50.55 AP@50), but zero-shot 3D detection collapses to 0.00 AP across both architectures and every 2D-input condition tested, including ground-truth 2D box prompts—isolating the failure to the 3D stage. Fine-tuning on WildBox recovers performance to 8.68$\pm$0.47 AP-BEV@0.50 and 13.17$\pm$0.69 AP3D macro. Depth contributes 84\% of normalised Hausdorff distance after fine-tuning and over 99\% in zero-shot, identifying monocular aerial depth as the dominant open problem in this regime. A coarse-to-fine curriculum—pretraining on a merged zebra class before fine-tuning on the Grévy's/plains split—improves macro 3D performance with less total compute, with the largest gains on the two zebra subclasses. WildBox is released with video-level splits, evaluation code, and baseline checkpoints to enable progress in 3D wildlife perception from drone video.
Show more
Surprisingly consistent failures of post-hoc OOD detectors reveal security concerns in open-set recognition with modern CNNs and ViTs, stemming from representation mismatch
Kamil Szyc ⋅ Tomasz Walkowiak ⋅ Henryk Maciejewski
Although many OOD detection methods have been proposed, no clear winner can be identified as winners are benchmark-specific. One problem with evaluating OOD detectors is the imprecise definition of the "out" distribution, as many commonly used OOD datasets designed to test for 'semantic shift' contain a substantial proportion of 'covariate shift' samples. Moreover, recent work finds that, on carefully crafted benchmarks focused on semantic shift, modern OOD detectors yield little improvement over the baseline MSP. We analyzed a vast number of post-hoc OOD detectors on these benchmarks. We found that they consistently misclassify a common, substantial subset of samples across modern CNN and ViT architectures. We showed that failures across all OOD detectors on this benchmark are primarily due to learned feature representations: we observe that feature extractors map many semantically distant OOD samples closer to ImageNet classes than some in-distribution samples. This misalignment raises safety concerns about DNNs operating in open-set recognition scenarios. We publish a new benchmark, along with a reproducible construction protocol, that focuses on shared hard examples drawn from semantically distinct datasets, including NINCO and ImageNet-OoD. Our new benchmark can reveal fundamental security gaps in current DNNs, as they fail to reliably distinguish semantically distant OOD samples from in-distribution samples.
Show more
FakeParts: a New Family of AI-Generated Video Forgeries
Ziyi LIU ⋅ Firas Gabetni ⋅ Awais H SANI ⋅ Xi Wang ⋅ Soobash Daiboo ⋅ Gaëtan Brison ⋅ Gianni Franchi ⋅ Vicky Kalogeiton
We introduce FakeParts, a new class of AI forgery videos characterised by subtle, localised manipulations to specific spatial regions or temporal segments of otherwise authentic videos. Unlike fully synthetic content, these partial manipulations blend seamlessly with real elements, making them particularly deceptive and difficult to detect. To address this critical gap, we present FakePartsBench, the first large-scale benchmark designed to capture the full spectrum of partial forgeries and to evaluate modern detection methods. Our dataset comprises over 81K videos, including 44K FakeParts with pixel- and frame-level manipulation annotations, ranging from altered facial expressions to object substitutions and background modification. Our user studies demonstrate that FakeParts reduces human detection accuracy by up to 26\% compared to full-video forgery contents, with similar performance degradation observed in over fifteen state-of-the-art detection models. This work identifies an urgent vulnerability in current detectors and provides the necessary resources to develop methods robust to partial manipulations.
Show more
Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases
Eric Gan ⋅ Aryan Bhatt ⋅ Buck Shlegeris ⋅ Julian Stastny ⋅ Vivek Hebbar
As AI systems are increasingly used to conduct research autonomously, misaligned systems could introduce subtle flaws that produce misleading results while evading detection. We introduce Auditing Sabotage Bench, a benchmark for evaluating the ability of auditors to detect and fix sabotage in ML research codebases. Our benchmark consists of 9 ML research codebases with sabotaged variants that produce qualitatively different experimental results. Each sabotage modifies implementation details, such as hyperparameters, training data, or evaluation code, while preserving the high-level methodology described in the paper. We evaluated frontier LLMs and LLM-assisted human auditors on our benchmark and found that both struggled to reliably detect and fix sabotage: the best performance was an AUROC of 0.77 and a top-1 fix rate of 42%, achieved by Gemini 3.1 Pro. We also tested LLMs as red teamers and found that LLM-generated sabotages were weaker than human-generated ones but still sometimes evaded same-capability LLM auditors. We release the benchmark to support research on monitoring and auditing techniques for AI-conducted research.
Show more
An Open-Source Training Dataset for Foundation Models for Black-box Optimization
Aaron Klein ⋅ Herilalaina Rakotoarison ⋅ Luca Thale-Bombien ⋅ David Salinas
Most black-box optimization methods require extensive hyperparameter tuning, often limiting their ability to generalize across different optimization domains. Foundation models for black-box optimization that learn optimization principles from a large collection of optimization trajectories offer a promising alternative, with the potential to outperform manually designed methods across diverse problem classes. However, prior work has either relied on non-public datasets or on purely synthetic data, limiting reproducibility and generalization to real-world problems. As a result, progress in this area has been constrained by the lack of large-scale, real-world, publicly available pre-training data. We introduce BBO-Pile, the first open-source dataset comprising over 500K optimization trajectories evaluated across 3095 different black-boxes for different optimizers which represents by far the largest public dataset for this task. Using this dataset, we train a family of foundation models at multiple scales, ranging from 2M to 80M parameters and from 200M to 2B training tokens, and study their scaling behavior with respect to compute. Our results demonstrate that large-scale pre-training is a viable and effective approach to imitate black-box optimization methods, paving the way for future research in this direction.
Show more
Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study
Hao Dong ⋅ Hongzhao Li ⋅ Shupan Li ⋅ Muhammad Haris Khan ⋅ Eleni Chatzi ⋅ Olga Fink
Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, with studies varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks focus predominantly on action recognition, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness. This lack of standardization obscures a reliable assessment of the field's advancement. To address this issue, we introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG, which standardizes evaluation across six datasets spanning three diverse tasks: action recognition, mechanical fault diagnosis, and sentiment analysis. MMDG-Bench encompasses six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection. With $7, 402$ neural networks trained in total across $95$ unique cross-domain tasks, MMDG-Bench yields five key findings: (1) under fair comparisons, recent specialized MMDG methods offer only marginal improvements over ERM baseline; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a substantial gap to upper-bound performance persists, indicating that MMDG remains far from solved; (4) trimodal fusion does not consistently outperform the strongest bimodal configurations; and (5) all evaluated methods exhibit significant degradation under corruption and missing-modality scenarios, with some methods further compromising model trustworthiness. We release MMDG-Bench to enable more rigorous, reproducible, and directly comparable evaluation, addressing current limitations in evaluation practices and providing a stronger foundation for future progress in multimodal domain generalization.
Show more
No Triangulation Without Representation: Generalization in Topological Deep Learning
Johannes S. Schmidt ⋅ Martin Carrasco ⋅ Ernst Röell ⋅ Guy Wolf ⋅ Nello Blaser ⋅ Bastian Rieck
Despite an ever-increasing interest in topological deep learning models that target higher-order datasets, there is no consensus on how to evaluate such models. This is exacerbated by the fact that topological objects permit operations, such as structural refinements, that are not appropriate for graph data. In this work, we extend MANTRA, a benchmark dataset containing manifold triangulations, to a larger class of manifolds with more diverse homeomorphism types. We show that, unlike prior claims, both graph neural networks (GNNs) and higher-order message passing (HOMP) methods can saturate the benchmark. However, we find that this is contingent on the right representation and feature assignment, emphasizing their importance in baseline models. We thus provide a novel evaluation protocol based on representational diversity and triangulation refinement. Surprisingly, we find no indication that existing models are capable of generalizing beyond the combinatorial structure of the data. This points towards a research gap in developing models that understand topological structure independent of scale. Our work thus provides the necessary scaffolding to evaluate future models and enable the development of topology-aware inductive biases.
Show more
Realism VS Accuracy: Event Sequence Forecasting from a Generative Modeling Perspective
Dmitry Osin ⋅ Egor Surkov ⋅ Petr Mokrov ⋅ Igor Udovichenko ⋅ Ivan Karpukhin ⋅ Evgeny Burnaev ⋅ Egor Shvetsov ⋅ Aleksandr Korotin
Event sequences are chains of random events, each described by heterogeneous (numerical and categorical) features. The problem of event sequence forecasting occurs in finance, healthcare, business process intelligence, and other fields. In this work, we study long-horizon forecasting of event sequences as a conditional generative modeling problem. We evaluate forecast quality from two perspectives: prediction accuracy and distributional realism. To support this, we present a benchmark that contributes metrics for both aspects, strong statistical baselines, recent generative models adapted from adjacent fields, and curated real-world datasets. Careful evaluation reveals that no method excels at both accuracy and realism, highlighting a fundamental trade-off between the two. Finally, we introduce a procedure that aggregates samples from a realism-strong generative model into a state-of-the-art predictor, demonstrating that distributional realism can be converted into prediction accuracy.
Show more
DynamAuction: a reinforcement learning environment for repeated auction with dynamic value
Benjamin Heymann ⋅ Eugénie Patard ⋅ Corentin Pla ⋅ Patrick Loiseau
A buyer who repeatedly participates in auctions for related items often faces a dynamic value problem: what they win today changes what tomorrow's items are worth. Despite the empirical importance of such effects, especially in display advertising, the algorithmic literature on repeated auctions has overwhelmingly focused on the learning problem—treating each opportunity as a contextual bandit—while sidestepping the planning problem induced by the dependence of current value on past wins. We introduce DynamAuction, a synthetic environment for single-buyer sequential bidding. Each won auction contributes reward through a kernel conditioned on a carry-over context that aggregates the bidder's past wins via a commutative monoid. The environment supports first- and second-price auctions, non-homogeneous Poisson arrivals, delayed feedback, saturation, and recency effects. We also introduce a class of Model Predictive Control algorithms built on top of the myopic-optimal valuation—the optimal bid in a second-price auction if no further opportunity were to follow. This class of myopic-optimal MPC consistently outperforms off-the-shelf deep RL baselines (PPO, SAC, TD3). We release DynamAuction to support RL research on repeated auctions.
Show more
ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games
DAWEI WANG ⋅ Di Zhao ⋅ Jianran Yu ⋅ CHENGMING ZHOU ⋅ Marci Chi ⋅ Xiaoyang Liu ⋅ BEINING ZHANG ⋅ Shuliang Chu ⋅ Yun Sing Koh ⋅ Gary Ushaw ⋅ Richard Davison
Large multimodal models (LMMs) are increasingly deployed as agents that perceive, reason, and act in interactive multi-agent environments. Yet most existing benchmarks for LMM-based agents in multi-agent settings remain turn-based: the environment and other agents pause while an agent performs inference. Such protocols obscure a fundamental challenge of real-time constraints, where the world continues to evolve and delayed decisions may become outdated before execution. Motivated by this limitation, we introduce ROAR, a multiplayer action-shooter game environment for evaluating LMM-based agents in multi-agent settings under real-time constraints. ROAR covers cooperative, competitive, mixed cooperative-competitive, and basic single-agent scenarios. We further propose a Multi-Agent Asynchronous Gym interface that decouples environment evolution from agent reasoning by running the environment and each agent in separate threads, enabling the environment to advance at fixed time intervals without waiting for LMM inference and allowing agents to reason independently without blocking one another. ROAR also features convenient deployment, multimodal observations, and customizable levels. ROAR offers a practical testbed for advancing LMM-based agents beyond static, turn-based evaluation toward real-time multi-agent decision-making.
Show more
Long-Rollout Stability in AI Weather Models: A Quantitative Benchmark and Analysis
Fanny Lehmann ⋅ Firat Ozdemir ⋅ Yun Cheng ⋅ Torsten Hoefler ⋅ Sebastian Schemm ⋅ Benedikt Soja ⋅ Siddhartha Mishra
While AI weather models excel at short-to-medium range forecasts (up to 15 days), they frequently suffer from ill-defined "instabilities" when rolled out over longer horizons. This work addresses the lack of a formal taxonomy by categorizing these failures into three distinct regimes — blow-up, drift, and loss of seasonality — through year-long rollouts of nine state-of-the-art AI weather models. Our analysis reveals that stability hinges on the treatment of small spatio-temporal scales: unstable models amplify high-frequency energy, while stable models act as denoisers when noise is added to their inputs. Far from reducing these models to mere stochastic parrots, our findings highlight that stable models generate unique weather trajectories, conditioned on the initial state. We verify our findings through ablation studies on architectural design choices, conducted using state-of-the-art Vision Transformer (ViT) AI weather model architectures.
Show more
Fluxtrapolation: A benchmark on extrapolating ecosystem fluxes
Anya Fries ⋅ Jacob A Nelson ⋅ Martin Jung ⋅ Markus Reichstein ⋅ Jonas Peters
We introduce FLUXtrapolation, a benchmark for extrapolating ecosystem fluxes under progressively harder distribution shifts. Ecosystem fluxes are central to understanding the carbon, water, and energy cycles, yet they can only be measured directly at sparsely located measurement towers. Producing global flux estimates therefore requires training models on observed sites using globally available covariates and predicting in unobserved regions, that is, upscaling. Flux upscaling is a challenging domain generalization problem that is affected by a shift in covariate distribution across climates, ecosystem types, and environmental conditions, as well as by conditional shift: important drivers remain unobserved at global scale. We provide a quantitative analysis of both these shifts in $P_X$ and $P_{Y\mid X}$. FLUXtrapolation is designed based on domain expertise on flux upscaling: it defines temporal, spatial, and temperature-based extrapolation scenarios and evaluates performance across held-out domains, temporal aggregations, and tail errors. In a pilot study, we find that baselines perform similarly under median hourly RMSE, but separate under the proposed tail-focused and multi-scale evaluation. FLUXtrapolation therefore poses a realistic and thus relevant challenge for machine learning methods under distribution shift; at the same time, progress on this benchmark would directly support the scientific goal of improving flux upscaling.
Show more
BraveATA: Benchmarking Broad and Verifiable End-to-End Automated Theoretical Analysis of Large Language Models
Dingzirui Wang ⋅ Xuanliang Zhang ⋅ Keyan Xu ⋅ Qingfu Zhu ⋅ Wanxiang Che
Theoretical analysis plays a crucial role in understanding the mechanisms, capabilities, and limitations of large language models (LLMs). However, existing theoretical studies of LLM remain largely human-driven, making it difficult to systematically explore the rapidly expanding space of LLM-related phenomena and mechanisms. In this paper, we introduce the task of Automated Theoretical Analysis of LLM, which requires LLMs to discover, formalize, and prove theoretical mechanisms underlying LLM behavior. To support this task, we construct BraveATA, the first broad and verifiable benchmark for automated theoretical analysis of LLMs. BraveATA is built from 178 papers covering 11 research directions. For each paper, we manually extract the core claim and annotate the corresponding informal theorem, formal theorem, and proof, with additional Lean4 formalizations to support objective and reproducible evaluation. Experiments on 6 mainstream LLM families show that current models remain far from fully automating theoretical analysis. They struggle to construct faithful mathematical abstractions of LLM-specific concepts and, most critically, to produce compilable formal proofs. We further explore a multi-agent framework for end-to-end theoretical analysis based on BraveATA, which reveals that current LLMs still have substantial difficulty generating core claims that are both novel and valuable. Our benchmark and findings provide a foundation for evaluating and advancing LLMs as automated theoretical researchers for future research.
Show more
DriveHierarchy: A Benchmark for Diagnosing VLM Driving Capabilities from Open-Loop Understanding to Closed-Loop Execution
Chengkai Xu ⋅ Jiaqi Liu ⋅ Yicheng Guo ⋅ Peng Hang ⋅ Jian Sun
Evaluating VLM-based autonomous driving remains difficult because driving competence is composite, where a capable system must ground traffic participants and hazards, integrate context across views and time, reason about future evolution, and act appropriately under closed-loop interaction. Existing benchmarks usually assess either open-loop understanding or closed-loop driving but provide limited structure for explaining how these abilities are organized, how they relate, and how they may inform model diagnosis and improvement. We present \textsc{DriveHierarchy}, a hierarchical benchmark that organizes VLM-based autonomous driving into four ranks, spanning perceptual grounding, contextual memory, mental reasoning, and closed-loop execution. To instantiate this hierarchy, we integrate multiple open-source autonomous-driving datasets into a unified open-loop benchmark with 76,798 question-answer pairs over 84,279 frames and develop a closed-loop simulation platform with interactive scenario construction on a real-world road network, from which 100 driving scenarios are curated for embodied evaluation. Experiments on 15 VLMs show that \textsc{DriveHierarchy} captures structured but non-redundant capability variation, relates open-loop understanding to closed-loop driving, and provides a practical basis for diagnosis and benchmark-guided optimization. \textsc{DriveHierarchy} therefore serves as a unified framework for evaluating and improving VLM-based autonomous driving systems. An anonymized project has been released on \url{https://github.com/PerfectXu88/DriveHierarchy}.
Show more
Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth
Viktor Kocur ⋅ Sithu Aung ⋅ Gabrielle Flood ⋅ Yaqing Ding ⋅ Lukas Bujnak ⋅ Torsten Sattler ⋅ Zuzana Kukelova
Monocular depth estimation has improved significantly in recent years, driven by increasingly powerful models and large-scale training data. Predicted depth is increasingly used as an input signal for downstream tasks such as Structure-from-Motion (SfM), visual localization, and SLAM. However, monocular depth estimators (MDEs) are still primarily evaluated in terms of depth accuracy. Standard metrics aggregate errors globally and may not reflect the usefulness of depth for downstream geometric tasks. We therefore propose Depth2Pose, a framework for evaluating MDEs in the context of downstream tasks. By combining depth predictions with feature correspondences in depth-aware geometric solvers, we use relative camera pose estimation accuracy as a task-driven proxy for depth quality. Traditional benchmarks require dense ground truth in the form of per-pixel depth, which is expensive to obtain. In contrast, our formulation requires only camera poses, which can be estimated efficiently, e.g., using Structure-from-Motion pipelines. As a result, our framework can be applied to scenes where ground-truth depth is difficult to obtain, for example due to large scene scale or heavy occlusions (e.g., vegetated environments). Leveraging this, we introduce the D2P dataset, which contains challenging scenes outside the distribution of commonly used training data. We show that methods performing well under standard depth error metrics on existing benchmarks also perform well under our pose-based metric when evaluated on the same datasets, but do not necessarily generalize to our more challenging dataset. Finally, we provide a simple and extensible evaluation framework. The dataset and code are available at kocurvik.github.io/depth2pose.
Show more
TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks
Andrej Tschalzev ⋅ Nick Erickson ⋅ Yuyang (Bernie) Wang ⋅ Huzefa Rangwala ⋅ Stefan Lüdtke ⋅ Christian Bartelt ⋅ Heiner Stuckenschmidt
Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remains a critical yet underexplored component of real-world modeling pipelines that is entirely absent from modern benchmarks, which creates an unquantified evaluation gap. In this work, we introduce **TabPrep**, a lightweight preprocessing pipeline composed of feature generators that are carefully designed to target three specific structural data patterns. We show that many widely used model classes exhibit predictable blind spots to these patterns and that systematic feature engineering alone can establish new peak performance. Across the TabArena benchmark, integrating TabPrep into model training and tuning consistently improves performance for tree-based, neural, linear, and foundation models, often surpassing gains achieved by model-centric innovations alone. TabPrep outperforms previous automated feature engineering approaches in performance, efficiency, and applicability across datasets, enabling integration into large-scale benchmarks. By releasing TabPrep, we enable researchers to integrate feature engineering into their benchmarking setup, filling a longstanding gap in tabular evaluations.
Show more
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
Mario Koddenbrock ⋅ Christoph Lange ⋅ Robin Legner ⋅ Martin Jaeger ⋅ Martin Kögler ⋅ Mariano N Bournazou ⋅ Peter Neubauer ⋅ Felix Bießmann ⋅ Erik Rodner
Machine learning (ML) has transformed many scientific fields, yet key applications still lack standardized benchmarks. Raman spectroscopy, a widely used technique for non-invasive molecular analysis, is one such field where progress is limited by fragmented datasets, inconsistent evaluation, and models that fail to capture the structure of spectral data. We introduce RamanBench, the first large-scale, fully reproducible benchmark for ML on Raman spectroscopy, consisting of streamlined data access (https://pypi.org/project/raman-data/), evaluation protocols and code (https://pypi.org/project/raman-bench/), as well as a live leaderboard (https://huggingface.co/spaces/HTW-KI-Werkstatt/RamanBench). It unifies 77 datasets (including 17 first released with this benchmark) across four domains, comprising 325,862 spectra and spanning classification and regression tasks under diverse experimental conditions. We benchmark 28 models under a standardized protocol, including classical methods (e.g., PLS), Raman-specific (e.g., RamanNet), tabular foundation models (TFMs; e.g., TabPFN), and time-series approaches (e.g., ROCKET). TFMs consistently outperform domain-specific and gradient boosting baselines, while time-series models remain competitive. However, no single method dominates across these heterogeneous datasets, revealing that Raman spectroscopy remains an open challenge for machine learning. Therefore, we invite the community to contribute new approaches to our living benchmark, with the potential to accelerate advances in critical applications such as medical diagnostics, biological research, and materials science.
Show more
StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning
Xixiang He ⋅ Baiqi Wu ⋅ Xingming Li ⋅ Ao Cheng ⋅ Qiyao Sun ⋅ Xuanyu Ji ⋅ Qingyong Hu
Multimodal large language models (MLLMs) often *know the rule but pick the wrong answer*: on abstract visual reasoning (AVR) tasks, a model can correctly describe what it sees and correctly name the underlying pattern, yet still fail to choose the matching candidate. Existing AVR benchmarks cannot detect this gap because they collapse perception, rule induction, and answer selection into a single right-or-wrong signal. We introduce **StemBind**, a shared-stem diagnostic benchmark that probes the same visual stem with three aligned questions: **P**erception (what is in the image), **R**ule (what pattern governs it), and **F**ull (which option completes it), so that a final-answer error can be attributed to a specific sub-step on the same evidence. StemBind contains **2,298** curated knowledge-light stems across nine auditable visual operations, totaling **19,533** P/R/F tasks, with each full item annotated by Sternberg's four reasoning stages: **S1 Encode**, **S2 Infer**, **S3 Map**, and **S4 Apply**. Evaluating **24 frontier MLLM configurations** (proprietary and open-source) yields four findings. **(i) The R-F chasm.** Rule accuracy exceeds full-item accuracy on **22 of 24** models, so most failures happen *after* the rule has been identified. **(ii) A persistent binding gap.** Even when P and R are both correct on the same stem, models still answer F incorrectly **51.2%** of the time. **(iii) The bottleneck is S3.** Process diagnostics and Stage-wise Stimulus Augmentation (SSA) localize the dominant failure to *rule-to-instance mapping*, the step that binds an inferred rule to the right candidate. **(iv) Scaling and thinking do not help.** Neither scaling up model size nor enabling explicit thinking mode reliably closes the gap; in paired comparisons, thinking mode lifts perception but *lowers* both rule and full-item accuracy. StemBind reframes AVR evaluation from final-answer ranking to locating *where* abstract visual reasoning breaks down, and identifies rule-to-instance binding as a concrete next target for vision-grounded reasoning.
Show more
Redefining Instance Matching: A Unified Framework for Part-Aware Matching in Panoptic Segmentation Evaluation
Erik Großkopf ⋅ Soumya Snigdha Kundu ⋅ Hendrik Möller ⋅ Nicolas Münster ⋅ Mehdi Astaraki ⋅ Paula T Buzduga ⋅ Kerstin Ritter ⋅ Benedikt Wiestler ⋅ Jan Kirschke ⋅ Jonathan Shapey ⋅ Tom Vercauteren ⋅ Florian Kofler
The Panoptic Quality (PQ) metric is the standard for jointly evaluating instance and semantic segmentation. However, its original definition relies on a One-to-One matching between predicted and ground truth segments, which is only straightforward when the IoU threshold exceeds 0.5. Below 0.5, multiple matching strategies emerge in a poorly explored problem space. We systematically elucidate this space by recasting segment matching as a constrained bipartite assignment problem. Independently bounding the prediction- and ground-truth-side degrees yields four matching strategies: One-to-One, Many-to-One, One-to-Many, and Many-to-Many. We show that the first three are well-defined within the PQ framework, while Many-to-Many falls outside it. These strategies become relevant when instances are fragmented, adjacent objects are difficult to delineate, or annotations are noisy. Central to our framework is a vertex-based accounting of TP, FN, and FP, anchored to ground truth and predicted segments rather than to matching edges. We further show that the framework extends naturally to part-aware panoptic segmentation, and we explore part-aware evaluation on biomedical data. Across configurable case studies we report how different combinations of thresholds and matching strategies behave in practice. We release a unified open-source package built on Panoptica. It exposes Voronoi-based region-wise analysis, part-aware evaluation, and Area Under Threshold Curve computations as configurable options.
Show more
WebArena-Pro: A Heterogeneous, Multimodal, Reproducible Benchmark for Web Agents
Imene Kerboua ⋅ Fatemeh Pesaran Zadeh ⋅ Xing Han Lu ⋅ Weijian Qi ⋅ Alexander Miller ⋅ Junyi Song ⋅ Yunjia Tian ⋅ Dongjin Kang ⋅ Seyeon Choi ⋅ Marzia Nouri ⋅ Ewen Gueguen ⋅ Matteo Boglioni ⋅ Fengyuan Liu ⋅ Zeyi Liao ⋅ Mengqi Yuan ⋅ Yue Li ⋅ Alexandre Lacoste ⋅ Alexandre Drouin ⋅ Spandana Gella ⋅ Huan Sun ⋅ Gunhee Kim ⋅ Siva Reddy
Web agents powered by large language and vision-language models are increasingly applied to realistic browser work that spans heterogeneous applications, multimodal content, and stateful workflows. However, existing reproducible web-agent benchmarks cover only a small number of web applications drawn from a few software categories, and restrict modality to text and vision. Live benchmarks broaden site coverage but sacrifice reproducibility, since pages and data drift between runs. Moreover, existing benchmarks do not meaningfully evaluate whether agents can understand and use audio and video content embedded within web tasks. To address these gaps, we introduce WebArena-Pro, a benchmark comprising 300 tasks across 20 self-hosted web applications in six domain categories, spanning distinct interface conventions, workflows, and data models. Across the evaluated agents, the best performance is achieved by Gemini 3.1 Pro, which attains 37.0 \% success under a 50-step budget, while open-source models' performance does not exceed 27.7\% success. Among reproducible, human-curated web agent benchmarks, WebArena-Pro provides the broadest application coverage and the most comprehensive multimodal support to date. The benchmark treats audio and video as core observations alongside text and vision, with dedicated actions for extracting information from each. WebArena-Pro runs each task in isolation and supports reproducible, parallel evaluation. Tasks are authored through a dedicated annotator interface, filtered by LLM-assisted triage, and finally validated by humans before release.
Show more
ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins
Yichen Zhou ⋅ Jonathan L Golob ⋅ Amir Mohammad Karimi Mamaghan ⋅ Stefan Bauer ⋅ Patrick Schwab
Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking on viral proteins remains limited, a critical gap given the need for proactive tools that can anticipate emerging mutations ahead of experimental validation. Here we introduce ViroGym, a comprehensive benchmark evaluating pLMs across three tasks: 79 deep mutational scanning (DMS) assays covering eukaryotic viruses with 552,065 mutated sequences across 7 phenotypic readouts, 21 influenza neutralisation tasks, and a real-world pandemic prediction task for SARS-CoV-2. We benchmark well-established pLMs on fitness landscapes, antigenic diversity, and pandemic forecasting, and find that the ProGen2 family consistently achieves the strongest performance across all three tasks. Crucially, DMS and neutralisation performance reliably identifies models that generalise to real-world emergence, even though the mutation sets they surface barely overlap, revealing that complementary in vitro benchmarks capture the evolutionary constraints needed for real-world mutation forecasting.
Show more
ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models
Mayank Kulkarni ⋅ Rakesh Vaideeswaran
Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively target single-answer retrieval. We introduce ListQA, a benchmark of 9,045 human-curated, cross-validated questions (estimated error rate below 2%) that require LLMs to recall multiple facts and compose them into structured lists of 3--10 elements. Questions range from flat lists `Which NCAA teams went undefeated between 2000 and 2024?` to hierarchical lists with sub-attributes `... and what was their record and result?`, spanning eight categories across 60+ countries. An evergreen design (explicit temporal constraints or historically immutable facts) keeps ground truth valid without periodic updates. For evaluation, we frame element matching as a linear sum assignment problem: optimal bipartite matching paired with LLM-based semantic grading jointly captures factual accuracy, hallucination, and format compliance. We benchmark 34 models (8 families, 42 configurations across standard and thinking-enabled inference) and find: the best standard-inference model scores below 37% LLM-Judge-F1, multi-faceted questions are harder across the board, and providing the source Wikipedia page lifts scores dramatically, confirming that models can organize facts but struggle to retrieve them from parameters alone. Extended analyses cover base vs. instruction-tuned checkpoints, oracle RAG, and token efficiency. We release the dataset and evaluation code.
Show more
Cosine is Human: The Reproducibility Ceiling of Perceptual Similarity
Benjamin TenWolde ⋅ Virginia de Sa
NIGHTS is a recent benchmark for human perceptual similarity over objects, scenes, and composition. DreamSim, the supervised metric introduced with NIGHTS, and later supervised metrics trained on the same data report large gains over self-supervised baselines. The standard NIGHTS evaluation uses the unanimous-agreement subset of a released 100,000-triplet corpus, leaving weak-majority and split-vote triplets outside the headline benchmark. We evaluate eleven perceptual similarity methods on the full released corpus, analyzing unanimous, weak-majority, and split-vote triplets separately. On the 64,420 non-tie triplets, cosine distance on a frozen self-supervised backbone reaches 74.1% accuracy without training, LoRA fine-tuning reaches 77–79%, and the DreamSim ensemble reaches 80.7%. A split-half reproducibility estimate from the triplet vote counts shows that majority labels reproduce only 70.1% of the time overall, and only 33–50% on weak-majority triplets. The extra accuracy comes from matching the collected votes, not from showing that the metric would better match a new set of human judgments. The evaluation target is also unstable across task formulations: NIGHTS’ 2AFC similarity labels and JND same/different labels disagree on 51–64% of shared stimuli. Fine-tuning changes the feature basis as well, increasing foreground-only fragility and making DreamSim the most fragile method under rotation, color, and blur. NIGHTS supports a narrower conclusion: cosine distance on self-supervised features captures the stable signal currently measurable in these labels. Additional claims of human alignment require a redesigned benchmark with more votes per triplet, axis-specific similarity questions, per-subject statistics, and ablation tests.
Show more
The BV4 Benchmark for Unsupervised Anomaly Detection in High-Dimensional Spectral Data Streams
Nicolas Rojas Varela ⋅ Julien Ah-Pine ⋅ Engelbert MEPHU NGUIFO
This paper introduces the BV4 Benchmark, a collection of high-dimensional spectral datasets for unsupervised anomaly detection in vacuum environment data streams. Collected via Optical Emission Spectroscopy (OES), the benchmark comprises nine experiments representing practical scenarios structured according to established anomaly detection literature and validated by domain experts to address challenges such as spatial anomalies, temporal anomalies, and concept drift. BV4 datasets are composed of 2048 distinct wavelengths recorded over time, accompanied by timestamps and ground truth labels, enabling rigorous evaluation of machine learning models under dynamic environmental changes within vacuum chambers. We demonstrate the utility of BV4 through a simple comparative evaluation against previous datasets for anomaly detection in spectral data streams and discuss the current limitations and future improvements of the benchmark to support meaningful evaluative claims in the area.
Show more
Text-to-CAD Evaluation with CADTests
Dimitrios Mallis ⋅ Marco Wang ⋅ Ahmet Karadeniz ⋅ Elisa Ricci ⋅ Anis Kacem ⋅ Djamila Aouada
Text-to-CAD has recently emerged as an important task with the potential to substantially accelerate design workflows. Despite its significance, there has been surprisingly little work on Text-to-CAD evaluation, and assessing CAD model generation performance remains a considerable challenge. In this work, we introduce a new evaluation perspective for Text-to-CAD based on automated testing. We propose CADTestBench, the first test-based benchmark for Text-to-CAD, based on CADTests, executable software tests that verify whether a generated CAD model satisfies the geometric and topological requirements of the input prompt. Using CADTestBench, we conduct comprehensive benchmarking of recent Text-to-CAD methods and further demonstrate that CADTests can also guide CAD model generation, yielding simple baselines that surpass performance of current methods. CADTestBench code and data are available at GitHub and Hugging Face dataset.
Show more
Lethe: Link Inference Attacks For Evaluation of Edge Unlearning Methods
Andrea D'Angelo ⋅ Francesco Gullo ⋅ Davide Mottin ⋅ Sayan Ranu ⋅ Giovanni Stilo
Graph Unlearning (GU) aims to remove the influence of specific training data from a Graph Neural Network without retraining anew. While most methods address removal of nodes and features, Edge Unlearning (EU) aiming at forgetting edges has recently attracted dedicated attention as a standalone problem due to the hardness of completely removing the impact of an edge in the model. Despite this, no dedicated benchmark for EU exists besides isolated evaluations on feature-rich datasets such as Cora and Citeseer where GNNs exploit node features rather than graph structure. As such, we show two systematic limitations in existing evaluations: dedicated EU methods can be slower than retraining anew, and accuracy metrics are uninformative due to the negligible impact of edge removal. To address this gap, we introduce Lethe, the first comprehensive benchmark for Edge Unlearning, featuring 15 methods and 8 datasets. Our benchmark emphasizes large graphs, introduces evaluation tasks of varying difficulty, and proposes empirical measures based on link inference attacks rather than proxy accuracy metrics. We provide a comprehensive empirical analysis, demonstrate the shortcomings of current evaluations, and release Lethe as a reproducible, extensible benchmark.
Show more
HelpBench: Assessing the Ability of LLMs to Provide Privacy, Safety, and Security Advice
Sarah Meiklejohn ⋅ Sunny Consolvo ⋅ Patrick G Kelley ⋅ Tara Matthews ⋅ Sai Teja Peddinti ⋅ Renee Shelby ⋅ Lenin Simicich ⋅ Kurt Thomas
People increasingly turn to large language models (LLMs) for advice on numerous aspects of their lives. This paper investigates LLM capabilities in responding to questions about digital privacy, safety, and security, such as regaining access to a lost or suspended account, removing malware from a device, dealing with scammers or technology-facilitated abuse, and more. We curated a benchmark of 450 questions representing authentic user situations and developed rubrics for each question to evaluate factual accuracy and tone-based qualities of a response. We then developed and applied an auto-rater to evaluate responses from 18 state-of-the-art LLMs. Our results indicate that while models provide high-quality advice (with scores of 82\% on average), all models have significant room to improve, especially when tailoring advice to complex or high-risk scenarios.
Show more
Eyes on VLM: Benchmarking Gaze Following and Social Gaze Prediction in Vision Language Models
Hengfei Wang ⋅ Anshul Gupta ⋅ Pierre Vuillecard ⋅ Jean-marc Odobez
Vision-language models (VLMs) have rapidly evolved into general-purpose multimodal reasoners with strong zero-shot generalization. In this context, VLMs could greatly benefit the analysis of human gaze and attention, a central task in human behavior understanding that requires reasoning about the physical scene as well as the activity, interactions, and social context. However, the extent to which VLMs can reliably understand human gaze and related attentional behaviors remains largely unexplored. In this work, we present EyeVLM, a systematic evaluation framework for gaze understanding in VLMs across two complementary dimensions: tasks and models. To assess gaze understanding capabilities, we focus on two core tasks. The first, gaze following, i.e., predicting the 2D location where a person is looking, has a geometric and visual processing focus, requiring a precise understanding of the human face, attention direction, 3D scene structure, and spatial grounding of attended targets. The second, social gaze prediction, requires social and relational reasoning over multi-person interactions (e.g., mutual gaze and shared attention), and may benefit more from the LLM semantic reasoning capabilities within VLMs. Regarding models, EyeVLM evaluates these tasks in two ways: a zero-shot setting with a diverse set of state-of-the-art open- and closed-source VLMs, exploring different prompting strategies; and a fine-tuning approach based on task-specific QA pairs, studying the impact of model scale and data scale. As benchmarks, we rely on existing gaze understanding datasets and perform a systematic comparison with state-of-the-art purely visual models. Overall, our results show that current VLMs lack precise gaze understanding capabilities. While standard training helps reduce the gap with visual models, significant improvements are still needed.
Show more
GeoDial: A Multimodal Dialog Tutoring Dataset for Geometry Problem-Solving with Visual Tutor Turns
Sankalan Pal Chowdhury ⋅ Junling Wang ⋅ Donya Rooein ⋅ April Wang ⋅ Mrinmaya Sachan
Several educational domains rely heavily on diagrams and visual cues, yet most existing tutoring datasets are limited to text-only interactions. This limits the development of AI tutors that can teach in visually grounded ways used by human instructors. Thus, we introduce GeoDial, a multimodal tutoring dataset of over 1.3K teacher-student dialogs in the domain of geometry collected from experienced math teachers, where instructional turns are explicitly grounded in diagram highlights. We propose a scalable annotation protocol that integrates dialog acts, visual highlighting, and feedback, enabling fine-grained supervision of both language and visual tutoring behavior. To illustrate the challenges posed by this setting, we fine-tune several vision–language models on GeoDial and evaluate their ability to generate tutoring utterances and diagram highlights. While supervised fine-tuning substantially improves the quality of generated dialog, it struggles to produce accurate diagram highlights, revealing a key limitation of current methods and highlighting the need for approaches that more effectively integrate visual reasoning with pedagogical interaction.
Show more
ScrapeGraphAI-100k: Dataset for Schema-Constrained LLM Generation
William Brach ⋅ Francesco Zuppichini ⋅ Lorenzo Padoan ⋅ Marco Vinciguerra
Producing output that conforms to a specified JSON schema underlies tool use, structured extraction, and knowledge base construction in modern large language models. Despite this centrality, public datasets for the task remain small, synthetic, or text-only, and rarely pair real page content with the prompts and schemas used in practice. We introduce ScrapeGraphAI-100k, 93{,}695 schema-constrained extraction events collected via opt-in ScrapeGraphAI telemetry in Q2--Q3 2025, deduplicated and balanced by schema from 9M raw events. The corpus spans 18{,}000+ unique schemas across 15 named languages plus a long-tail Other category, with English and Traditional Chinese covering 88\% of detected content, each instance pairs Markdown-converted page content with a prompt, schema, LLM response, and per-example jsonschema-rs structural conformance labels (semantic correctness is out of scope, and raw HTML is deferred beyond v1.0). We characterize structural diversity across the corpus and identify sharp failure thresholds as schema complexity grows. As a case study, a 1.7B student fine-tuned on this data closely tracks the output distribution of its GPT-5-nano teacher, though it still trails a 30B-A3B reference (3.3B active parameters) on schema compliance. We offer this distillation result as preliminary evidence that grounding schema-constrained generation in real practitioner workloads at scale enables training and benchmarking that prior synthetic or text-only corpora could not support.
Show more
InCLAD: A Continual Learning Benchmark for Industrial Visual Anomaly Detection
Łukasz Marcjan ⋅ Dorota Wilk-Kołodziejczyk ⋅ Roberto Corizzo
Industrial anomaly detection benchmarks have advanced rapidly, yet they remain largely static and therefore underrepresent deployment settings in which inspection systems must be extended from one industrial component to another over time. In our setting, continual industrial anomaly detection is framed as a task-incremental problem: each component defines a task, while the learning objective remains semi-supervised. The goal extends beyond conventional anomaly detection, since models should preserve their performance on all components as they are sequentially exposed to new components. This framing is especially relevant because many existing industrial pipelines either assume fully supervised defect labels or study only isolated datasets, leaving open the question of how semi-supervised anomaly detectors behave across a sequence of heterogeneous components. We address this gap by organizing the benchmark as a sequence of related component-level tasks and by defining multiple scenarios, including random, easy-to-hard, and hard-to-easy. In addition to single-dataset continual scenarios, we propose challenging scenarios spanning across multiple datasets. We further evaluate continual performance from two complementary perspectives: image-level anomaly detection and pixel-level anomaly localization. By recasting industrial anomaly detection as a continual benchmark rather than a static train/test problem, our work establishes a concrete path toward studying knowledge retention, adaptation, and generalization in evolving industrial environments.
Show more
TabBioMed: A Large-Scale Benchmark for Biomedical Tabular Learning
Pau Mateo-Bernadó ⋅ Saivenkata Nagavyjayanthi Polapragada ⋅ Pol Arbiol Rakuljic ⋅ Laia M Pladevall ⋅ David Bonet ⋅ Marçal Comajoan Cara ⋅ Jesus Gonzalez Ferrer ⋅ Jordi Abante ⋅ Daniel Mas Montserrat ⋅ David Haussler ⋅ Alexander Ioannidis
Recent advances in machine learning have accelerated biomedical prediction, yet evaluation on structured biomedical data remains fragmented, especially for datasets with high dimensionality, missingness, class imbalance, and small-sample, many-feature regimes. We introduce TabBioMed, a large-scale benchmark of 95 curated public biomedical tabular datasets spanning electronic health records \cite{data2016secondary}, drug response \cite{wu2018moleculenet}, genomics, transcriptomics, proteomics, metabolomics \cite{yang2025mlomics}, single-cell omics \cite{heumos2023best}, and systems biology. TabBioMed unifies these datasets through a standardized, dataset-aware preprocessing and evaluation framework, enabling reproducible comparison across classification, regression, and multi-target tasks. We benchmark 23 models across classical baselines, gradient-boosted trees, neural tabular architectures, and emerging tabular foundation models. Foundation models deliver the strongest family-level predictive performance. However, their gains are regime-dependent: tree-based models and tuned MLPs remain highly competitive, often dominating under inference-time or training-cost constraints. We further show that foundation-model advantages are most meaningful in low-noise biomedical tasks, where small absolute gains translate into large reductions in residual error. Overall, TabBioMed reveals that biomedical tabular learning is strongly context-dependent, with optimal model choice shaped by data regime, task type, and computational budget. By releasing curated datasets, preprocessing code, baselines, and results, TabBioMed provides a reproducible foundation for practical model selection and future methods development in biomedical machine learning, available at https://anonymous.4open.science/r/tabbiomed.
Show more
ProSearch: Benchmarking Multi-Constraint Protocol Retrieval in Experimental Science
Wenliang Liang ⋅ Zhiyuan Ning ⋅ Haolin Chen ⋅ Yuanchun Zhou ⋅ Wenjuan Cui ⋅ Yi Du
In real-world AI for Science scenarios, retrieval systems are increasingly expected to support experimental workflows such as experimental design and method reproduction. These workflows require retrieving protocols that satisfy a set of interdependent constraints, such as specific materials and operating conditions. However, systematic evaluation of retrievers in such multi-constraint scenarios remains lacking. To address this gap, we introduce ProSearch, a scientific retrieval benchmark for multi-constraint experimental protocol retrieval. ProSearch has three key features: (1) realism, as queries are instantiated from real method paragraphs using expert-defined experimental fields, query templates, and filtering rules; (2) complexity, involving 60 distinct fields, such as experimental subjects, materials, and operating conditions, together with dependencies among these fields; and (3) diagnosticity, as each query contains 3 to 5 explicit hard constraints and covers challenging cases such as numeric constraints and negation constraints. We construct ProSearch through LLM-based extraction, data-driven constraint selection, controlled query generation, and evidence-based relevance validation. The final benchmark contains 1,551 high-quality queries across six experimental science domains. We use ProSearch to systematically evaluate 19 representative retrievers, spanning lexical matching, general dense retrieval, scientific retrieval, and reasoning-oriented retrieval. The results reveal that current retrievers generally struggle with multi-constraint protocol retrieval, with the best-performing model achieving an average nDCG@10 of 0.290. Further constraint-wise and error analyses reveal key failure modes of existing retrievers.
Show more
AgentKernelArena: Benchmarking Performance and Generalization of AI Coding Agents on GPU Kernels Optimization
Sharareh Younesian ⋅ Wenwen Ouyang ⋅ sina rafati ⋅ Mehdi Rezagholizadeh ⋅ Sharon Zhou ⋅ Ji Liu ⋅ Yue Liu ⋅ Yuchen Yang ⋅ Hao Li ⋅ Ziqiong Liu ⋅ Dong Li ⋅ Vikram Appia ⋅ Zhenyu Gu ⋅ Emad Barsoum
GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial low-level expertise. Recent AI coding agents can iteratively read code, invoke compilers and profilers, and refine implementations, yet existing kernel benchmarks evaluate single LLM calls rather than full agent workflows, and none include both kernel-to-kernel optimization and unseen-configuration generalization testing. We present AgentKernelArena, an open-source benchmark for measuring AI coding agents on GPU kernel optimization. The benchmark contains 196 tasks spanning HIP-to-HIP optimization, Triton-to-Triton optimization, and PyTorch-to-HIP translation, and evaluates complete agent workflows in isolated workspaces using gated compilation, correctness, and performance checks, centralized scoring and an unseen-configuration generalization protocol that tests whether optimizations transfer to input configurations the agent never observed. Across production agents including Cursor Agent, Claude Code, and Codex Agent, we find near-perfect compilation and high correctness rates on most task categories, with the strongest configurations achieving mean speedups of up to 6.89$\times$ on PyTorch-to-HIP, 6.69$\times$ on HIP-to-HIP, and 2.13$\times$ on Triton-to-Triton tasks. Our unseen-configuration evaluation shows that HIP-to-HIP and Triton-to-Triton optimizations largely transfer to unseen input shapes, while PyTorch-to-HIP exhibits substantial correctness drops, indicating that agents generating kernels from scratch frequently hardcode shape-specific assumptions. AgentKernelArena is designed as a modular, extensible framework for rigorous evaluation of agentic GPU kernel optimization across agents, tasks, and hardware targets. Code and data are available at \url{https://anonymous.4open.science/r/AgentKernelArena_Neurips2026-F646}.
Show more
OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models
Negin Baghbanzadeh ⋅ Pritam Sarkar ⋅ Michael Colacci ⋅ Abeer Badawi ⋅ Adibvafa Fallahpour ⋅ Arash Afkanpour ⋅ Leonid Sigal ⋅ Ali Etemad ⋅ Elham Dolatabadi
High-stakes clinical use of large vision–language models (LVLMs) requires reasoning that is grounded in visual evidence and clinical knowledge, not just correct final answers. We introduce OpenMedReason, a large-scale, open multimodal medical reasoning corpus comprising approximately 450K image–question–answer instances whose reasoning traces are primarily derived from curated biomedical, human-authored scientific articles. OpenMedReason provides high-fidelity supervision beyond synthetic chains of thought, covering diverse medical domain vision modalities such as radiological scans, microscopic images, visible light photographs, charts, and others. We complement it with OpenMedReason Bench, a held-out benchmark that allows fine-grained evaluation of LVLMs along three complementary axes of capability, including perception, medical knowledge, and rationale, enabling diagnostic evaluation beyond final answer accuracy. OpenMedReason is a rich training resource that exhibits its effectiveness in both supervised fine-tuning (SFT) and reinforcement-based alignment. Training with OpenMedReason yields a 20\% average improvement in VQA accuracy over the base model and achieves 4.2\% the strongest performance among comparable-scale medical LVLMs. Fine-grained performance analysis confirms that the gains are not concentrated in any single axis: OpenMedReason improves perception, medical knowledge, and rationale jointly, and its reasoning traces are preferred over those of the base model in 86.1\% of pairwise comparisons. We release the dataset and code.
Show more
Mosaic: A Benchmark Suite for Differentiable Physics Solvers
Andrin Rehmann ⋅ Heiko Zimmermann ⋅ Dion Häfner
Differentiable physics solvers underpin solver-in-the-loop ML training, gradient-based optimal control, and inverse problems, yet the practical cost of obtaining correct, usable gradients from a given solver on a given problem is largely undocumented. Integration effort, computational cost, gradient accuracy, and numerical conditioning vary widely across solvers and are discoverable only by trial and error. We introduce Mosaic, an extensible benchmarking framework for differentiable PDE solvers that standardizes access to solver gradients. Each solver is packaged as a containerized component (Tesseract) exposing a uniform gradient API regardless of language or automatic differentiation (AD) strategy, enabling researchers to evaluate, compare, and build on non-trivial physical solvers. Our evaluation of 14 solvers across fluid dynamics, structural mechanics, and heat transfer demonstrates that the benchmark surfaces practically relevant differences: order-of-magnitude variation in computational cost and Jacobian conditioning, alongside structural incompatibilities that eliminate solvers from realistic tasks entirely. Despite this variation, all solvers that produce gradients converge to similar optima, indicating that the practical barriers are memory limits, numerical stability, and setup compatibility rather than gradient accuracy alone. Mosaic is open-source and available at https://anonymous.4open.science/r/mosaic-bench.
Show more
MONET: A Massive, Open, Non-redundant and Enriched Text-to-image dataset
Benjamin Aubin ⋅ Gonzalo Iñaki Quintana ⋅ Onur Tasar ⋅ Sanjeev Sreetharan ⋅ Urszula Czerwinska ⋅ Damien Henry ⋅ Clément Chadebec
Training large text-to-image models requires high-quality, curated datasets with diverse content and detailed captions. Yet the cost and complexity of collecting, filtering, deduplicating, and re-captioning such corpora at scale hinder open and reproducible research in the field. We introduce MONET, an open *Apache 2.0* dataset of $103.8$M image–text pairs collected from 2.9B raw pairs across heterogeneous open sources through successive stages of safety filtering, domain-based filtering, exact and near-duplicate removal, and re-captioning with multiple vision-language models covering short to long-form descriptions, and further augmented with synthetically generated samples. Each image is shipped with pre-computed embeddings and annotations to accelerate downstream use. To validate the effectiveness of MONET, we train a 4B-parameter latent diffusion model *exclusively* on it and reach competitive GenEval and DPG scores, demonstrating that our dataset lowers the barrier to large-scale, reproducible text-to-image research.
Show more
Chem-PerturBridge: a harmonized compendium of small molecule perturbation transcriptomic effects
Artur Szałata ⋅ Olga Novitskaia ⋅ Maiia Shulman ⋅ Matthew Mella ⋅ Altynbek Zhubanchaliyev ⋅ Fabian Theis
Large perturbation models require training data encompassing chemical, cellular, and assay diversity. Current transcriptomic resources for small-molecule modeling, however, are fragmented across technologies, metadata conventions, controls, doses, and preprocessing pipelines. We introduce Chem-PerturBridge, a harmonized multi-dataset resource comprising over 37k compounds, 136 cellular contexts, and 1.25M transcriptomic samples across eight assay types, with standardized identifiers, metadata, and replicate-aware condition-level effects. We use the resource to evaluate matched-condition agreement across datasets and replicate agreement within datasets. Matched same-compound conditions generally show weak agreement in fine-grained logFC rankings and magnitudes across most dataset pairs, often falling below same-context different-compound baselines. In contrast, logFC direction agreement is substantially more stable and usually exceeds these baselines. We further evaluate Chem-PerturBridge as a pretraining resource for compound representation learning. Under a compound-held-out OP3 evaluation split, embeddings pretrained on Chem-PerturBridge improve over L1000-only embeddings, Morgan fingerprints, and the descriptor-free OP3 baseline across metrics. Chem-PerturBridge therefore supports both diagnostic evaluation of cross-dataset signature agreement and model-oriented reuse of heterogeneous perturbation transcriptomic data.
Show more
Gender Artifacts from Art History to Text-to-Image Generation
Piera Riccio ⋅ Miriam Doh ⋅ Benedikt Höltgen ⋅ Noa Garcia ⋅ Nanne van Noord
Artistic styles are rooted in specific socio-historical contexts that encode social hierarchies, including distinct constructions of gender. Yet in AI research, style has long been treated as a surface-level visual property: a filter of color, brushstroke, and texture applied to otherwise content-neutral scenes. We introduce the first dataset to investigate the interplay between gender representation and style in both historical and generated images. StyleGender comprises 74k images spanning 19 artistic styles, comprising art historical images with style and gender annotations, T2I-generated images under controlled style and gender prompts, and a semantically aligned set enabling direct art history-to-generation comparison. By proposing two Set Gender Artifact (SGA) metrics (PixelSGA and MaskSGA), capturing gender signals at the pixel level and in compositional structure, we show that (1) gender representation shapes visual features across artistic styles, (2) style keywords carry these patterns into T2I generation, and (3) generative models tend to amplify gender artifacts beyond what is observed in historical sources.
Show more
CurveBench: A Benchmark for Exact Topological Reasoning over Nested Jordan Curves
Amirreza Mohseni ⋅ Mona Mohammadi ⋅ Morteza Saghafian ⋅ Naser Talebizadeh Sardari
We introduce CurveBench, a benchmark for hierarchical topological reasoning from visual input. CurveBench consists of $\textbf{756 images}$ of pairwise non-intersecting Jordan curves across easy, polygonal, topographic-inspired, maze-like, and dense counting configurations. Each image is annotated with a rooted tree encoding the containment relations between planar regions. We formulate the task as structured prediction: given an image, a model must recover the full rooted containment tree induced by the curves. Despite the visual simplicity of the task, the strongest evaluated model, $\texttt{Gemini-3.1-Pro-Preview}$, achieves only $\textbf{71.1\\%}$ tree-generation accuracy on CurveBench-Easy and $\textbf{19.1\\%}$ on CurveBench-Hard. We further demonstrate benchmark utility through RLVR-style fine-tuning of open-weight vision-language models. Our trained $\texttt{Qwen3-VL-8B}$ model improves over $\texttt{Qwen3-VL-8B-Thinking}$ from $\textbf{2.8\\%}$ to $\textbf{33.3\\%}$ tree-generation accuracy on CurveBench-Easy, exceeding $\texttt{GPT-5.4}$ and $\texttt{Claude-Opus-4.5}$ under our evaluation protocol. The remaining gap, especially on CurveBench-Hard, shows that exact topology-aware visual reasoning remains far from solved.
Show more
MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image
Alan Arazi ⋅ Eilam Shapira ⋅ Shoham Grunblat ⋅ Mor Ventura ⋅ Elad Hoffer ⋅ Gioia Blayer ⋅ David Holzmüller ⋅ Lennart Purucker ⋅ Gael Varoquaux ⋅ Frank Hutter ⋅ Roi Reichart
Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generalizable representations of numerical and categorical structured data. However, they lack native support for unstructured modalities such as text and image, and rely on frozen, pretrained embeddings to process them. On established Multimodal Tabular Learning benchmarks, we show that tuning the embeddings to the task improves performance. Existing benchmarks, however, often focus on the mere co-occurrence of modalities; this leads to high variance across datasets and masks the benefits of task-specific tuning. To address this gap, we introduce MulTaBench, a benchmark of 40 datasets, split equally between image-tabular and text-tabular tasks. We focus on predictive tasks where the modalities provide complementary predictive signal, and where generic embeddings lose critical information, necessitating Target-Aware Representations that are aligned with the task. Our experimental results demonstrate that the gains from target-aware representation tuning generalize across both text and image modalities, several tabular learners, encoder scales, and embedding dimensions. MulTaBench constitutes the largest image-tabular benchmarking effort to date, spanning high-impact domains such as healthcare and e-commerce. It is designed to enable the research of novel architectures which incorporate joint modeling and target-aware representations, paving the way for the development of novel Multimodal Tabular Foundation Models.
Show more
Models That Know How Evaluations Are Designed Score Safer
Katharina Deckenbach ⋅ Haritz Puerto ⋅ Jonas Geiping ⋅ Sahar Abdelnabi
The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings. Prior work has identified test-time contextual cues, such as hypothetical scenarios, as a source of verbalized evaluation awareness and subsequent behavioral shift. In this paper, we investigate a potential explanation of this phenomenon: evaluation meta-knowledge, defined as parametric knowledge about the structural traits that characterize evaluations. Similar to dataset contamination, where benchmark exposure leads to higher performance through memorization, we hypothesize that models trained on texts describing evaluation practices may implicitly learn to recognize and respond to evaluation-like contexts, for instance through exposure to scientific articles or social media posts about AI benchmarking. To test this, we fine-tune models on synthetic documents describing evaluation traits such as verifiable structures or moral dilemmas. Evaluating this fine-tuned model on six safety benchmarks, we find that it is significantly safer than the base model and control model. This behavioral shift persists even when restricting the analysis to responses lacking explicit verbalization of evaluation awareness. Our results demonstrate that evaluation meta-knowledge may inflate safety benchmark performance, introducing a novel confounder that is independent of explicit memorization or verbalized evaluation awareness, thus, challenging to detect. These findings have important implications for the design and interpretation of AI safety evaluations.
Show more
APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings
Philipp Spohn ⋅ Leander Girrbach ⋅ Zeynep Akata
Typical LLM responses tend to follow a default style, even though users often have distinct preferences regarding tone, verbosity, and formality that they do not explicitly state in their prompts. Evaluating whether personalization methods can adapt to these implicit preferences is challenging, since users typically provide prompts rather than reference responses, style preferences are not factually verifiable, and reference-free LLM judges may conflate personalization with general response quality. To address these challenges, we introduce the Arbitrary Preference Mapping (APM) benchmark, which decouples user attributes (e.g. enthusiastic) from response principles (e.g. persuasive) via a hidden, randomized mapping $\mathbf{C}$ that maps user attributes to preferences about response traits. Because $\mathbf{C}$ carries no semantic content and is resampled across runs, models cannot exploit stereotypical associations and must infer preferences from conversation history. Using this unbiased evaluation methodology, we adapt retrieval-augmented, prompt-optimization, and routing personalization methods and evaluate them on Llama-3.1-8B and Qwen-3.5-27B. Our results show that routing is the most reliable approach, while RAG benefits only from the stronger base LLM, and soft prompt optimization fails to improve significantly over a non-personalized baseline. Our extensive evaluation reveals that in this realistic setting, personalization remains challenging, but our adapted methods show promise.
Show more
LSC-Parlament: An Automatically Aligned Catalan Sign Language Dataset from Parliament Videos.
Carlos Escolano ⋅ Gerard Sant ⋅ Marc J Garcia ⋅ Joan G Cortés ⋅ Marcel Granero Moya ⋅ Francesca De Luca Fornaciari ⋅ Maite Melero
Progress in Sign Language Translation (SLT) is frequently hampered by the "data bottleneck," a challenge particularly acute for regional languages such as Catalan Sign Language (LSC). In this paper, we present LSC-Parlament, a large-scale, multimodal dataset for LSC derived from Catalan Parliament sessions spanning 2021 to 2024. The dataset comprises over 87 hours of LSC video, synchronized with speech and text translations in either Catalan or Spanish. To curate this resource, we developed a fully automated pipeline that enables alignment between sign language, speech, and text without requiring prior human-annotated data. We provide a systematic evaluation of the dataset across multiple dimensions, including video tracking quality, audio transcription accuracy, and translation benchmarks. Our experiments compare transfer learning, zero-shot, and supervised translation settings, demonstrating significant knowledge transfer between spoken languages in the LSC-Spanish modality. By releasing LSC-Parlament, we provide a challenging open-domain benchmark to foster research in inclusive and scalable translation technologies.
Show more
Feedback Forensics: A Toolkit to Measure AI Personality
Arduin Findeis ⋅ Timo Kaufmann ⋅ Eyke Hüllermeier ⋅ Robert Mullins
Some traits making a “good” AI model are hard to describe upfront. For example, should responses be more polite or more casual? Such traits are sometimes summarized as model character or personality. Without a clear objective, conventional benchmarks based on automatic checks struggle to measure such traits. Evaluation methods using human feedback such as Chatbot Arena have emerged as a popular alternative. These methods infer “better” personality and other desirable traits implicitly by ranking multiple model responses relative to each other. Recent issues with model releases highlight limitations of these existing opaque evaluation approaches: goblin-obsessed, sycophantic and generally over-the-top personalities have resulted in model rollbacks and extensive media coverage. Despite these known issues, limited public tooling exists to explicitly evaluate model personality. We introduce Feedback Forensics: an open-source toolkit to track AI personality and related behavioural traits, both those encouraged by human (or AI) feedback, and those exhibited across AI models trained and evaluated on such feedback. Feedback Forensics is adaptive and can automatically find new traits appearing in models. We demonstrate the toolkit’s usefulness in two steps: (A) first we analyse the personality traits encouraged in popular human feedback datasets including Chatbot Arena, MultiPref and PRISM; and (B) then use our toolkit to analyse how much popular models exhibit such traits. We release (1) our Feedback Forensics toolkit alongside (2) a web app tracking AI personality in popular models and feedback datasets and (3) the underlying annotation data.
Show more
ProgramBench: Can Language Models Rebuild Programs From Scratch?
John Yang ⋅ Kilian Lieret ⋅ Jeffrey Ma ⋅ Parth Thakkar ⋅ Dmitrii Pedchenko ⋅ Sten Sootla ⋅ Emily McMilin ⋅ Pengcheng Yin ⋅ Rui Hou ⋅ Gabriel Synnaeve ⋅ Diyi Yang ⋅ Ofir Press
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable’s behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Show more
THEIA: A Multimodal Dataset and Benchmark for Vision-Language Analysis of Layout
Giuseppe Chiari ⋅ Michele Piccoli ⋅ Federico Viola ⋅ Davide Zoni
The integration of artificial intelligence into computer-aided design frameworks has sparked a shift in the design of analog integrated circuits (ICs), transitioning the field from using manual and algorithmic-based solutions to adopting automated and intelligent paradigms. In this scenario, the GDSII file represents the industry-standard database containing the ultimate and most accurate source of information of the analog circuit, encapsulating the complex physical geometries and parasitic realities that define tape out performance. This paper proposes THEIA, a novel dataset containing thousands of layout images paired with question-answer conversations, along with a benchmark that employs a fine-tuned vision-language model (VLM) to analyze GDSII files of analog circuits, enabling designers to interact with and query physical layouts as intuitive, meaningful entities. Experimental results using thousands of analog designs across five realistic tasks demonstrate that the proposed fine-tuned VLM outperforms state-of-the-art general-purpose VLMs by a significant margin (up to 73%), highlighting a fundamental gap between general-purpose multimodal reasoning and domain-specific layout understanding.
Show more
Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping
Chouaib Bencheikh Lehocine ⋅ Adam Lilja ⋅ Junsheng Fu ⋅ Lars Hammarstrand
Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (SOTA) methods commonly predict map elements as ordered sequences of points that form polylines and polygons. The evaluation of these methods relies predominantly on mean average precision (mAP) based on thresholded Chamfer distance (CD). This framework lacks sensitivity to point ordering and provides limited granularity in assessing geometric quality, making it difficult to distinguish which methods truly excel over others. In this work, we address these limitations on two fronts. For the single-instance similarity measure, we introduce sequence optimal sub-pattern assignment (SOSPA), an order-aware metric that enables fine-grained evaluation of individual geometries while satisfying all metric axioms. For the multi-instance evaluation framework, we propose polyline localisation and detection (PLD), a soft metric that jointly captures detection quality and geometric accuracy, replacing the hard thresholding of mAP with a principled soft assignment. Through evaluations on nuScenes, we demonstrate that PLD effectively ranks SOTA online mapping methods (MapTRv2, StreamMapNet, MapTracker) while providing a decomposed error analysis. This analysis identifies detection capability as the dominant bottleneck in current methods, revealing a performance trend that mAP fails to capture. Code for evaluation using our metrics will be released.
Show more
Flexible Intensities Matter: A comprehensive re-evaluation of Classical and Neural Temporal Point Processes
Hendrik Alexander Mehrtens ⋅ Florian Buettner ⋅ Oliver Stegle
Temporal point processes (TPPs) are the canonical framework for event sequence data, with applications spanning seismology, social media, and electronic health records. Recent neural extensions have been reported to substantially outperform classical baselines such as Hawkes processes, but a systematic comparison across model families and intensity parameterizations is lacking, and large-scale medical benchmarks — arguably one of the most promising application domains — have not been considered. Here, we revisit this comparison through a systematic benchmark of neural and non-neural TPPs on existing datasets and on two large-scale medical benchmarks derived from UK Biobank disease histories and MIMIC-IV ICU records. First, our results show that the previously reported gap between Hawkes processes and neural TPPs largely disappears once Hawkes processes are equipped with flexible and learnable kernels. Second, we find that the flexibility of the intensity parameterization also limits prior neural TPPs: a transformer with a spline-based intensity head outperforms prior neural TPP implementations. Third, our benchmark underscores the value of complex datasets when assessing the performance of TPPs. In fact, exclusively on the MIMIC-IV ICU records did we find clear evidence for structures — time-varying or higher-order interactions — beyond what pairwise flexible Hawkes process kernels can capture. Our results guide both future method development and practitioners’ model choice. Our TPP framework is released as an open-source package, providing consistent and feature rich implementations of classical and neural TPPs.
Show more
The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation
Jorma Valjakka ⋅ Juhani Kivimäki ⋅ Juha Mylläri ⋅ Jukka K Nurminen
In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often evaluated with hallucination detection benchmarks using open-domain question answering (QA) datasets that contain questions and corresponding short reference answers. Currently, these benchmarks use an LLM to generate answers to questions within the QA dataset. Then, various automated strategies are applied to label these answers as hallucinated or not by comparing them with the reference answers. This evaluation setting creates a methodological ambiguity between two criteria: reference faithfulness --- whether the answer is fully supported by the reference --- and factual correctness --- whether the answer is free from contradictions and factually false specific claims. In practice, automated labelers may apply the former criterion even when the intended target is the latter. We study this potential criterion mismatch using 900 human-labeled question--answer pairs spanning three commonly used QA datasets and three generator models. Our human annotations target answer-level factual correctness. As automated label generation strategies, we evaluate lexical similarity metrics, a reference-entailment NLI baseline, and seven LLM judges under controlled prompt variants. We compare the effect of using either a faithfulness-oriented prompt, which asks the judge model to treat any answer content unsupported by the reference as a hallucination, or a factual-correctness prompt, which explicitly distinguishes between absence from the reference and factual error. Our experiments reveal substantial disagreement both among different automated label-generation strategies and between these automated labels and human annotations. Many strategies also exhibit strong biases towards certain error types. For most LLM judges, replacing the faithfulness-oriented prompt with the factual-correctness prompt significantly improves agreement with human annotations, showing that automated hallucination labels depend strongly on how the target criterion is specified. For open-domain QA benchmarks targeting factual correctness, relying on strict reference faithfulness can introduce systematic measurement bias. Label-source choice should therefore be considered a fundamental part of benchmark design and made explicit, validated, and matched with the benchmark goal.
Show more
What DNA Foundation Models Learn Beyond Sequence Composition
Vincenzo Y. Civale ⋅ Andrew Bagdanov ⋅ Alberto Magi
A central question in genomic representation learning is whether DNA foundation models~(FMs) encode information beyond simple sequence composition, or whether their apparent representational power largely reflects signals already captured by classical compositional statistics. We address this by establishing $k$-mer frequency vectors as a rigorous biologically grounded baseline and introducing a geometric decomposition that partitions FM embeddings into a component linearly predictable from $k$-mer frequencies and an orthogonal residual representing genuinely non-compositional information. By evaluating each component separately as a downstream feature space, we directly measure what FMs encode beyond composition and whether that content is beneficial, neutral, or detrimental. Across 57 genomic classification datasets and 2 gene expression regression tasks, we compare five FMs (NTv3-650M, HyenaDNA, DNABERT-2, Caduceus-Ph, Evo2-1B) against $k$-mer features ($k \in \{4,5,6\}$) under a controlled frozen-embedding protocol with a fixed Random Forest classifier. NTv3 is the only model to consistently outperform $k$-mers (63.2\% of datasets, mean MCC $= 0.562$), with gains concentrated in splice-site and promoter tasks; HyenaDNA, DNABERT-2, and Evo2 are outperformed on 86\%, 95\%, and 93\% of datasets. Wilcoxon signed-rank tests with FDR correction confirm these patterns are systematic and robust to dataset heterogeneity. Our decomposition reveals that FM advantage originates exclusively from the non-compositional residual, and only in tasks with inherently positional regulatory signals. For HyenaDNA and DNABERT-2, the residual actively degrades classification. The fraction of embedding variance explained by $k$-mers ($R^2$) is uncorrelated with downstream quality, demonstrating that the quantity of non-compositional information is not a proxy for its utility. These results show that, in the frozen-embedding regime, current DNA FM representations largely recapitulate sequence composition, and provide a principled diagnostic for predicting which task conditions are likely to benefit from non-compositional structure.
Show more
Dataset Collections: Challenges of Large-Scale Data Aggregation in 3D Medical Image Datasets
Yannick Kirchhoff ⋅ Saikat Roy ⋅ Elisa Stegmeier ⋅ Hamideh Haghiri ⋅ Constantin Ulrich ⋅ Maximilian R. Rokuss ⋅ Tassilo Wald ⋅ Karol Gotkowski ⋅ Benjamin Hamm ⋅ Raphael Stock ⋅ Nico Disch ⋅ Michael Baumgartner ⋅ Dimitrios Bounias ⋅ Anand Deshpande ⋅ Stefan Dinkelacker ⋅ Stefan Dvoretskii ⋅ Katharina Eckstein ⋅ Selen Erkan ⋅ Jessica Kächele ⋅ Kim-Celine Kahl ⋅ Balint Kovacs ⋅ Lucas Kulla ⋅ Moritz Langenberg ⋅ Philipp Schader ⋅ Stephen Schaumann ⋅ Darya Trofimova ⋅ Shuhan Xiao ⋅ Sebastian Ziegler ⋅ David Zimmerer ⋅ Marco Nolden ⋅ Fabian Isensee ⋅ Klaus Maier-Hein ⋅ Ralf Floca
The development of deep learning-based medical AI depends on high-quality annotated data. Yet, dataset creation remains constrained by the need for expert radiologist annotations, limiting curated datasets to relatively small cohorts. To overcome this limitation, recent efforts increasingly aggregate disparate public datasets into large-scale dataset collections. However, combining datasets at scale often obscures data provenance, introducing significant risks of reuse propagation, unintended leakage, bias amplification and distorted data distributions. In this work, we study these problems on a massive collection of 760k 3D radiological images across 1069 public datasets and provide practical resources to address it. Firstly, we formalize dataset collections as a distinct paradigm in medical image analysis and characterize their structural challenges, including reuse propagation, duplication, and provenance fragmentation. Secondly, we perform one of the first large-scale empirical studies of duplication on medical image collections and demonstrate extensive exact and near-duplicate reuse across public datasets via perceptual hashing and manual tracing on our 760k 3D radiological images across 1069 public datasets, with 118k near or exact duplicates. We make these hashes publicly available, thereby providing a standardized reference for identifying potential overlaps and shared provenance for researchers. Thirdly, owing to the insufficient robustness of hashing in isolation, we introduce MAP (Metadata for Aggregation and Provenance), a lightweight, portable metadata profile for documenting provenance, known overlaps, and deduplication decisions in dataset collections. Our contributions enable dataset collections to shift from coarse aggregations to reliable and provenance-aware data scaling mechanisms in medical image analysis. A webtool supporting our work is made available here:
Show more
Reasoning-Trace Collapse: Evaluating the Loss of Explicit Reasoning During Fine-Tuning
Lukas Twist ⋅ Helen Yannakoudakis ⋅ Jie Zhang
Explicit reasoning models are trained to produce intermediate reasoning traces before final answers, but downstream fine-tuning is often performed on ordinary instruction--response data that contains no such traces. We show that this mismatch can induce *reasoning-trace collapse*: a fine-tuned model continues to produce plausible final answers while losing the structurally valid explicit reasoning traces that made it a reasoning model in the first place. We introduce a structural evaluation framework that separates answer correctness from reasoning-trace validity, measuring valid, empty, missing, and truncated reasoning alongside reasoning-conditioned task performance. Using this framework, we study open-weight reasoning models and find that standard supervised fine-tuning can rapidly suppress valid reasoning traces, and that answer-only metrics can substantially obscure this failure: in several settings, performance conditional on valid reasoning remains high while the rate of valid reasoning falls sharply. We further show that simple loss-masking strategies substantially mitigate collapse without requiring teacher-generated reasoning traces. These results suggest that evaluations of fine-tuned reasoning models should report structural reasoning reliability in addition to final-answer performance, especially when adaptation data does not contain explicit reasoning traces.
Show more
Causal-VLM: Dense Causal Captioning in Videos
Asmar Nadeem ⋅ Mahrukh Awan ⋅ Muhammad Awais ⋅ Robert Dawes ⋅ Adrian Hilton ⋅ Armin Mustafa
Dense video captioning describes multiple events in long videos. However, each event is predicted independently, ignoring the causal relationships with the other events in the video. These causal relationships are important for human-like descriptions of long videos. Our experiments show that exisiting vision-language models fail at predicting causal relationships in long-form videos. No prior work jointly generates captions and event timestamps and predicts causal relationships over multi-event sequences in long-form videos. We address this through two contributions. First, we create a benchmark dataset with 21.6K videos and 85K events by using a large language model to generate annotations from dense captions in YouCook2 and ActivityNet. We validate this benchmark dataset through multimodal alignment, vision-language models, and human judgment. Second, we propose a novel architecture for dense causal captioning that not only generates captions but also learns causal relationships. This novel architecture not only predicts causality effectively (0.73 F1 ActivityNet, 0.64 F1 YouCook2) but also improves the vision-language model's core capabilities: caption quality increases by +6.3 CIDEr on ActivityNet and +7.4 on YouCook2, while temporal grounding becomes accurate as causal predictions force precise event boundary localization.
Show more
Liars' Bench: Evaluating Lie Detectors for Language Models
Kieron Kretschmar ⋅ Walter Laurito ⋅ Sharan Maiya ⋅ Samuel Marks
Prior work has introduced techniques for detecting when large language models (LLMs) lie, that is, generate statements they believe are false. However, these techniques are typically validated in narrow settings that do not capture the diverse lies LLMs can generate. We introduce LIARS' BENCH, a testbed consisting of 72,863 examples of lies and honest responses generated by a range of open-weight models across seven datasets. Our settings capture qualitatively different types of lies and vary along two dimensions: the model's reason for lying and the object of belief targeted by the lie. Evaluating both black- and white-box lie detection techniques on LIARS' BENCH, we find that existing techniques systematically fail to identify certain types of lies, especially in settings where it's not possible to determine whether the model lied from the transcript alone. Overall, LIARS' BENCH reveals limitations in prior techniques and provides a practical testbed for guiding progress in lie detection.
Show more
Toward Embodied World Agents via Embodied-Planning Dataset and Interactive World Models
Xiaokun Feng ⋅ Junshu Tang ⋅ zeyi lin ⋅ Ling-Hao Chen ⋅ Shuai Shao ⋅ Qinglin Lu ⋅ Kaiqi Huang
Building embodied agents that can perceive, reason, and act across diverse 3D worlds is a central research direction toward artificial general intelligence. Yet most existing work on embodied agents remains confined to specific tasks in a single scene and transfers poorly to new environments, leaving general embodied agents for open-world settings an important but under-explored direction. We refer to such agents as Embodied World Agents (EWAs), whose defining property is the ability to understand diverse visual environments and execute general actions in open-world conditions. To advance research on EWAs, we first release EmbodiedWorld-200K, a large-scale dataset of about over 200K samples for open-world embodied planning that spans diverse scenes. Each example is equipped with multi-level annotations, ranging from low-level camera-motion trajectories to high-level semantic instructions, and is accompanied by unified evaluation splits and task-specific metrics, providing a shared substrate for offline pretraining and fair evaluation of EWAs. Beyond offline supervision, a defining characteristic of an EWA is its interaction with the environment; training solely on offline data is therefore insufficient, especially for generalizing to scenes not covered in the training distribution. Building dedicated real interactive environments, however, is often prohibitively expensive. To address this, we propose a novel training scheme that employs a video world model as a low-cost environment proxy and performs interactive reinforcement-learning (RL) post-training of the EWA within it. Empirically, pretraining on EmbodiedWorld-200K substantially improves the open-world embodied planning ability of the EWA, and further RL post-training inside a world model yields clear gains on out-of-distribution scenes. The full dataset, baseline checkpoints, and the accompanying training and evaluation code for supporting future research on EWAs are publicly available at https://xiaokunfeng.github.io/EmbodiedWorld-200K/.
Show more
PRISM: Polarimetric Road-surface Intelligent Sensing and Measurement Dataset
Minseo Jung ⋅ Chanwoo Kim ⋅ Donggue Kim ⋅ Hyunjun Kim ⋅ Jeongwoo Park ⋅ Kichun Jo
RGB cameras struggle on the road surface itself: pavement is nearly texture-less at the centimetre scale, and dry, wet, and icy asphalt can look photometrically identical. Polarization is the natural complement, since Fresnel optics ties the angle of linear polarization (AoLP) to surface-normal azimuth and the degree of linear polarization (DoLP) to refractive index, but no public dataset has made it possible to test whether this physical promise translates into measurable gains on real roads. We introduce PRISM, a polarimetric road-surface dataset of 47,098 time-synchronized frames combining trichromatic linear polarization, co-boresighted RGB, 128-channel LiDAR, and RTK-GNSS/INS, collected across proving-ground and open-road environments under clear, overcast, rainy, foggy, and snowy conditions. Dense road-surface elevation ground truth, validated against as-designed proving-ground geometry, accompanies frame-level labels over five materials and five surface states. Two benchmark tracks are defined on these data: road-surface condition classification and bird's-eye-view elevation estimation. Both share a five-variant input ablation crossing RGB, monochromatic and trichromatic polarization, and their combinations. Reference baselines turn the physical promise into a measurable one. Adding trichromatic polarization to RGB reduces elevation MAE from 3.61 to 3.26cm and improves every threshold metric, while yielding consistent gains on condition classification across backbones. The improvement is task-conditional: trichromatic resolution drives the elevation gain, where AoLP encodes geometric information that intensity cannot, and stratified evaluation localizes where polarimetric channels matter most. PRISM, together with reference baselines, evaluation and stratification scripts, and a datasheet, is publicly available at \url{https://huggingface.co/datasets/NeurIPS-2026-PRISM/PRISM-Dataset}.
Show more
OWCE: Revealing Language Model Complementarity via Online Weakness-Conditioned Evaluation
Hankyul Baek ⋅ Jaewon Noh ⋅ Sang Seo ⋅ Sungpil Shin ⋅ Yongsu Kim
While modern applications of large language models (LLMs) rarely rely on a single model, evaluation still collapses each model into a single benchmark score. This simplification obscures each model's specific weaknesses and how those weaknesses are shared or unique across models. To address this, this paper presents Online Weakness-Conditioned Evaluation (OWCE), an evaluation framework that diagnoses each model's recurring weaknesses, generates weakness-conditioned follow-up items at increasing difficulty, and cross-evaluates the resulting items on all models. Beyond a scalar score, OWCE produces per-model weakness profiles, enabling direct comparison of where models fail and which others recover those weaknesses. These profiles expose pairwise structure that scalar scores leave invisible. This paper finds that each model has a best-matched counterpart that reduces its failure rate by up to $25$ percentage points. This counterpart is determined by weakness profiles rather than overall benchmark ranking, indicating that the highest-scoring model is not always the most effective partner for a given target. This paper evaluates OWCE empirically across seven benchmarks and seven models, and conducts a mathematical analysis showing that the generated items can provide more discriminative information under benchmark saturation. Code is available at https://anonymous.4open.science/r/OWCE-57E7
Show more
Not Another Text Benchmark: Putting the “Visual" Back in Visual Question Answering for Large Video Models
Rwiddhi Chakraborty ⋅ Yinong O Wang ⋅ Cheng Zhang ⋅ Fan Bai ⋅ Zhuoran Yu ⋅ Michael Kampffmeyer ⋅ Yong Jae Lee ⋅ Fernando D De la Torre ⋅ Robert Jenssen
Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained text and vision encoders. The usefulness of such models have also been demonstrated on a wide range of benchmarks, with an important caveat - the dominant approach in these benchmarks evaluates multiple choice reasoning via text options. This is a natural way to test text-based reasoning in these models, and has led to significant insights regarding model behavior in the community. In this work, we ask a different question - what happens when the evaluation modality is visual, rather than text? We introduce three new vision-centric evaluation benchmarks in temporal frame retrieval, video future prediction, and causal memory distortion, all designed around evaluating visual understanding capabilities in large video models. Our approach complements the existing approaches to evaluate video understanding in frontier models. We show that current frontier models exhibit significant weakness when attempting to reason through visual queries, rather than text. We conclude with an extended analysis section that provides pointers for future improvements in visual understanding for large video models.
Show more
REFORM-3D: A Representation-Centric Evaluation Framework for 3D Medical Vision Foundation Models
Maregu Assefa ⋅ Muhammad Muzammal Naseer ⋅ Divya Velayudhan ⋅ Gedamu Kumie ⋅ Naoufel Werghi
Evaluation of 3D medical foundation models is still dominated by downstream task scores, which entangle representation quality with adaptation choices. This is especially critical in medical imaging, where spacing and anisotropy can substantially alter image statistics while the anatomy itself remains unchanged. We introduce \textbf{REFORM-3D}, a \textbf{R}epresentation-centric \textbf{E}valuation \textbf{F}ramew\textbf{OR}k for benchmarking frozen 3D medical vision foundation \textbf{M}odels across three questions: Acquisition Robustness, asking whether features remain stable when spacing varies while anatomy is fixed; Cross-Modal Anatomical Alignment, asking whether same-organ CT-MR pairs remain recoverable under balanced bidirectional retrieval; and Anatomical Generalization, asking whether limited adaptation recovers held-out organ signals beyond a frozen-feature baseline under case-disjoint evaluation. These are also pretraining questions, because what a 3D encoder learns depends on how volumes are geometrically presented during self-supervised pretraining. We therefore employ REFORM-3D to evaluate encoders pretrained on 62K unlabeled multimodal 3D volumes spanning diverse anatomical structures across MR, CT, PET, CTA, and CBCT under three explicit geometric assumptions: isotropic sampling, voxel-relative sampling, and spacing-aware resampling. By exposing the encoder to acquisition geometry during pretraining rather than treating geometry only as a downstream issue, REFORM-3D evaluates whether learned representations preserve stable anatomical structures or remain sensitive to acquisition and modality shortcuts.
Show more
Beyond Bag-of-Words: Diagnosing Compositional Binding Failures in Vision-Language Models
Cristian Sbrolli ⋅ Toshihiko Yamasaki ⋅ Matteo Matteucci
Modern vision-language models struggle with basic compositional reasoning, failing to bind attributes to objects or relations to their referents. Existing benchmarks either rely on noisy real images that conflate confounding visual variables with the reasoning failure, or use simplistic synthetic scenes lacking the realism modern VLMs are tuned for. We introduce Auto-Comp, a fully automated, concept-driven pipeline that bridges this gap by generating photorealistic compositional benchmarks at scale. Its core innovation is a parallel A/B construction: for each concept, the pipeline emits a Minimal sample (template caption, isolated objects on a white background) and a Contextual sample (LLM-rewritten caption, objects embedded in a realistic scene), isolating core binding ability from visio-linguistic complexity. We instantiate four task families spanning the two canonical axes of compositional binding: Color and Shape-Color (attribute binding), and Position and Relative Size (relational binding). We evaluate over 25 VLMs spanning CLIP, SigLIP, hard-negative-trained, and frontier generative models. The findings are consistent across architectures and scales: every model exhibits a large Swap-vs-Confusion gap, with low-entropy distractors (e.g., repeated objects or colors) exposing failures beyond the known bag-of-words limitations. We further uncover a task-dependent trade-off: visio-linguistic context aids relational reasoning but hinders attribute binding through visual clutter. We publicly release the pipeline and benchmarks.
Show more
General Agent Evaluation
Elron Bandel ⋅ Asaf Yehudai ⋅ Lilach Edelstein ⋅ Yehoshua Sagron ⋅ Yotam Perlitz ⋅ Elad Venezian ⋅ Natalia Razinkov ⋅ Natan Ergas ⋅ Shlomit S Ifergan ⋅ Segev Shlomov ⋅ Michal Jacovi ⋅ Leshem Choshen ⋅ Liat Ein-Dor ⋅ Yoav Katz ⋅ Michal Shmueli-Scheuer
General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measured how agent architecture shapes performance across heterogeneous protocols and diverse unfamiliar environments. This is the first systematic study, comparing tool-calling, MCP, code-generation, and CLI agents on the same benchmarks with the same models. Two gaps blocked such a study: existing harnesses require per-benchmark wiring or fixed protocol classes (web for BrowserGym, CLI for Harbor), and benchmarks themselves expect human-authored prompts, context, and integration glue. To enable this study, we contribute (1) a unifying protocol that bridges existing benchmark and agent protocols; (2) an evaluation harness that surfaces any benchmark to any general-purpose agent and backbone model; and (3) the first Open General Agent Leaderboard of agent configurations, a full factorial over 5 agent architectures × 5 backbone LLMs (three closed-source, two open-weight) × 6 benchmarks spanning software engineering, customer service, deep research, and personal assistance. We find that (i) general agents adapt to every tested domain without per-domain customization; (ii) agent architecture choice swings results by up to 12pp within a single model, yet backbone model choice dominates overall performance; (iii) on 4 of 6 tested benchmarks, top general agents are indistinguishable from the leading heavily-customized domain-specific agents; (iv) open-weight models tested exhibit "generality sinks" absent from frontier closed-source models: they consistently collapse on specific agent architectures or benchmarks. Code, harness, leaderboard, and traces will be released upon acceptance.
Show more
GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines
Jafar Isbarov ⋅ Umid Suleymanov ⋅ I Shumailov ⋅ Murat Kantarcioglu
AI agents are increasingly embedded in continuous integration and continuous delivery (CI/CD) workflows to autonomously review pull requests, triage issues, and maintain codebases. We study fully automated configurations in which external contributions invoke agents without prior human approval. Because these agents ingest untrusted content while operating with repository credentials and tools, prompt injection can have software supply chain consequences. We present GitInject, an open-source framework for evaluating prompt injection in live GitHub Actions workflows. Unlike benchmarks that simulate tool calls, GitInject provisions ephemeral repositories and executes actual workflows, exposing platform semantics such as sandbox constraints, credential handling, and permission boundaries. We use GitInject to evaluate eleven named attack scenarios across five workflows from four AI providers, covering configuration-file injection, credential exfiltration, judgment manipulation, and availability. Every provider is susceptible to at least one high-severity attack in the tested configurations. The most consequential vulnerabilities are structural: attacker-controlled configuration files from an unmerged contribution are loaded as operator-level instructions, allowing the agent--infrastructure interaction to expose credentials or corrupt review judgments. Two configuration-file attacks replicate on GitLab and Hugging Face as well as GitHub. We evaluate workflow-level countermeasures and show their coverage and utility tradeoffs. We release GitInject and all attack scenarios.
Show more
MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Bowen Zheng ⋅ Jintao Lin ⋅ Chang-Heng Yi ⋅ HAORAN YANG ⋅ Binxiao Huang ⋅ Chenyang Lei ⋅ Rui Liu ⋅ Xihui Liu
Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose **MDMR-Bench**, a **M**ulti-**D**imensional benchmark for **M**ulti-**R**eference image generation. MDMR-Bench is designed based on realistic usage scenarios and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. Our dataset is available at [Kaggle](https://www.kaggle.com/datasets/6f695be8363ee17928c25f36fa976fd10f976f9f6d4925bc0df74f38dfce268c). The code is available at [GitHub](https://anonymous.4open.science/r/Neurips2026_ED_paper3412).
Show more
Do Composed Image Retrieval Benchmarks Require Multimodal Composition?
Matteo Attimonelli ⋅ Alessandro De Bellis ⋅ Aryo Gema ⋅ Rohit Saxena ⋅ Monica Sekoyan ⋅ Wai-Chung Kwan ⋅ Claudio Pomo ⋅ Alessandro Suglia ⋅ Dietmar Jannach ⋅ Tommaso Di Noia ⋅ Pasquale Minervini
Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is to retrieve a target image satisfying both. In principle, strong performance on CIR benchmarks is assumed to require multimodal composition, i.e., combining complementary information from reference image and textual modification. In this work, we show that this assumption does not always hold. Across four widely used CIR benchmarks and eleven Generalist Multimodal Embedding models, a large fraction of queries can be solved using a single modality (from 32.2% to 83.6%), revealing pervasive unimodal shortcuts. Thus, high CIR performance can arise from unimodal signals rather than true multimodal composition. To better understand this issue, we perform a two-stage audit. First, we identify shortcut-solvable queries through cross-model analysis. Second, we conduct human validation on 4,741 shortcut-free queries, of which only 1,689 are well-formed, with common issues including ambiguous edits and mismatched targets. Re-evaluating models on this validated subset reveals qualitatively different behaviour: queries can no longer be solved with a single modality, and successful retrieval requires combining both inputs. While accuracy decreases, reliance on multimodal information increases. Overall, current CIR benchmarks conflate shortcut-solvable, noisy, and genuinely compositional queries, leading to an overestimation of model capability in multimodal composition. Datasets and annotations are available here.
Show more
Beyond Success Rates: Trainability and Extractability in Offline GCRL
Jan Malte Töpperwien ⋅ Aditya Mohan ⋅ Marius Lindauer
Offline goal-conditioned reinforcement learning (GCRL) is typically benchmarked by the best tuned success rate of each method. This score measures attainable performance, but it does not reveal how reliably a learned goal-conditioned signal can be extracted into a policy: a method could succeed across many value-learning and extraction settings, or only at a narrow, hard-to-find configuration. We study this gap across four methods, GCIQL, GCIVL, QRL, and CRL, under a shared advantage-weighted regression (AWR) extractor. For each method, we construct trainability landscapes over the optimizer learning rate, which affects value learning and actor optimization, and AWR temperature, which controls how selectively the actor imitates high-advantage transitions. Across AntMaze, Cube, and Scene, we observe distinct regimes: high-scoring methods may be broadly accessible or brittle, while broad relative basins may still sit below low absolute ceilings. To interpret these differences, we pair landscapes with post-hoc diagnostics of future-vs-random goal discrimination and AWR weight concentration. % whether the learned advantage function separates matched future goals from random goals, and whether AWR spreads weight across many behavior-supported transitions or concentrates on a small tail. Their relationship to downstream success is task-dependent. On AntMaze, where future goals align with path-like progress, these diagnostics explain landscape regimes. On Cube and Scene, goal ranking and manipulation control decouple: methods can rank goals well while failing downstream, or succeed through action-conditioned advantages despite weak future-vs-random separation. These results show that peak tuned success alone does not establish broadly extractable goal-conditioned behavior. Trainability landscapes expose this gap, while extraction diagnostics offer a lower-cost lens on how learned signals become policies.
Show more
FineVision: Open Data Is All You Need
Luis Wiedmann ⋅ Orr Zohar ⋅ Amir Mahla ⋅ Xiaohan Wang ⋅ Rui Li ⋅ Thibaud Frere ⋅ Leandro Von Werra ⋅ Aritra Roy Gosthipaty ⋅ Andrés Marafioti
The advancement of vision-language models (VLMs) is hampered by a fragmented landscape of inconsistent and contaminated public datasets. We introduce FineVision, a meticulously collected, curated, and unified corpus of 24 million samples, the largest open resource of its kind. We unify more than 200 sources into 185 subsets via a semi-automated, human-in-the-loop pipeline: automation performs bulk ingestion and schema mapping, while reviewers audit the mappings and spot-check outputs from every subset to verify that source annotations are converted faithfully and formatted consistently; issues trigger targeted fixes and re-runs. The workflow further applies rigorous de-duplication within and across sources and decontamination against 66 public benchmarks. FineVision also encompasses agentic/GUI tasks with a unified action space; reviewers validate schemas and inspect a sample of trajectories to confirm executable fidelity. Models trained on FineVision consistently outperform those trained on existing open mixtures across a broad evaluation suite, underscoring the benefits of scale, data hygiene, and balanced automation with human oversight. We release the corpus and curation tools to accelerate data-centric VLM research.
Show more
Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games
Lorenzo Magnino ⋅ Jiacheng Shen ⋅ Matthieu Geist ⋅ Olivier Pietquin ⋅ Mathieu Lauriere
The rise of Mean Field Games (MFGs) has fostered a growing family of algorithms designed to solve large-scale multi-agent systems. However, the field currently lacks a standardized evaluation protocol, forcing researchers to rely on bespoke, isolated, and often simplistic environments. This fragmentation makes it difficult to assess the robustness, generalization, and failure modes of emerging methods. To address this gap, we propose a comprehensive benchmark suite for MFGs, focusing on the discrete-time, discrete-space, stationary settings. We introduce a taxonomy of problem classes, ranging from no-interaction and monotone games to potential and dynamics-coupled games, and provide prototypical environments for each. Furthermore, we present MF-Garnets, a method for generating random MFG instances to facilitate rigorous statistical testing. We benchmark a variety of learning algorithms across these environments, including a novel population-based approach (MF-PSO). Based on our knowledge, this is the first benchmark for MFGs.
Show more
MINEGRID: A Multi-Modal Benchmark for LLMs Evaluation on Dynamic Modeling of Power Systems
Duange Guo ⋅ Yujian Yuan ⋅ Shanyu Li ⋅ Xinhua Wu ⋅ Yulong Liu
Dynamic modeling and simulation of modern power systems are inefficient and error-prone due to the manual translation required for complex differential-algebraic equations (DAEs) and the reliance on information in unstructured, multimodal documents. Vision-language model (VLM) capable of synthesizing multimodal information from diagrams has the potential to automate this modeling process; however, it lacks specialized datasets to ensure physical consistency and structural accuracy and systematic frameworks to evaluate the performance. This paper presents \textbf{MINEGRID}, a first-of-its-kind multimodal benchmark for automating the conversion of power systems diagrams into DAE systems. The benchmark includes a comprehensive dataset of control diagrams paired with expert-verified LaTeX equations, JSON metadata, and grid diagrams paired with topology annotations, both stratified by physical and structural complexity. This benchmark also evaluates the generation progress from multimodal structures to precise DAE terms using metrics covering structural correctness, algebraic accuracy, and physical consistency. In the experiments, the proposed benchmark provides a comprehensive performance analysis across multiple state-of-the-art models to establish a foundational baseline for automated power system modeling, bridging the gap between multimodal document understanding and automated modeling of power systems. The dataset and the code are available at https://anonymous.4open.science/r/Minegrid-Benchmark-4280.
Show more
OpenWhistle: A Large-Scale Longitudinal Dataset and Benchmark of Bottlenose Dolphin Vocalizations
Faadil Mustun ⋅ Chiara Semenzin ⋅ Roberto Dessi ⋅ Pablo Robin Guerrero ⋅ pierre orhan ⋅ Alexis Emanuelli ⋅ Emanuele Rossi ⋅ Yair Lakretz ⋅ Gonzalo de Polavieja ⋅ Germán Sumbre
Recent advances in bioacoustics have been driven by large-scale corpora and standardized benchmarks, yet existing resources are overwhelmingly bird-centric and shallow per species, limiting their use for studying the structure of a single species' communication system. This gap is particularly acute for cetaceans: despite bottlenose dolphins (\textit{Tursiops truncatus}) being a compelling case of complex vocal communication among non-human mammals, existing dolphin datasets are small, fragmented, and largely closed. We introduce OpenWhistle, the largest publicly available dataset of dolphin vocalizations. It comprises approximately 180,000 whistles (114 hours) recorded over five years from a stable pod of five individuals in a semi-natural environment, paired with a curated subset of 8,354 expert-annotated whistles and reproducible evaluation protocols for whistle-type detection and classification. We further release the full processing pipeline for whistle detection, segmentation, and categorization. To demonstrate its utility, we pretrain a Wav2Vec2.0 model adapted to dolphin acoustics on the OpenWhistle corpus and show that it learns effective representations, outperforming general-purpose bioacoustic models such as AVES and BioLingual on both tasks while leaving meaningful headroom for future work. By releasing the dataset, pipeline, and evaluation protocol, we provide the first open dolphin whistle dataset tailored for training self-supervised models, laying the groundwork for advancing dolphin communication research and developing models that capture fine-grained acoustic structure within species.
Show more
Ledger: A Path-Validated, Database-Grounded Benchmark for Enterprise Web Agents
Xiao Yang ⋅ Mo Sha ⋅ Yiran Li ⋅ Chengjin Tian ⋅ Sheng Wang ⋅ Fangyuan Zhou ⋅ Feifei Li
Web agents promise to automate consequential operations on enterprise web systems, yet the benchmarks used to evaluate them score correctness on surface-level proxies: DOM/URL predicates, action-trace matchers, or vision-language verdicts on a closing screen. These proxies are inadequate for enterprise resource planning (ERP) systems, where every action commits to a transactional ledger and the criterion of correctness is what the ledger now records, not what the page now renders. We argue that ERP web operations should be evaluated against what a run actually wrote to the relational backend, with the server-side operation trace recorded alongside the outcome to provide the audit provenance that enterprise governance requires. We introduce Ledger, a path-validated, database-grounded benchmark for browser agents on a production-grade open-source ERP: 424 oracle-certified task instances across 53 functional modules, scored by an Atomic Edit Set validator (field-level $F_1$ against a PostgreSQL oracle, under a typed normalization function) paired with a Canonical Operation Tracer (server-side instrumentation of the ERP’s JSON-RPC dispatch). Each episode runs under pg_restore snapshot isolation; no language-model judge is invoked; three formal guarantees (score determinism, episode isolation, tracer completeness) make scores auditable and exactly replayable. A cross-paradigm evaluation of seven contemporary browser agents shows Ledger discriminates architectures sharply, with task-success rate spanning 81.6% down to 13.7% across the roster while long-horizon enterprise workflows remain far from saturated.
Show more
spora: A Unified Multimodal Dataset for Spatial Proteomics
Benedikt von Querfurth ⋅ Johann Wenckstern ⋅ Eeshaan Jain ⋅ Lukas Klein ⋅ Phil F Cheng ⋅ Yexiang Cheng ⋅ Cédric Vincent-Cuaz ⋅ Martina Haberecker ⋅ Pascal Frossard ⋅ Andreas Wicki ⋅ Olivier Michielin ⋅ Charlotte Bunne
Spatial proteomics (SP) captures the molecular composition and spatial organization of tissues at single-cell resolution, opening a direct view of the cellular ecosystems that drive disease. Protein panels, acquisition platforms, and tissue contexts differ from one cohort to the next, and a coherent picture of tissue biology can only emerge from models trained to bridge this heterogeneity. Foundation models offer the natural path forward, but their training depends on large, harmonized corpora that span platforms, panels, and cohorts. No public resource currently approaches that scale; available data sit in small, narrowly scoped releases with incompatible formats and conventions. We present spora, a large-scale multimodal SP dataset of 13,546 standardized multiplex images from 5,477 patients across 35 cohorts and five major SP technologies. Curated with biomedical experts, spora provides model-training-ready formats together with cell and tissue segmentations, structured clinical metadata, and, where available, paired H&E enabling the training of virtual staining models. It is complemented by spora [io], a unified interface for tiling, sampling, and standardization across all cohorts and modalities, and spora [bench], a benchmark suite spanning cell phenotyping, diagnostic and therapeutic prediction as well as virtual staining tasks across twelve cohorts that places current SP foundation models on common ground against task-specific baselines.
Show more
AutoExpert: Automating 3D LiDAR Annotation from Expert-Crafted Guidelines
Yechi Ma ⋅ Wei Hua ⋅ Shu Kong
The contemporary paradigm of scaling data annotation, crucial for developing machine learning solutions, is to hire ordinary human annotators and instruct them with expert-crafted guidelines to label data. This paradigm is laborious, tedious, and costly, motivating us to study an open problem, auto-annotation with expert-crafted guidelines (dubbed AutoExpert). We develop benchmarks by redesigning the evaluation protocol and re-annotating data with nuScenes and PandaSet, two 3D detection datasets for autonomous driving research that provide expert-crafted annotation guidelines. Their guidelines define 18 and 25 object classes, respectively, using nuanced language descriptions and a few visual examples. Following the guidelines that require using 3D cuboids to label LiDAR data, AutoExpert requires algorithms to learn on few-shot labeled images and texts to perform the task of 3D detection on LiDAR data. Apparently, the challenges of AutoExpert lie in the data-modality and task discrepancy. Nevertheless, public foundation models (FMs) serve as promising tools to tackle these challenges. To address AutoExpert, we adopt a conceptually simple pipeline consisting of three components: (1) 2D object detection and segmentation in RGB images, (2) lifting 2D detections into 3D using known sensor poses, and (3) 3D cuboids generation for the 2D detections. Within this pipeline, we enhance and evaluate a variety of methods such as open-vocabulary detectors, few-shot detectors, and self-supervised learned detectors. We also develop novel techniques, leading to refined components that boost 3D detection mAP from 12.1 to 25.4 on the AutoExpert-nuScenes benchmark.
Show more
CompleteRXN: Toward Completing Open Chemical Reaction Databases
Gabriel Vogel ⋅ Minouk Noordsij ⋅ Evgeny A Pidko ⋅ Jana M. Weber
Chemical reaction datasets such as USPTO suffer from substantial incompleteness, frequently missing byproducts, co-reactants, and stoichiometric coefficients. This limits their applicability and reliability in downstream applications. Here, we introduce CompleteRXN, a large-scale supervised benchmark for reaction completion under realistic missing-data conditions. We construct a dataset of aligned incomplete and atom-balanced reactions by mapping USPTO records to curated mechanistic reactions. We evaluate representative baselines, including a novel encoder-decoder reaction completion model with constrained decoding, the Constrained Reaction Balancer (CRB), and a recent algorithmic method, SynRBL. On our CompleteRXN benchmark, the CRB achieves high performance across splits of increasing difficulty, reaching 99.20% equivalence accuracy on the random split and 91.12% on the extreme out-of-distribution split. SynRBL produces many balanced and chemically plausible completions, but with lower accuracy on the benchmark test splits. Across all methods, performance degrades with increasing incompleteness. We observe a substantial drop when evaluating on reactions outside the benchmark (full uncurated USPTO), highlighting the gap between benchmark performance and practical robustness and motivating future work.
Show more
MutQA: A Cross-Validated Q&A Dataset for Genetic Mutations
Robert McCourt ⋅ Oladimeji Macaulay ⋅ David Arredondo ⋅ Luis E Tafoya ⋅ Oluoma Edeh ⋅ Kushal Virupakshappa ⋅ Yue Hu ⋅ My Nguyen Bach ⋅ Enrique A Ruiz ⋅ Shrey Poshiya ⋅ Debjani P Hudgens ⋅ Deepali Kundnani ⋅ Avinash Sahu
We introduce \textsc{MutQA}, a large-scale dataset of 330,387 citation-grounded question--answer pairs linking genetic mutations to their functional consequences as described in the biomedical literature. Each record pairs a protein-coding mutation with a natural-language question and an evidence-backed answer anchored to one of 27,442 PubMed articles, spanning 6,620 genes and 88,587 unique variants. The dataset is constructed using \textsc{CrossValQA}, a framework that enforces \emph{information isolation} between independent LLMs: one model generates QA pairs from an article, a second model answers the same question without access to the first model's response, and a third model judges agreement---ensuring that retained pairs are reproducible from the source text rather than hallucinated. This bidirectional cross-validation retains 82.7\% of generated pairs as \emph{cross-validated}. Human evaluation on 1{,}000 samples with two annotators ($\kappa{=}0.61$) confirms high factual fidelity. Fine-tuning \textsc{MutaPLM} on MutQA yields large gains across all three sequence-conditioned tasks: ROUGE-L on free-form mutation-effect description rises from $12.79$ to $22.99$, judge-mean correctness on question-aware answering nearly doubles ($1.45 \to 2.82$ on a 0--5 scale), and exact-match accuracy on inverse mutation recovery improves from $19.4\%$ to $32.1\%$. \textsc{CrossValQA} is domain-agnostic; alongside the genetic \textsc{MutQA} release we apply the same framework to proteomics as a proof of concept (Appendix~\ref{app:proteomics}). We release the dataset, construction framework, and evaluation code to support reproducible research in biomedical NLP and protein language modeling.
Show more
Hidden Positives: Why Code Retrieval Benchmarks Underestimate Model Quality
Maria Ivanova ⋅ Vitaly Kadulin ⋅ Dmitrii Babaev
Text-code embeddings underpin modern code intelligence systems, but progress in this area is held back by a systemic problem: code retrieval benchmarks routinely treat valid alternative implementations as negatives, making cross-model comparison unstable and depressing measured performance. We document this problem with a detailed analysis of six widely used benchmarks and show that, depending on the dataset, between 44\% and 87\% of items a strong LLM judge marks as relevant are missing from the original labels. To address this, we adopt a unified evaluation protocol in which a single LLM judge re-annotates the top-10 retrieved candidates for every model under a fixed prompt, replacing fragmented benchmark heuristics with a consistent semantic standard. Applying the same idea at training time, we use a smaller LLM-supervised reranker to construct MegaCode, the largest semantically curated dataset for code search to date of 193M positive (text, code) pairs, roughly an order of magnitude larger than prior curated corpora, and train a family of embedding models on it at four scales (0.5B-7B) with contrastive learning. Under a unified LLM-judged protocol, our 0.5B model matches or exceeds all prior open code-retrieval embeddings up to 8B parameters, and our 7B model improves average MRR@10 by 3.5 points over the strongest baseline.
Show more
Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability
Joël Roman Ky ⋅ Salah GHAMIZI ⋅ Maxime Cordy
Vision-Language Models (VLMs) map complex visual inputs to semantic spaces, but interpreting the cross-modal reasoning of VLMs currently relies on post-hoc explainers evaluated via unimodal perturbation metrics. We expose a limitation in this paradigm: because multimodal datasets contain language priors and modality biases, VLMs frequently exhibit cross-modal redundancy, allowing them to answer visual queries using text alone. Consequently, unimodal metrics penalize faithful explainers, triggering an evaluation collapse where visual and textual rankings fundamentally contradict each other. To resolve this, we introduce Synergistic Faithfulness ($\mathcal{F}_{syn}$), a scalable metric rooted in the Shapley Interaction Index that strictly isolates the joint Harsanyi dividend between modalities, serving as a highly accurate surrogate ($\rho = 0.92$) while achieving a $24\times$ computational speedup. Evaluating 8 distinct XAI methods across 3 VLM architectures and 3 benchmark datasets, reveals that explainers proposed for VLMs heavily over-index on visual salience and significantly underperform adapted attention-based methods in capturing true cross-modal synergy. By decoupling visual plausibility from cross-modal faithfulness, this work provides a rigorous evaluation framework required to safely audit VLM reasoning in high-stakes deployments.
Show more
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu ⋅ Jiarui Hu ⋅ Xingyu Jiang ⋅ Pengyu Zou ⋅ Han Li ⋅ Chao Peng ⋅ Peter O'Hearn ⋅ Earl Barr ⋅ Mark Harman ⋅ Federica Sarro ⋅ He Ye
We introduce TerminalWorld, a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from "in-the-wild" terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 19 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 representative, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld-Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks (e.g., Terminal-Bench), with only a weak correlation to their scores (Pearson $r=0.20$). The automated engine makes TerminalWorld authentic and scalable by construction, enabling it to evaluate agents in real-world terminal environments as developer practices evolve. Data and code are available at https://github.com/EuniAI/TerminalWorld.
Show more
ORBIT: A Framework for Multi-Agent Security Evaluations
Ben Hagag ⋅ William L Anderson ⋅ Srija Chakraborty ⋅ Christian Schroeder de Witt
Multi-agent LLM systems are increasingly deployed for complex, long-horizon tasks or emerge as a natural consequence of agents interacting in the wild. Yet, they often have significant security vulnerabilities: the flexible protocols that enable task generalization also expose novel threats, from cascading prompt injection to inter-agent collusion. Progress in defending against these threats has been slowed by a lack of empirical evaluations, requiring bespoke environment development for every new defense and making standardized comparison impossible. Existing evaluations address isolated threat models or single-agent settings, but none jointly vary attack, defense, and architecture across realistic multi-agent environments. To address this gap, we introduce ORBIT, a configurable evaluation framework for empirical multi-agent security research, built on UK AISI's Inspect. ORBIT provides configurable communication topologies, memory, scheduling, and agent roles. It supports four threat types (indirect prompt injection, misuse, compromised agents, collusion) and four defense strategies (security prompting, guardian agents, monitors, dual-LLM patterns). The benchmark suite spans five scenario families covering browser use, computer use, agentic coding, customer service, and cooperative allocation. Using ORBIT, we conduct controlled experiments varying defenses, attacks, topologies, and models. We find evidence of significant gaps in defense transferability across threats, security–performance tradeoffs, and interactions between architectural choices and defense effectiveness. To accelerate empirical multi-agent security work, we make ORBIT available open-source at: https://anonymous.4open.science/r/orbit-D1A9.
Show more
PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios
Chris Zhu ⋅ Sasha Cui ⋅ Will S Dufallo ⋅ Runzhi Jin ⋅ Zhen Xu ⋅ Linjun Zhang ⋅ Daylian M Cain
We present an in-depth evaluation of LLMs' ability to negotiate, a central business task requiring strategic reasoning, theory of mind, and economic value creation. To do so, we introduce **PieArena**, a large-scale negotiation benchmark grounded in multi-agent interactions over realistic scenarios adapted from MBA negotiation courses at an elite business school. We evaluate language agents across three pairing regimes: mirror-play, cross-play, and human–LM play. We develop a ranking model for continuous negotiation payoffs that yields order-invariant, uncertainty-quantified leaderboards while correcting for systematic experimental asymmetries. We further study the effects of joint-intentionality agentic scaffolding and find asymmetric gains, with large improvements for mid- and lower-tier LMs and diminishing returns for frontier LMs. As calibration anchors, we collect human–human and human–LM negotiation data from trained business school students, finding that a representative frontier language agent (GPT-5) matches or exceeds this human baseline in our evaluation settings. Beyond deal outcomes, PieArena provides a multi-dimensional behavioral profile that reveals cross-model heterogeneity in instruction compliance, computation accuracy, as well as judge-assessed deception and reputation, illustrating the value of evaluation beyond outcome-only leaderboards.
Show more
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
ZHAOYUE SUN ⋅ Hainiu Xu ⋅ Andero Uusberg ⋅ James Gross ⋅ Petr Slovak ⋅ Yulan He
Emotion understanding is a core capability for LLMs to interact effectively with humans, yet existing evaluation paradigms rely on discrete emotion label prediction and fail to capture the cognitive processes underlying emotion generation. Grounded in appraisal theory, we introduce CAREBench, the first benchmark with complete inferential chain annotations from both first- and third-person perspectives on real-world narratives, spanning appraisal reasoning, appraisal ratings, and multi-label emotion annotation. We propose a process-level evaluation framework and conduct systematic experiments across six LLMs organized around four research questions. We find that stronger models match or surpass human observers on certain tasks, yet fall short on appraisal reasoning and positive emotion recognition; performance across chain steps and sensitivity to appraisal interventions exhibit dissociations across models; and current models have not internalized the mechanisms needed to capture human subjective heterogeneity. These findings suggest that downstream emotion prediction metrics may overestimate LLMs' true emotion understanding, and CAREBench provides a foundation for more diagnostically informative evaluation of LLMs' affective cognitive capabilities.
Show more
BESS-Bench: Benchmarking Spectral Representations for Be-Star Variability
Matthieu Le Lain ⋅ Sébastien Lefèvre
Time-resolved stellar spectroscopy is a primary observational window onto mass-loss, disc formation and rotational physics, phenomena that single-epoch surveys cannot capture. Be stars are rapidly rotating B-type stars that episodically grow and lose a circumstellar gas disc, which imprints a variable emission profile on the hydrogen H-alpha Balmer line (6562.8 Angstrom) on timescales from days to decades. Decoding these spectral variations is both a key step in understanding stellar evolution and an open challenge for machine learning on irregularly sampled scientific time series. Yet no benchmark exists for these spectra: professional surveys deliver a single epoch per star, and the amateur-driven BeSS database, although it hosts hundreds of thousands of multi-epoch spectra, has never been curated for machine learning. We close this gap with BESS-Bench: 339,115 optical spectra of 1,468 Be stars over 35 years, all expert-reviewed, from amateurs (81.7%) and professionals (17.7%), with a scored H-alpha benchmark slice of 26,937 spectra and three tasks under a unified evaluation protocol. SpecProbe measures how well frozen embeddings recover six scalar line features (width, peak separation, line-core depth, asymmetry, equivalent width, peak intensity). LineTransfer tests cross-line generalisation (H-beta to H-alpha). EWForecast predicts the next-epoch equivalent width of H-alpha (a disc-mass proxy) from a star's recent spectroscopic history. We release a compact masked auto-encoder baseline (BeMAE) and benchmark it against PCA and two zero-shot time-series foundation models (Chronos-Bolt and TimesFM-2.0). The ranking is sharply task-dependent: BeMAE recovers shape features with an R2 that is 4.24x that of PCA(10) (bootstrap CI95 [4.12, 4.33]), yet a PCA+ridge pipeline lowers EW forecasting MAE by 5.8% over persistence, beating both foundation models, and calling as such for further development of AI models in this context. Dataset, weights, frozen splits and evaluation pipeline are released under CC-BY-4.0, so the scientific community can adopt BESS-Bench as a shared testbed and contribute new representations and models to advance our understanding of stellar variability.
Show more
TurtLES: A Large-Scale Benchmark for Turbulent 3D Neural PDE Surrogates
Jarno Platenburg ⋅ Armand Kassaï Koupaï ⋅ Paola Cinnella ⋅ Patrick Gallinari
Machine-learning surrogates are increasingly used to accelerate computational fluid dynamics (CFD), yet progress is limited by the lack of benchmarks capturing realistic, time-dependent turbulent flows. We introduce TURTLES, a 13 TB dataset of high-fidelity implicit large-eddy simulations of three-dimensional turbulent cylinder wakes in the shear-layer transition regime. The dataset contains 400 trajectories of 400 time steps each, with 3–9 million points per frame sampled on irregular point clouds, spanning variations in geometry, Reynolds number, and angle of attack. Unlike existing datasets, TURTLES captures fully three-dimensional turbulence with an active energy cascade driven by vortex stretching, combining (i) dense irregular meshes, (ii) long-horizon temporal dynamics, and (iii) physically realistic turbulent behavior. We benchmark state-of-the-art neural operators on long autoregressive rollouts and identify key failure modes, including temporal instability, loss of small-scale energy, and challenges in learning on large irregular domains. TURTLES provides a new standard benchmark for evaluating surrogate models in high-resolution, industrial-scale CFD regimes.
Show more
Seahorse: A Unified Benchmarking Framework for Spatiotemporal Event Modeling
Yahya Aalaila ⋅ Sebastian Vollmer ⋅ Gerrit Großmann
Spatiotemporal point processes (STPPs) model event data in continuous time and space, with applications in mobility, epidemiology, and public safety. Recent neural STPPs span expressive intensities, density models, continuous-time latent dynamics, normalizing-flow spatial decoders, and score-based generative mechanisms. Yet comparison remains fragile because implementations differ in preprocessing, coordinate normalization, splits, likelihood conventions, and evaluation protocols. We present \textsc{Seahorse}, a unified framework for reproducible STPP experimentation. \textsc{Seahorse} provides an encode--evolve--decode interface and an executable benchmarking contract that standardizes dataset handling, configuration resolution, training, tuning, checkpointing, raw-coordinate likelihood reporting, and run artifacts across heterogeneous STPP families. It supports classical baselines, neural likelihood models, continuous-time Neural STPP variants, and sample-based generative models under one external protocol. Using HawkesNest as a controlled synthetic stress-test suite, we show how diagnostic benchmarking can probe neural STPP behavior under increasing spatiotemporal entanglement and other structural changes. Together, \textsc{Seahorse} and these controlled evaluations provide reusable infrastructure for comparing, diagnosing, and extending STPP models beyond implementation-specific leaderboards.
Show more
Benchmarking Sensor-Fault Robustness in Forecasting
Alexander Windmann ⋅ Philipp Wittenberg ⋅ Gianluca Manca ⋅ Marcel Dix ⋅ Jens U. Brandt ⋅ Oliver Niggemann
Cyber-physical system (CPS) forecasting models depend on sensor streams with noisy, biased, missing, or temporally misaligned readings, yet standard forecasting evaluation often selects models by nominal error without showing whether they remain robust under such faults. We introduce SensorFault-Bench, a shared CPS-grounded sensor-fault stress-test protocol for evaluating forecasting architectures and robustness-improvement methods, together with an operational taxonomy that organizes the method comparison. Across four real-world datasets and eight scored scenarios governed by a standardized severity model, it reports worst-scenario degradation, clean mean squared error (MSE), and worst-scenario fault-time MSE, separating relative robustness from absolute error. A disjoint fault-transfer split lets explicit fault-training methods train on adjacent fault families while reported scores use separate benchmark scenarios. Empirically, forecasting architectures favored by clean MSE can degrade sharply under faults, and clean-MSE rankings can disagree with worst-scenario fault-time error rankings. Chronos-2, the evaluated zero-shot foundation-model representative, matches or trails the last-value naive forecaster in clean MSE on the two single-target datasets and has the largest worst-scenario degradation on ETTh1 and Traffic, where all channels are forecast targets. For the evaluated robustness-improvement method set, paired deltas show selective degradation reductions: projected gradient descent adversarial training and randomized training lead where value faults dominate observed degradation, while fault augmentation leads where availability faults dominate. SensorFault-Bench provides open-source code, documented data access, and reproduction and extension guides, so new datasets, architectures, and robustness-improvement methods can be evaluated under the same CPS sensor-fault robustness protocol.
Show more
POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents
Qiaoyuan ZHENG ⋅ yiqu yang ⋅ Qi Gao ⋅ Imanol Schlag
LLM agents increasingly have access to private user data and act on the user's behalf when interacting with third-party systems. The user defines what may and must not be shared, and the agent must robustly follow that intent even when third-party systems behave adversarially. We introduce $\textbf{POLAR-Bench}$ ($\textbf{Pol}$icy-$\textbf{a}$ware adve$\textbf{r}$sarial Benchmark), in which a trusted model with a privacy policy and a task converses with a third-party model that adversarially probes for both task-relevant and protected attributes. Across 10 domains and 7${,}$852 samples, we score privacy and utility by deterministic set-membership and vary privacy policy dimension and attack strategy along two orthogonal axes, producing a $5\times5$ diagnostic surface per model. Our results reveal a sharp split: current frontier models withhold over 99% of protected attributes, while smaller open-weight models in the 1--30B range, the class users most commonly run as their own trusted agent on-device or via private inference, score notably worse, with the weakest leaking over half. POLAR-Bench thus localizes where each model's intent-following breaks down, providing a foothold for privacy alignment where it matters most.
Show more
Successful Page Load