QuantiPhy Challenge: Quantitative Physical Reasoning for VLMs

Quantitative physical reasoning---estimating object sizes, velocities, and accelerations from video---is essential for robotics, autonomous systems, biomechanics, and scientific video analysis, yet remains poorly evaluated for modern Vision--Language Models (VLMs). The QuantiPhy Challenge introduces the first large-scale community benchmark for kinematic inference from natural-language questions and physical priors, derived from the QuantiPhy benchmark: given a video, a question, a physical prior, and a target unit, participants predict a single numerical answer for size, velocity, or acceleration across four diagnostic categories. Submissions are CSV files scored against a hidden test set using Mean Relative Accuracy (MRA). Two tracks (a Main Track admitting any system, including closed-source APIs, and an Open-Weight Track restricted to publicly available models), a ready-to-run starting kit, and diagnostic input-faithfulness analyses target novel methods for scale estimation, motion measurement, and physically grounded multimodal reasoning, and expose how today's strongest VLMs use---or fail to use---visual evidence and provided priors.