Separating Environment and Sensor Shift from Acquisition Protocol in Physical Vision Benchmarks

In real-world deployments, vision systems must operate under varying environmental conditions, such as illumination changes. Recent work has explored adaptive sensing strategies that select camera parameters according to the current environment and downstream task to improve task performance. Reliable physical benchmarks are therefore essential for evaluating such strategies. We study ImageNet-ES-Diverse, a physical benchmark that evaluates the adaptive exposure control strategy \textit{Lens} by recapturing printed ImageNet images under varying illumination conditions. We construct four controlled acquisition pipelines that vary printed-image scale and camera zoom, and conduct a pilot study with directly captured physical objects. We find that the acquisition choices substantially change benchmark difficulty on all downstream baselines. Moreover, the advantage of \textit{Lens} over auto exposure observed on the original benchmark does not persist in either our collected pipelines or real-object pilot study. Oracle exposure selection nevertheless retains substantial gains, indicating remaining potential for sensor control. These findings highlight acquisition protocol as an important factor in physical benchmark evaluation and motivate explicit protocol reporting and evaluation across multiple acquisition setups when assessing adaptive-sensing benefits.