WayPOP: A Panoramic Open-Set Panoptic Tracking Benchmark

Autonomous driving perception increasingly relies on panoramic multi-camera setups for holistic, spatiotemporal scene understanding. However, existing panoptic tracking benchmarks assume a fixed semantic taxonomy, failing to evaluate whether systems can robustly detect, segment, and track out-of-distribution (OOD) objects across time and multiple viewpoints. Creating a realistic open-set benchmark for this setting is challenging: fully synthetic data suffers from domain gaps, while 2D augmentations lack multi-view and temporal geometric consistency. To address this gap, we introduce WayPOP, the first benchmark for multi-view open-set panoptic tracking (MV-OSPT) in autonomous driving. WayPOP augments real-world sequences from the Waymo PVPS dataset with diverse 3D assets using a geometry-grounded pipeline. By leveraging actual LiDAR geometry, out-of-taxonomy objects are seamlessly integrated with realistic scales, multi-view projections, and depth-aware occlusions across five synchronized cameras and time. We formalize a structured task hierarchy for MV-OSPT and adapt established metrics to separately evaluate known and unknown regions for both segmentation and tracking. Our comprehensive benchmarking of representative baselines reveals that maintaining consistent unknown-object identities during camera transitions, viewpoint changes, and temporal occlusions remains a critical open challenge.