Timezone: »
The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking under real-world conditions. We further propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is more appropriate for evaluating sequences of arbitrary length. Finally, we provide several baselines to evaluate the status of existing methods on this new challenging dataset. We have made our datasets, metric, benchmark servers, and baselines publicly available, and hope this will inspire future research.
Author Information
Mark Weber (Technical University Munich)
Jun Xie
Maxwell Collins (Google Inc.)
Yukun Zhu (University of Toronto)
Paul Voigtlaender (RWTH Aachen University)
Hartwig Adam (Google)
Bradley Green (Google AI)
Andreas Geiger (MPI Tübingen)
Bastian Leibe (RWTH Aachen University-)
Daniel Cremers (Technical University of Munich)
Aljosa Osep (TUM Munich)
Laura Leal-Taixé (TUM)
Liang-Chieh Chen (Google Inc.)
More from the Same Authors
-
2021 : DENETHOR: The DynamicEarthNET dataset for Harmonized, inter-Operable, analysis-Ready, daily crop monitoring from space »
Lukas Kondmann · Aysim Toker · Marc Rußwurm · Andrés Camero · Devis Peressuti · Grega Milcinski · Pierre-Philippe Mathieu · Nicolas Longepe · Timothy Davis · Giovanni Marchisio · Laura Leal-Taixé · Xiaoxiang Zhu -
2021 Poster: On the Frequency Bias of Generative Models »
Katja Schwarz · Yiyi Liao · Andreas Geiger -
2021 Oral: Shape As Points: A Differentiable Poisson Solver »
Songyou Peng · Chiyu Jiang · Yiyi Liao · Michael Niemeyer · Marc Pollefeys · Andreas Geiger -
2021 Poster: ATISS: Autoregressive Transformers for Indoor Scene Synthesis »
Despoina Paschalidou · Amlan Kar · Maria Shugrina · Karsten Kreis · Andreas Geiger · Sanja Fidler -
2021 Poster: Shape As Points: A Differentiable Poisson Solver »
Songyou Peng · Chiyu Jiang · Yiyi Liao · Michael Niemeyer · Marc Pollefeys · Andreas Geiger -
2021 Poster: Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation »
Florian Bernard · Daniel Cremers · Johan Thunberg -
2021 Poster: Projected GANs Converge Faster »
Axel Sauer · Kashyap Chitta · Jens Müller · Andreas Geiger -
2021 Poster: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images »
Shaofei Wang · Marko Mihajlovic · Qianli Ma · Andreas Geiger · Siyu Tang -
2020 Poster: Make One-Shot Video Object Segmentation Efficient Again »
Tim Meinhardt · Laura Leal-Taixé -
2020 Poster: Deep Shells: Unsupervised Shape Correspondence with Optimal Transport »
Marvin Eisenberger · Aysim Toker · Laura Leal-Taixé · Daniel Cremers -
2018 Poster: Searching for Efficient Multi-Scale Architectures for Dense Image Prediction »
Liang-Chieh Chen · Maxwell Collins · Yukun Zhu · George Papandreou · Barret Zoph · Florian Schroff · Hartwig Adam · Jonathon Shlens -
2017 : Google Lens »
Hartwig Adam -
2017 Poster: The Numerics of GANs »
Lars Mescheder · Sebastian Nowozin · Andreas Geiger -
2017 Spotlight: The Numerics of GANs »
Lars Mescheder · Sebastian Nowozin · Andreas Geiger -
2016 Poster: Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images »
Vladimir Golkov · Marcin Skwark · Antonij Golkov · Alexey Dosovitskiy · Thomas Brox · Jens Meiler · Daniel Cremers -
2016 Oral: Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images »
Vladimir Golkov · Marcin Skwark · Antonij Golkov · Alexey Dosovitskiy · Thomas Brox · Jens Meiler · Daniel Cremers -
2015 Poster: Skip-Thought Vectors »
Jamie Kiros · Yukun Zhu · Russ Salakhutdinov · Richard Zemel · Raquel Urtasun · Antonio Torralba · Sanja Fidler -
2015 Poster: 3D Object Proposals for Accurate Object Class Detection »
Xiaozhi Chen · Kaustav Kundu · Yukun Zhu · Andrew G Berneshawi · Huimin Ma · Sanja Fidler · Raquel Urtasun