Timezone: »
The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking under real-world conditions. We further propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is more appropriate for evaluating sequences of arbitrary length. Finally, we provide several baselines to evaluate the status of existing methods on this new challenging dataset. We have made our datasets, metric, benchmark servers, and baselines publicly available, and hope this will inspire future research.
Author Information
Mark Weber (Technical University Munich)
Jun Xie
Maxwell Collins (Google Inc.)
Yukun Zhu (University of Toronto)
Paul Voigtlaender (RWTH Aachen University)
Hartwig Adam (Google)
Bradley Green (Google AI)
Andreas Geiger (MPI Tübingen)
Bastian Leibe (RWTH Aachen University-)
Daniel Cremers (Technical University of Munich)
Aljosa Osep (TUM Munich)
Laura Leal-Taixé (TUM)
Liang-Chieh Chen (Google Inc.)
More from the Same Authors
-
2021 : DENETHOR: The DynamicEarthNET dataset for Harmonized, inter-Operable, analysis-Ready, daily crop monitoring from space »
Lukas Kondmann · Aysim Toker · Marc Rußwurm · Andrés Camero · Devis Peressuti · Grega Milcinski · Pierre-Philippe Mathieu · Nicolas Longepe · Timothy Davis · Giovanni Marchisio · Laura Leal-Taixé · Xiaoxiang Zhu -
2022 Poster: What Makes Graph Neural Networks Miscalibrated? »
Hans Hao-Hsun Hsu · Yuesong Shen · Christian Tomani · Daniel Cremers -
2022 Poster: Deep Combinatorial Aggregation »
Yuesong Shen · Daniel Cremers -
2022 : PolarMOT: How Far Can Geometric Relations Take Us in 3D Multi-Object Tracking? »
Aleksandr Kim · Guillem Braso · Aljosa Osep · Laura Leal-Taixé -
2022 : PolarMOT: How far can geometric relations take us in 3D multi-object tracking? »
Aleksandr Kim · Guillem Braso · Aljosa Osep · Laura Leal-Taixé -
2022 : A Graph Is More Than Its Nodes: Towards Structured Uncertainty-Aware Learning on Graphs »
Hans Hao-Hsun Hsu · Yuesong Shen · Daniel Cremers -
2022 : Improving Zero-shot Generalization and Robustness of Multi-modal Models »
Yunhao Ge · Jie Ren · Ming-Hsuan Yang · Yuxiao Wang · Andrew Gallagher · Hartwig Adam · Laurent Itti · Balaji Lakshminarayanan · Jiaping Zhao -
2023 Poster: DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model »
Xiuye Gu · Yin Cui · Jonathan Huang · Abdullah Rashwan · Xuan Yang · Xingyi Zhou · Golnaz Ghiasi · Weicheng Kuo · Huizhong Chen · Liang-Chieh Chen · David Ross -
2023 Poster: ReMaX: Relaxing for Better Training on Efficient Panoptic Segmentation »
Shuyang Sun · Weijun Wang · Andrew Howard · Qihang Yu · Philip Torr · Liang-Chieh Chen -
2023 Poster: FC-CLIP: Open-Vocabulary Panoptic Segmentation with a Single Frozen Convolutional CLIP »
Qihang Yu · Ju He · Xueqing Deng · Xiaohui Shen · Liang-Chieh Chen -
2022 Spotlight: Deep Combinatorial Aggregation »
Yuesong Shen · Daniel Cremers -
2022 Spotlight: Lightning Talks 3B-1 »
Tianying Ji · Tongda Xu · Giulia Denevi · Aibek Alanov · Martin Wistuba · Wei Zhang · Yuesong Shen · Massimiliano Pontil · Vadim Titov · Yan Wang · Yu Luo · Daniel Cremers · Yanjun Han · Arlind Kadra · Dailan He · Josif Grabocka · Zhengyuan Zhou · Fuchun Sun · Carlo Ciliberto · Dmitry Vetrov · Mingxuan Jing · Chenjian Gao · Aaron Flores · Tsachy Weissman · Han Gao · Fengxiang He · Kunzan Liu · Wenbing Huang · Hongwei Qin -
2022 Spotlight: What Makes Graph Neural Networks Miscalibrated? »
Hans Hao-Hsun Hsu · Yuesong Shen · Christian Tomani · Daniel Cremers -
2022 Spotlight: Lightning Talks 1B-1 »
Qitian Wu · Runlin Lei · Rongqin Chen · Luca Pinchetti · Yangze Zhou · Abhinav Kumar · Hans Hao-Hsun Hsu · Wentao Zhao · Chenhao Tan · Zhen Wang · Shenghui Zhang · Yuesong Shen · Tommaso Salvatori · Gitta Kutyniok · Zenan Li · Amit Sharma · Leong Hou U · Yordan Yordanov · Christian Tomani · Bruno Ribeiro · Yaliang Li · David P Wipf · Daniel Cremers · Bolin Ding · Beren Millidge · Ye Li · Yuhang Song · Junchi Yan · Zhewei Wei · Thomas Lukasiewicz -
2022 Poster: Learning to Discover and Detect Objects »
Vladimir Fomenko · Ismail Elezi · Deva Ramanan · Laura Leal-Taixé · Aljosa Osep -
2022 Poster: Quo Vadis: Is Trajectory Forecasting the Key Towards Long-Term Multi-Object Tracking? »
Patrick Dendorfer · Vladimir Yugay · Aljosa Osep · Laura Leal-Taixé -
2022 Poster: The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes »
Peter Kocsis · Peter Súkeník · Guillem Braso · Matthias Niessner · Laura Leal-Taixé · Ismail Elezi -
2021 Poster: On the Frequency Bias of Generative Models »
Katja Schwarz · Yiyi Liao · Andreas Geiger -
2021 Oral: Shape As Points: A Differentiable Poisson Solver »
Songyou Peng · Chiyu Jiang · Yiyi Liao · Michael Niemeyer · Marc Pollefeys · Andreas Geiger -
2021 Poster: ATISS: Autoregressive Transformers for Indoor Scene Synthesis »
Despoina Paschalidou · Amlan Kar · Maria Shugrina · Karsten Kreis · Andreas Geiger · Sanja Fidler -
2021 Poster: Shape As Points: A Differentiable Poisson Solver »
Songyou Peng · Chiyu Jiang · Yiyi Liao · Michael Niemeyer · Marc Pollefeys · Andreas Geiger -
2021 Poster: Sparse Quadratic Optimisation over the Stiefel Manifold with Application to Permutation Synchronisation »
Florian Bernard · Daniel Cremers · Johan Thunberg -
2021 Poster: Projected GANs Converge Faster »
Axel Sauer · Kashyap Chitta · Jens Müller · Andreas Geiger -
2021 Poster: MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Images »
Shaofei Wang · Marko Mihajlovic · Qianli Ma · Andreas Geiger · Siyu Tang -
2020 Poster: Make One-Shot Video Object Segmentation Efficient Again »
Tim Meinhardt · Laura Leal-Taixé -
2020 Poster: Deep Shells: Unsupervised Shape Correspondence with Optimal Transport »
Marvin Eisenberger · Aysim Toker · Laura Leal-Taixé · Daniel Cremers -
2018 Poster: Searching for Efficient Multi-Scale Architectures for Dense Image Prediction »
Liang-Chieh Chen · Maxwell Collins · Yukun Zhu · George Papandreou · Barret Zoph · Florian Schroff · Hartwig Adam · Jonathon Shlens -
2017 : Google Lens »
Hartwig Adam -
2017 Poster: The Numerics of GANs »
Lars Mescheder · Sebastian Nowozin · Andreas Geiger -
2017 Spotlight: The Numerics of GANs »
Lars Mescheder · Sebastian Nowozin · Andreas Geiger -
2016 Poster: Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images »
Vladimir Golkov · Marcin Skwark · Antonij Golkov · Alexey Dosovitskiy · Thomas Brox · Jens Meiler · Daniel Cremers -
2016 Oral: Protein contact prediction from amino acid co-evolution using convolutional networks for graph-valued images »
Vladimir Golkov · Marcin Skwark · Antonij Golkov · Alexey Dosovitskiy · Thomas Brox · Jens Meiler · Daniel Cremers -
2015 Poster: Skip-Thought Vectors »
Jamie Kiros · Yukun Zhu · Russ Salakhutdinov · Richard Zemel · Raquel Urtasun · Antonio Torralba · Sanja Fidler -
2015 Poster: 3D Object Proposals for Accurate Object Class Detection »
Xiaozhi Chen · Kaustav Kundu · Yukun Zhu · Andrew G Berneshawi · Huimin Ma · Sanja Fidler · Raquel Urtasun