Timezone: »
Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.
Author Information
Vincent Sitzmann (Stanford University)
Vincent is an incoming Assistant Professor at MIT EECS, where he will lead the Scene Representation Group (scenerepresentations.org). Currently, he is a Postdoc at MIT's CSAIL with Josh Tenenbaum, Bill Freeman, and Fredo Durand. He finished his Ph.D. at Stanford University. His research interest lies in neural scene representations - the way neural networks learn to represent information on our world. His goal is to allow independent agents to reason about our world given visual observations, such as inferring a complete model of a scene with information on geometry, material, lighting etc. from only few observations, a task that is simple for humans, but currently impossible for AI.
Michael Zollhoefer (Facebook Reality Labs)
Gordon Wetzstein (Stanford University)
Related Events (a corresponding poster, oral, or spotlight)
-
2019 Poster: Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations »
Wed. Dec 11th 06:45 -- 08:45 PM Room East Exhibition Hall B + C #71
More from the Same Authors
-
2021 Spotlight: Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering »
Vincent Sitzmann · Semon Rezchikov · Bill Freeman · Josh Tenenbaum · Fredo Durand -
2021 : 3D Neural Scene Representations for Visuomotor Control »
Yunzhu Li · Shuang Li · Vincent Sitzmann · Pulkit Agrawal · Antonio Torralba -
2021 : 3D Neural Scene Representations for Visuomotor Control »
Yunzhu Li · Shuang Li · Vincent Sitzmann · Pulkit Agrawal · Antonio Torralba -
2022 : Learning Controllable Adaptive Simulation for Multi-scale Physics »
Tailin Wu · Takashi Maruyama · Qingqing Zhao · Gordon Wetzstein · Jure Leskovec -
2022 : Heterogeneous reconstruction of deformable atomic models in Cryo-EM »
Youssef Nashed · Ariana Peck · Julien Martel · Axel Levy · Bongjin Koo · Gordon Wetzstein · Nina Miolane · Daniel Ratner · Frederic Poitevin -
2023 Poster: Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision »
Ayush Tewari · Tianwei Yin · George Cazenavette · Semon Rezchikov · Josh Tenenbaum · Fredo Durand · Bill Freeman · Vincent Sitzmann -
2023 Poster: FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flow »
Cameron Smith · Yilun Du · Ayush Tewari · Vincent Sitzmann -
2022 Poster: NeuForm: Adaptive Overfitting for Neural Shape Editing »
Connor Lin · Niloy Mitra · Gordon Wetzstein · Leonidas Guibas · Paul Guerrero -
2022 Poster: Generative Neural Articulated Radiance Fields »
Alexander Bergman · Petr Kellnhofer · Wang Yifan · Eric Chan · David Lindell · Gordon Wetzstein -
2022 Poster: Amortized Inference for Heterogeneous Reconstruction in Cryo-EM »
Axel Levy · Gordon Wetzstein · Julien N.P Martel · Frederic Poitevin · Ellen Zhong -
2022 Poster: Decomposing NeRF for Editing via Feature Field Distillation »
Sosuke Kobayashi · Eiichi Matsumoto · Vincent Sitzmann -
2021 : 3D Neural Scene Representations for Visuomotor Control »
Yunzhu Li · Shuang Li · Vincent Sitzmann · Pulkit Agrawal · Antonio Torralba -
2021 : Gordon Wetzstein Talk »
Gordon Wetzstein -
2021 Poster: Learning Signal-Agnostic Manifolds of Neural Fields »
Yilun Du · Katie Collins · Josh Tenenbaum · Vincent Sitzmann -
2021 Poster: Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering »
Vincent Sitzmann · Semon Rezchikov · Bill Freeman · Josh Tenenbaum · Fredo Durand -
2021 Poster: A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose »
Shih-Yang Su · Frank Yu · Michael Zollhoefer · Helge Rhodin -
2021 Poster: Fast Training of Neural Lumigraph Representations using Meta Learning »
Alexander Bergman · Petr Kellnhofer · Gordon Wetzstein -
2020 Poster: Implicit Neural Representations with Periodic Activation Functions »
Vincent Sitzmann · Julien N.P Martel · Alexander Bergman · David Lindell · Gordon Wetzstein -
2020 Poster: MetaSDF: Meta-Learning Signed Distance Functions »
Vincent Sitzmann · Eric Chan · Richard Tucker · Noah Snavely · Gordon Wetzstein -
2020 Oral: Implicit Neural Representations with Periodic Activation Functions »
Vincent Sitzmann · Julien N.P Martel · Alexander Bergman · David Lindell · Gordon Wetzstein