Timezone: »
Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting with light, in a process called rendering. Enabling ML models to understand image formation might be key for generalization. However, due to an essential rasterization step involving discrete assignment operations, rendering pipelines are non-differentiable and thus largely inaccessible to gradient-based ML techniques. In this paper, we present DIB-Render, a novel rendering framework through which gradients can be analytically computed. Key to our approach is to view rasterization as a weighted interpolation, allowing image gradients to back-propagate through various standard vertex shaders within a single framework. Our approach supports optimizing over vertex positions, colors, normals, light directions and texture coordinates, and allows us to incorporate various well-known lighting models from graphics. We showcase our approach in two ML applications: single-image 3D object prediction, and 3D textured object generation, both trained using exclusively 2D supervision.
Author Information
Wenzheng Chen (University of Toronto)
Huan Ling (University of Toronto, NVIDIA)
Jun Gao (University of Toronto)
Edward Smith (McGill University)
Jaakko Lehtinen (NVIDIA Research; Aalto University)
Alec Jacobson (University of Toronto)
Sanja Fidler (University of Toronto)
More from the Same Authors
-
2023 Poster: For SALE: State-Action Representation Learning for Deep Reinforcement Learning »
Scott Fujimoto · Wei-Di Chang · Edward Smith · Shixiang (Shane) Gu · Doina Precup · David Meger -
2022 Spotlight: GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images »
Jun Gao · Tianchang Shen · Zian Wang · Wenzheng Chen · Kangxue Yin · Daiqing Li · Or Litany · Zan Gojcic · Sanja Fidler -
2022 Poster: VectorAdam for Rotation Equivariant Geometry Optimization »
Selena Zihan Ling · Nicholas Sharp · Alec Jacobson -
2022 Poster: GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images »
Jun Gao · Tianchang Shen · Zian Wang · Wenzheng Chen · Kangxue Yin · Daiqing Li · Or Litany · Zan Gojcic · Sanja Fidler -
2022 Poster: Breaking Bad: A Dataset for Geometric Fracture and Reassembly »
Silvia Sellán · Yun-Chun Chen · Ziyi Wu · Animesh Garg · Alec Jacobson -
2021 Poster: Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis »
Tianchang Shen · Jun Gao · Kangxue Yin · Ming-Yu Liu · Sanja Fidler -
2021 Poster: Scalable Neural Data Server: A Data Recommender for Transfer Learning »
Tianshi Cao · Sasha (Alexandre) Doubov · David Acuna · Sanja Fidler -
2021 Poster: DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer »
Wenzheng Chen · Joey Litalien · Jun Gao · Zian Wang · Clement Fuji Tsang · Sameh Khamis · Or Litany · Sanja Fidler -
2021 Poster: EditGAN: High-Precision Semantic Image Editing »
Huan Ling · Karsten Kreis · Daiqing Li · Seung Wook Kim · Antonio Torralba · Sanja Fidler -
2021 Poster: ATISS: Autoregressive Transformers for Indoor Scene Synthesis »
Despoina Paschalidou · Amlan Kar · Maria Shugrina · Karsten Kreis · Andreas Geiger · Sanja Fidler -
2021 Poster: Don’t Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence »
Tianshi Cao · Alex Bie · Arash Vahdat · Sanja Fidler · Karsten Kreis -
2021 Poster: Active 3D Shape Reconstruction from Vision and Touch »
Edward Smith · David Meger · Luis Pineda · Roberto Calandra · Jitendra Malik · Adriana Romero Soriano · Michal Drozdzal -
2021 Poster: Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation »
David Acuna · Jonah Philion · Sanja Fidler -
2020 : Sanja Fidler »
Sanja Fidler -
2020 Poster: Variational Amodal Object Completion »
Huan Ling · David Acuna · Karsten Kreis · Seung Wook Kim · Sanja Fidler -
2020 Poster: Learning Deformable Tetrahedral Meshes for 3D Reconstruction »
Jun Gao · Wenzheng Chen · Tommy Xiang · Alec Jacobson · Morgan McGuire · Sanja Fidler -
2020 Poster: 3D Shape Reconstruction from Vision and Touch »
Edward Smith · Roberto Calandra · Adriana Romero · Georgia Gkioxari · David Meger · Jitendra Malik · Michal Drozdzal -
2019 : Carl Doersch, Raquel Urtasun, Sanja Fidler moderated by Natalia Neverova »
Raquel Urtasun · Sanja Fidler · Natalia Neverova · Ilija Radosavovic · Carl Doersch -
2019 : Sanja Fidler - TBA »
Sanja Fidler -
2019 : Panel »
Sanja Fidler · Josh Tenenbaum · Tatiana López-Guevara · Danilo Jimenez Rezende · Niloy Mitra -
2019 : Sanja Fidler »
Sanja Fidler -
2019 Demonstration: Toronto Annotation Suite »
Amlan Kar · Sanja Fidler · Jun Gao · Seung Wook Kim · Huan Ling -
2018 Poster: A Neural Compositional Paradigm for Image Captioning »
Bo Dai · Sanja Fidler · Dahua Lin -
2018 Poster: Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation »
Edward Smith · Scott Fujimoto · David Meger -
2017 : Panel Discussion »
Felix Hill · Olivier Pietquin · Jack Gallant · Raymond Mooney · Sanja Fidler · Chen Yu · Devi Parikh -
2017 : Connecting high-level semantics with low-level vision »
Sanja Fidler -
2017 Poster: Teaching Machines to Describe Images with Natural Language Feedback »
Huan Ling · Sanja Fidler -
2016 Poster: Proximal Deep Structured Models »
Shenlong Wang · Sanja Fidler · Raquel Urtasun -
2015 Poster: Skip-Thought Vectors »
Jamie Kiros · Yukun Zhu · Russ Salakhutdinov · Richard Zemel · Raquel Urtasun · Antonio Torralba · Sanja Fidler -
2015 Poster: 3D Object Proposals for Accurate Object Class Detection »
Xiaozhi Chen · Kaustav Kundu · Yukun Zhu · Andrew G Berneshawi · Huimin Ma · Sanja Fidler · Raquel Urtasun -
2012 Poster: 3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model »
Sanja Fidler · Sven Dickinson · Raquel Urtasun -
2012 Spotlight: 3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model »
Sanja Fidler · Sven Dickinson · Raquel Urtasun -
2009 Poster: Evaluating multi-class learning strategies in a generative hierarchical framework for object detection »
Sanja Fidler · Marko Boben · Ales Leonardis