Timezone: »
We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Current work on scene representation learning either ignores scene background or treats the whole scene as one object. Meanwhile, work that considers scene compositionality treats scene objects only as image patches or 2D layers with alpha maps. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the whole scene, and finally render them into realistic images. This allows BlockGAN to reason over occlusion and interaction between objects’ appearance, such as shadow and lighting, and provides control over each object’s 3D pose and identity, while maintaining image realism. BlockGAN is trained end-to-end, using only unlabelled single images, without the need for 3D geometry, pose labels, object masks, or multiple views of the same scene. Our experiments show that using explicit 3D features to represent objects allows BlockGAN to learn disentangled representations both in terms of objects (foreground and background) and their properties (pose and identity).
Author Information
Thu Nguyen-Phuoc (University of Bath)
Christian Richardt (University of Bath)
Long Mai (Adobe Research)
Yongliang Yang (University of Bath)
Niloy Mitra (University College London)
More from the Same Authors
-
2023 Poster: PyNeRF: Pyramidal Neural Radiance Fields »
Haithem Turki · Michael Zollhöfer · Christian Richardt · Deva Ramanan -
2021 Poster: TöRF: Time-of-Flight Radiance Fields for Dynamic Scene View Synthesis »
Benjamin Attal · Eliot Laidlaw · Aaron Gokaslan · Changil Kim · Christian Richardt · James Tompkin · Matthew O'Toole -
2021 Poster: A Multi-Implicit Neural Representation for Fonts »
Pradyumna Reddy · Zhifei Zhang · Zhaowen Wang · Matthew Fisher · Hailin Jin · Niloy Mitra -
2021 Poster: SketchGen: Generating Constrained CAD Sketches »
Wamiq Para · Shariq Bhat · Paul Guerrero · Tom Kelly · Niloy Mitra · Leonidas Guibas · Peter Wonka -
2018 Poster: RenderNet: A deep convolutional network for differentiable rendering from 3D shapes »
Thu Nguyen-Phuoc · Chuan Li · Stephen Balaban · Yongliang Yang -
2018 Poster: Unsupervised Attention-guided Image-to-Image Translation »
Youssef Alami Mejjati · Christian Richardt · James Tompkin · Darren Cosker · Kwang In Kim