Timezone: »
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.
Author Information
Hongyi Xu (Google AI)
Thiemo Alldieck (Google)
Cristian Sminchisescu (Lund University/Google)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Spotlight: H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion »
Dates n/a. Room
More from the Same Authors
-
2021 Poster: Relative Flatness and Generalization »
Henning Petzka · Michael Kamp · Linara Adilova · Cristian Sminchisescu · Mario Boley -
2021 Poster: REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision »
Mihai Fieraru · Mihai Zanfir · Teodor Szente · Eduard Bazavan · Vlad Olaru · Cristian Sminchisescu -
2011 Poster: Probabilistic Joint Image Segmentation and Labeling »
Adrian Ion · Joao Carreira · Cristian Sminchisescu -
2011 Spotlight: Probabilistic Joint Image Segmentation and Labeling »
Adrian Ion · Joao Carreira · Cristian Sminchisescu -
2010 Poster: Convex Multiple-Instance Learning by Estimating Likelihood Ratio »
Fuxin Li · Cristian Sminchisescu -
2009 Poster: Efficient Match Kernel between Sets of Features for Visual Recognition »
Liefeng Bo · Cristian Sminchisescu -
2009 Spotlight: Efficient Match Kernel between Sets of Features for Visual Recognition »
Liefeng Bo · Cristian Sminchisescu -
2007 Poster: People Tracking with the Laplacian Eigenmaps Latent Variable Model »
Zhengdong Lu · Miguel A. Carreira-Perpinan · Cristian Sminchisescu