Timezone: »
This paper introduces an approach to regularize 2.5D surface normal and depth predictions at each pixel given a single input image. The approach infers and reasons about the underlying 3D planar surfaces depicted in the image to snap predicted normals and depths to inferred planar surfaces, all while maintaining fine detail within objects. Our approach comprises two components: (i) a fourstream convolutional neural network (CNN) where depths, surface normals, and likelihoods of planar region and planar boundary are predicted at each pixel, followed by (ii) a dense conditional random field (DCRF) that integrates the four predictions such that the normals and depths are compatible with each other and regularized by the planar region and planar boundary information. The DCRF is formulated such that gradients can be passed to the surface normal and depth CNNs via backpropagation. In addition, we propose new planar wise metrics to evaluate geometry consistency within planar surfaces, which are more tightly related to dependent 3D editing applications. We show that our regularization yields a 30% relative improvement in planar consistency on the NYU v2 dataset.
Author Information
Peng Wang (UCLA)
Xiaohui Shen (Adobe Research)
Bryan Russell (Adobe)
Scott Cohen (Adobe Research)
Brian Price (Adobe)
Alan Yuille (JHU)
More from the Same Authors
-
2021 : Occluded Video Instance Segmentation: Dataset and ICCV 2021 Challenge »
Jiyang Qi · Yan Gao · Yao Hu · Xinggang Wang · Xiaoyu Liu · Xiang Bai · Serge Belongie · Alan Yuille · Philip Torr · Song Bai -
2021 : Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping »
prakhar kaushik · Adam Kortylewski · Alex Gain · Alan Yuille -
2022 : Volumetric Neural Human for Robust Pose Optimization via Analysis-by-synthesis »
Pengliang Ji · Angtian Wang · Yi Zhang · Adam Kortylewski · Alan Yuille -
2022 : Synthetic Tumors Make AI Segment Tumors Better »
Qixin Hu · Junfei Xiao · Alan Yuille · Zongwei Zhou -
2022 : Assembling Existing Labels from Public Datasets to\\Diagnose Novel Diseases: COVID-19 in Late 2019 »
Zengle Zhu · Mintong Kang · Alan Yuille · Zongwei Zhou -
2022 : Making Your First Choice: To Address Cold Start Problem in Vision Active Learning »
Liangyu Chen · Yutong Bai · Siyu Huang · Yongyi Lu · Bihan Wen · Alan Yuille · Zongwei Zhou -
2022 Poster: Monocular Dynamic View Synthesis: A Reality Check »
Hang Gao · Ruilong Li · Shubham Tulsiani · Bryan Russell · Angjoo Kanazawa -
2021 Poster: Glance-and-Gaze Vision Transformer »
Qihang Yu · Yingda Xia · Yutong Bai · Yongyi Lu · Alan Yuille · Wei Shen -
2021 Poster: Are Transformers more robust than CNNs? »
Yutong Bai · Jieru Mei · Alan Yuille · Cihang Xie -
2021 Poster: Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose »
Angtian Wang · Shenxiao Mei · Alan Yuille · Adam Kortylewski -
2019 Poster: Learning elementary structures for 3D shape generation and matching »
Theo Deprelle · Thibault Groueix · Matthew Fisher · Vladimir Kim · Bryan Russell · Mathieu Aubry -
2017 Poster: Label Distribution Learning Forests »
Wei Shen · KAI ZHAO · Yilu Guo · Alan Yuille -
2017 Poster: Predicting Scene Parsing and Motion Dynamics in the Future »
Xiaojie Jin · Huaxin Xiao · Xiaohui Shen · Jimei Yang · Zhe Lin · Yunpeng Chen · Zequn Jie · Jiashi Feng · Shuicheng Yan -
2016 Poster: Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images »
Junhua Mao · Jiajing Xu · Kevin Jing · Alan Yuille -
2014 Workshop: Modern Nonparametrics 3: Automating the Learning Pipeline »
Eric Xing · Mladen Kolar · Arthur Gretton · Samory Kpotufe · Han Liu · Zoltán Szabó · Alan Yuille · Andrew G Wilson · Ryan Tibshirani · Sasha Rakhlin · Damian Kozbur · Bharath Sriperumbudur · David Lopez-Paz · Kirthevasan Kandasamy · Francesco Orabona · Andreas Damianou · Wacha Bounliphone · Yanshuai Cao · Arijit Das · Yingzhen Yang · Giulia DeSalvo · Dmitry Storcheus · Roberto Valerio -
2014 Poster: Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations »
Xianjie Chen · Alan Yuille -
2014 Poster: Learning From Weakly Supervised Data by The Expectation Loss SVM (e-SVM) algorithm »
Jun Zhu · Junhua Mao · Alan Yuille -
2010 Poster: Gaussian sampling by local perturbations »
George Papandreou · Alan Yuille -
2010 Poster: Functional form of motion priors in human motion perception »
HongJing Lu · Tungyou Lin · Alan L Lee · Luminita Vese · Alan Yuille -
2010 Poster: A unified model of short-range and long-range motion perception »
Shuang Wu · Xuming He · HongJing Lu · Alan Yuille -
2009 Poster: Modeling the spacing effect in sequential category learning »
HongJing Lu · Matthew Weiden · Alan Yuille -
2008 Poster: A Hierarchical Image Model for Polynomial-Time 2D Parsing »
Long Zhu · Yuanhao Chen · Yuan Lin · Alan Yuille -
2008 Poster: Model selection and velocity estimation using novel priors for motion patterns »
Alan Yuille · Shuang Wu · HongJing Lu -
2008 Spotlight: A Hierarchical Image Model for Polynomial-Time 2D Parsing »
Long Zhu · Yuanhao Chen · Yuan Lin · Alan Yuille -
2008 Oral: Model selection and velocity estimation using novel priors for motion patterns »
Alan Yuille · Shuang Wu · HongJing Lu -
2007 Workshop: The Grammar of Vision: Probabilistic Grammar-Based Models for Visual Scene Understanding and Object Categorization »
Virginia Savova · Josh Tenenbaum · Leslie Kaelbling · Alan Yuille -
2007 Poster: The Noisy-Logical Distribution and its Application to Causal Inference »
Alan Yuille · HongJing Lu -
2007 Poster: Rapid Inference on a novel AND/OR graph: Detection, Segmentation and Parsing of Articulated Deformable Objects in Cluttered Backgrounds »
Yuanhao Chen · Long Zhu · Chenxi Lin · Alan Yuille · Hongjiang Zhang -
2006 Talk: Unsupervised Learning of a Probabilistic Grammar for Object Detection and Parsing »
Long Zhu · Yuanhao Chen · Alan Yuille -
2006 Poster: Unsupervised Learning of a Probabilistic Grammar for Object Detection and Parsing »
Long Zhu · Yuanhao Chen · Alan Yuille