Timezone: »

 
Poster
Deep Fourier Up-Sampling
man zhou · Hu Yu · Jie Huang · Feng Zhao · Jinwei Gu · Chen Change Loy · Deyu Meng · Chongyi Li

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #604
in Poster Session 6 »

Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling. However, spatial up-sampling operators (e.g., interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency. In contrast, the Fourier domain is in accordance with the nature of global modeling according to the spectral convolution theorem. Unlike the spatial domain that easily performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property. In this study, we propose a theoretically feasible Deep Fourier Up-Sampling (FourierUp) to solve these issues. We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp's designs. FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks. Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp. Code will be publicly available.

Keywords: Image restoration · image de-raining · image de-hazing · image super-resolution
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

man zhou (University of Science and Technology of China)
Hu Yu (University of Science and Technology of China)
Jie Huang (University of Science and Technology of China)
Feng Zhao (University of Science and Technology of China)
Jinwei Gu (Nvidia)
Chen Change Loy (Nanyang Technological University)
Deyu Meng (Xi'an Jiaotong University)
Chongyi Li ( Nanyang Technological University)

More from the Same Authors