Timezone: »

 
Poster
Deep Multimodal Fusion by Channel Exchanging
Yikai Wang · Wenbing Huang · Fuchun Sun · Tingyang Xu · Yu Rong · Junzhou Huang

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #206
in Poster Session 2 »

Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications. Yet, current methods including aggregation-based and alignment-based fusion are still inadequate in balancing the trade-off between inter-modal fusion and intra-modal processing, incurring a bottleneck of performance improvement. To this end, this paper proposes Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

Author Information

Yikai Wang (Tsinghua University)
Wenbing Huang (Tsinghua University)
Fuchun Sun (Tsinghua)
Tingyang Xu (Tencent AI Lab)
Yu Rong (Tencent AI Lab)
Junzhou Huang (University of Texas at Arlington / Tencent AI Lab)

More from the Same Authors