Timezone: »

Modulating early visual processing by language
Harm de Vries · Florian Strub · Jeremie Mary · Hugo Larochelle · Olivier Pietquin · Aaron Courville

Tue Dec 05 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #79

It is commonly assumed that language refers to high-level visual concepts while leaving low-level visual processing unaffected. This view dominates the current literature in computational models for language-vision tasks, where visual and linguistic inputs are mostly processed independently before being fused into a single representation. In this paper, we deviate from this classic pipeline and propose to modulate the \emph{entire visual processing} by a linguistic input. Specifically, we introduce Conditional Batch Normalization (CBN) as an efficient mechanism to modulate convolutional feature maps by a linguistic embedding. We apply CBN to a pre-trained Residual Network (ResNet), leading to the MODulatEd ResNet (\MRN) architecture, and show that this significantly improves strong baselines on two visual question answering tasks. Our ablation study confirms that modulating from the early stages of the visual processing is beneficial.

Author Information

Harm de Vries (Université de Montréal)
Florian Strub (University of Lille)
Jeremie Mary (INRIA / Univ. Lille)
Hugo Larochelle (Google Brain)
Olivier Pietquin (Google Research Brain Team)
Aaron Courville (U. Montreal)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors