Timezone: »

 
Poster
COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
Simon Ging · Mohammadreza Zolfaghari · Hamed Pirsiavash · Thomas Brox

Tue Dec 08 09:00 PM -- 11:00 PM (PST) @ Poster Session 2 #630
in Poster Session 3 »

Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters.

Author Information

Simon Ging (University of Freiburg)
Mohammadreza Zolfaghari (University of Freiburg)
Hamed Pirsiavash (University of Maryland, Baltimore County)
Thomas Brox (University of Freiburg)

More from the Same Authors