Timezone: »

 
Spotlight
OmniVL: One Foundation Model for Image-Language and Video-Language Tasks
Junke Wang · Dongdong Chen · Zuxuan Wu · Chong Luo · Luowei Zhou · Yucheng Zhao · Yujia Xie · Ce Liu · Yu-Gang Jiang · Lu Yuan

Wed Dec 07 05:00 PM -- 07:00 PM (PST) @
in Lightning Talks 4B-4 »

This paper presents OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. It adopts a unified transformer-based visual encoder for both image and video inputs, and thus can perform joint image-language and video-language pretraining. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer (e.g., use image-language to help video-language). To this end, we propose a \emph{decoupled} joint pretraining of image-language and video-language to effectively decompose the vision-language modeling into spatial and temporal dimensions and obtain performance boost on both image and video tasks. Moreover, we introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label (e.g., image classification), video-label (e.g., video action recognition) data together, so that both supervised and noisily supervised pretraining data are utilized as much as possible. Without incurring extra task-specific adaptors, OmniVL can simultaneously support visual only tasks (e.g., image classification, video action recognition), cross-modal alignment tasks (e.g., image/video-text retrieval), and multi-modal understanding and generation tasks (e.g., image/video question answering, captioning). We evaluate OmniVL on a wide range of downstream tasks and achieve state-of-the-art or competitive results with similar model size and data scale.

Keywords: Vision-Language Pretraining · Unified Foundation Model

Author Information

Junke Wang (Fudan University)
Dongdong Chen (Microsoft Cloud AI)
Zuxuan Wu (Fudan University)
Chong Luo (MSRA)
Luowei Zhou (Microsoft)
Yucheng Zhao (University of Science and Technology of China)
Yujia Xie (Georgia Institute of Technology)
Ce Liu (Microsoft)
Yu-Gang Jiang (Fudan University)
Lu Yuan (Microsoft)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors