Timezone: »

 
Poster
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners
Zhenhailong Wang · Manling Li · Ruochen Xu · Luowei Zhou · Jie Lei · Xudong Lin · Shuohang Wang · Ziyi Yang · Chenguang Zhu · Derek Hoiem · Shih-Fu Chang · Mohit Bansal · Heng Ji

Wed Nov 30 02:00 PM -- 04:00 PM (PST) @ Hall J #1041
in Poster Session 4 »

The goal of this work is to build flexible video-language models that can generalize to various video-to-text tasks from few examples. Existing few-shot video-language learners focus exclusively on the encoder, resulting in the absence of a video-to-text decoder to handle generative tasks. Video captioners have been pretrained on large-scale video-language datasets, but they rely heavily on finetuning and lack the ability to generate text for unseen tasks in a few-shot setting. We propose VidIL, a few-shot Video-language Learner via Image and Language models, which demonstrates strong performance on few-shot video-to-text tasks without the necessity of pretraining or finetuning on any video datasets. We use image-language models to translate the video content into frame captions, object, attribute, and event phrases, and compose them into a temporal-aware template. We then instruct a language model, with a prompt containing a few in-context examples, to generate a target output from the composed content. The flexibility of prompting allows the model to capture any form of text input, such as automatic speech recognition (ASR) transcripts. Our experiments demonstrate the power of language models in understanding videos on a wide variety of video-language tasks, including video captioning, video question answering, video caption retrieval, and video future event prediction. Especially, on video future event prediction, our few-shot model significantly outperforms state-of-the-art supervised models trained on large-scale video datasets.Code and processed data are publicly available for research purposes at https://github.com/MikeWangWZHL/VidIL.

Keywords: language model · in-context learning · video-language · few-shot
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Zhenhailong Wang (University of Illinois at Urbana-Champaign)
Zhenhailong Wang

Zhenhailong Wang is a second year Master Student in Computer Science at UIUC. He is a graduate research assistant working with Prof. Heng Ji. His current research interest lies in multimodal and multitask natural language processing.

Manling Li (University of Illinois, Urbana Champaign)
Ruochen Xu (Microsoft)
Luowei Zhou (Microsoft)
Jie Lei (Meta)
Xudong Lin (Columbia University)
Shuohang Wang (Microsoft)
Ziyi Yang (Stanford University)
Chenguang Zhu (Stanford University)
Derek Hoiem (University of Illinois)
Shih-Fu Chang (Columbia University)
Mohit Bansal (UNC Chapel Hill)
Heng Ji (University of Illinois)

More from the Same Authors