Timezone: »

 
Poster
Multimodal Few-Shot Learning with Frozen Language Models
Maria Tsimpoukelli · Jacob L Menick · Serkan Cabi · S. M. Ali Eslami · Oriol Vinyals · Felix Hill

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @
in Poster Session 6 »

When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned image and caption data, we train a vision encoder to represent each image as a sequence of continuous embeddings, such that a pre-trained, frozen language model presented with this prefix generates the appropriate caption. The resulting system is a multimodal few-shot learner, with the surprising ability to learn a variety of new tasks when conditioned on examples, represented as a sequence of any number of interleaved image and text embeddings. We demonstrate that it can rapidly learn words for new objects and novel visual categories, do visual question-answering with only a handful of examples, and make use of outside knowledge, by measuring a single model on a variety of established and new benchmarks.

Keywords: Vision · Few Shot Learning

Author Information

Maria Tsimpoukelli (DeepMind)
Jacob L Menick (Google DeepMind)
Serkan Cabi (DeepMind)
S. M. Ali Eslami (DeepMind)
Oriol Vinyals (Google DeepMind)
Felix Hill (Deepmind)

More from the Same Authors