Timezone: »
Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learning capabilities. We perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as visual question-answering, where the model is prompted with a question which it has to answer, captioning tasks, which evaluate the ability to describe a scene or an event, and close-ended tasks such as multiple-choice visual question-answering. For tasks lying anywhere on this spectrum, a single Flamingo model can achieve a new state of the art with few-shot learning, simply by prompting the model with task-specific examples. On numerous benchmarks, Flamingo outperforms models fine-tuned on thousands of times more task-specific data.
Author Information
Jean-Baptiste Alayrac (DeepMind)
Jeff Donahue (DeepMind)
Pauline Luc (Deepmind)
Antoine Miech (DeepMind)
Iain Barr (Deepmind)
Yana Hasson (DeepMind)
Karel Lenc (DeepMind)
Arthur Mensch (DeepMind)
Katherine Millican (DeepMind)
Malcolm Reynolds (DeepMind)
Roman Ring (DeepMind)
Eliza Rutherford (University of Oxford)
Serkan Cabi (DeepMind)
Tengda Han (University of Oxford)
Zhitao Gong (Auburn University)
Sina Samangooei (Five)
Marianne Monteiro (Universidade Federal de Campina Grande)
Jacob L Menick (Google DeepMind)
Sebastian Borgeaud (DeepMind)
Andy Brock (DeepMind)
Aida Nematzadeh (DeepMind)
Sahand Sharifzadeh (DeepMind)
Sahand is a research scientist at DeepMind. He did his Ph.D at LMU Munich under Prof. Volker Tresp focusing on developing a model of semantic memory and studying it's role in perception. In particular he studied the role of top-down processes and symbol grounding in computer vision.
Mikołaj Bińkowski (DeepMind S2)
Ricardo Barreira
Oriol Vinyals (DeepMind)
Oriol Vinyals is a Research Scientist at Google. He works in deep learning with the Google Brain team. Oriol holds a Ph.D. in EECS from University of California, Berkeley, and a Masters degree from University of California, San Diego. He is a recipient of the 2011 Microsoft Research PhD Fellowship. He was an early adopter of the new deep learning wave at Berkeley, and in his thesis he focused on non-convex optimization and recurrent neural networks. At Google Brain he continues working on his areas of interest, which include artificial intelligence, with particular emphasis on machine learning, language, and vision.
Andrew Zisserman (DeepMind & University of Oxford)
Karén Simonyan (Inflection AI)
More from the Same Authors
-
2021 : PASS: An ImageNet replacement for self-supervised pretraining without humans »
Yuki Asano · Christian Rupprecht · Andrew Zisserman · Andrea Vedaldi -
2021 : Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents »
Jane Wang · Michael King · Nicolas Porcel · Zeb Kurth-Nelson · Tina Zhu · Charles Deck · Peter Choy · Mary Cassin · Malcolm Reynolds · Francis Song · Gavin Buttimore · David Reichert · Neil Rabinowitz · Loic Matthey · Demis Hassabis · Alexander Lerchner · Matt Botvinick -
2021 : A Step Towards Efficient Evaluation of Complex Perception Tasks in Simulation »
Jonathan Sadeghi · Blaine Rogers · Sina Samangooei · Puneet Dokania · John Redford -
2021 : PASS: An ImageNet replacement for self-supervised pretraining without humans »
Yuki Asano · Christian Rupprecht · Andrew Zisserman · Andrea Vedaldi -
2021 : 3D Spinal Column Segmentation with Single Plane 2D-Projected Annotations »
Rhydian Windsor · Amir Jamaludin · Timor Kadir · Andrew Zisserman -
2022 Poster: Segmenting Moving Objects via an Object-Centric Layered Representation »
Junyu Xie · Weidi Xie · Andrew Zisserman -
2022 : Intra-Event Aware Imitation Game for Fast Detector Simulation »
Hosein Hashemi · Nikolai Hartmann · Sahand Sharifzadeh · James Kahn · Thomas Kuhr -
2022 : Probing Representations of Numbers in Vision and Language Models »
Ivana Kajic · Aida Nematzadeh -
2023 Poster: Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion »
Yash Bhalgat · Iro Laina · João Henriques · Andrea Vedaldi · Andrew Zisserman -
2023 Poster: Improving Category Discovery When No Representation Rules Them All »
Sagar Vaze · Andrea Vedaldi · Andrew Zisserman -
2023 Poster: Perception Test: A Diagnostic Benchmark for Multimodal Video Models »
Viorica Patraucean · Lucas Smaira · Ankush Gupta · Adria Recasens · Larisa Markeeva · Dylan Banarse · Skanda Koppula · joseph heyward · Mateusz Malinowski · Yi Yang · Carl Doersch · Tatiana Matejovicova · Yury Sulsky · Antoine Miech · Alexandre Fréchette · Hanna Klimczak · Raphael Koster · Junlin Zhang · Stephanie Winkler · Yusuf Aytar · Simon Osindero · Dima Damen · Andrew Zisserman · Joao Carreira -
2023 Workshop: 4th Workshop on Self-Supervised Learning: Theory and Practice »
Tengda Han · Ishan Misra · Pengtao Xie · Mathilde Caron · Hilde Kuehne -
2022 Spotlight: Lightning Talks 6A-3 »
Junyu Xie · Chengliang Zhong · Ali Ayub · Sravanti Addepalli · Harsh Rangwani · Jiapeng Tang · Yuchen Rao · Zhiying Jiang · Yuqi Wang · Xingzhe He · Gene Chou · Ilya Chugunov · Samyak Jain · Yuntao Chen · Weidi Xie · Sumukh K Aithal · Carter Fendley · Lev Markhasin · Yiqin Dai · Peixing You · Bastian Wandt · Yinyu Nie · Helge Rhodin · Felix Heide · Ji Xin · Angela Dai · Andrew Zisserman · Bi Wang · Xiaoxue Chen · Mayank Mishra · ZHAO-XIANG ZHANG · Venkatesh Babu R · Justus Thies · Ming Li · Hao Zhao · Venkatesh Babu R · Jimmy Lin · Fuchun Sun · Matthias Niessner · Guyue Zhou · Xiaodong Mu · Chuang Gan · Wenbing Huang -
2022 Spotlight: Segmenting Moving Objects via an Object-Centric Layered Representation »
Junyu Xie · Weidi Xie · Andrew Zisserman -
2022 Panel: Panel 3C-3: PyramidCLIP: Hierarchical Feature… & Flamingo: a Visual… »
Jeff Donahue · Yuting Gao -
2022 : Aida Nematzadeh: On Evaluating Neural Representations »
Aida Nematzadeh -
2022 Workshop: Self-Supervised Learning: Theory and Practice »
Ishan Misra · Pengtao Xie · Gul Varol · Yale Song · Yuki Asano · Xiaolong Wang · Pauline Luc -
2022 Poster: An empirical analysis of compute-optimal large language model training »
Jordan Hoffmann · Sebastian Borgeaud · Arthur Mensch · Elena Buchatskaya · Trevor Cai · Eliza Rutherford · Diego de Las Casas · Lisa Anne Hendricks · Johannes Welbl · Aidan Clark · Thomas Hennigan · Eric Noland · Katherine Millican · George van den Driessche · Bogdan Damoc · Aurelia Guy · Simon Osindero · Karén Simonyan · Erich Elsen · Oriol Vinyals · Jack Rae · Laurent Sifre -
2022 Poster: Associating Objects and Their Effects in Video through Coordination Games »
Erika Lu · Forrester Cole · Weidi Xie · Tali Dekel · Bill Freeman · Andrew Zisserman · Michael Rubinstein -
2022 Poster: TAP-Vid: A Benchmark for Tracking Any Point in a Video »
Carl Doersch · Ankush Gupta · Larisa Markeeva · Adria Recasens · Lucas Smaira · Yusuf Aytar · Joao Carreira · Andrew Zisserman · Yi Yang -
2022 Poster: Zero-Shot Video Question Answering via Frozen Bidirectional Language Models »
Antoine Yang · Antoine Miech · Josef Sivic · Ivan Laptev · Cordelia Schmid -
2021 Workshop: Meaning in Context: Pragmatic Communication in Humans and Machines »
Jennifer Hu · Noga Zaslavsky · Aida Nematzadeh · Michael Franke · Roger Levy · Noah Goodman -
2021 : Opening remarks »
Jennifer Hu · Noga Zaslavsky · Aida Nematzadeh · Michael Franke · Roger Levy · Noah Goodman -
2021 Poster: Attention over Learned Object Embeddings Enables Complex Visual Reasoning »
David Ding · Felix Hill · Adam Santoro · Malcolm Reynolds · Matt Botvinick -
2021 Poster: Multimodal Few-Shot Learning with Frozen Language Models »
Maria Tsimpoukelli · Jacob L Menick · Serkan Cabi · S. M. Ali Eslami · Oriol Vinyals · Felix Hill -
2021 : Live Q&A session: Oriol Vinyals (DeepMind) »
Oriol Vinyals -
2021 : Invited Talk: Oriol Vinyals (DeepMind) »
Oriol Vinyals -
2021 Oral: Attention over Learned Object Embeddings Enables Complex Visual Reasoning »
David Ding · Felix Hill · Adam Santoro · Malcolm Reynolds · Matt Botvinick -
2021 Panel: The Consequences of Massive Scaling in Machine Learning »
Noah Goodman · Melanie Mitchell · Joelle Pineau · Oriol Vinyals · Jared Kaplan -
2020 : QA: Oriol Vinyals »
Oriol Vinyals -
2020 : Invited Talk: Oriol Vinyals »
Oriol Vinyals -
2020 Poster: Evolving Normalization-Activation Layers »
Hanxiao Liu · Andy Brock · Karen Simonyan · Quoc V Le -
2020 Spotlight: Evolving Normalization-Activation Layers »
Hanxiao Liu · Andy Brock · Karen Simonyan · Quoc V Le -
2020 Poster: Pointer Graph Networks »
Petar Veličković · Lars Buesing · Matthew Overlan · Razvan Pascanu · Oriol Vinyals · Charles Blundell -
2020 Poster: CrossTransformers: spatially-aware few-shot transfer »
Carl Doersch · Ankush Gupta · Andrew Zisserman -
2020 Spotlight: Pointer Graph Networks »
Petar Veličković · Lars Buesing · Matthew Overlan · Razvan Pascanu · Oriol Vinyals · Charles Blundell -
2020 Session: Orals & Spotlights Track 28: Deep Learning »
Oriol Vinyals · Guido Montufar -
2020 Poster: Self-supervised Co-Training for Video Representation Learning »
Tengda Han · Weidi Xie · Andrew Zisserman -
2020 Poster: Self-Supervised MultiModal Versatile Networks »
Jean-Baptiste Alayrac · Adria Recasens · Rosalia Schneider · Relja Arandjelović · Jason Ramapuram · Jeffrey De Fauw · Lucas Smaira · Sander Dieleman · Andrew Zisserman -
2020 Poster: Training Generative Adversarial Networks by Solving Ordinary Differential Equations »
Chongli Qin · Yan Wu · Jost Tobias Springenberg · Andy Brock · Jeff Donahue · Timothy Lillicrap · Pushmeet Kohli -
2020 Spotlight: Training Generative Adversarial Networks by Solving Ordinary Differential Equations »
Chongli Qin · Yan Wu · Jost Tobias Springenberg · Andy Brock · Jeff Donahue · Timothy Lillicrap · Pushmeet Kohli -
2020 : Keynote Presentation I Oriol Vinyals »
Oriol Vinyals -
2019 : Grandmaster Level in StarCraft II using Multi-Agent Reinforcement Learning - Invited Talk »
Oriol Vinyals -
2019 : The MineRL competition »
Misa Ogura · Joe Booth · Sophia Sun · Nicholay Topin · Brandon Houghton · William Guss · Stephanie Milani · Oriol Vinyals · Katja Hofmann · JIA KIM · Karolis Ramanauskas · Florian Laurent · Daichi Nishio · Anssi Kanervisto · Alexey Skrynnik · Artemij Amiranashvili · Christian Scheller · KAIXIN WANG · Yanick Schraner -
2019 Poster: Are Labels Required for Improving Adversarial Robustness? »
Jean-Baptiste Alayrac · Jonathan Uesato · Po-Sen Huang · Alhussein Fawzi · Robert Stanforth · Pushmeet Kohli -
2019 Poster: Generating Diverse High-Fidelity Images with VQ-VAE-2 »
Ali Razavi · Aaron van den Oord · Oriol Vinyals -
2019 Poster: Unsupervised Learning of Object Keypoints for Perception and Control »
Tejas Kulkarni · Ankush Gupta · Catalin Ionescu · Sebastian Borgeaud · Malcolm Reynolds · Andrew Zisserman · Volodymyr Mnih -
2019 Poster: Classification Accuracy Score for Conditional Generative Models »
Suman Ravuri · Oriol Vinyals -
2019 Poster: Sim2real transfer learning for 3D human pose estimation: motion to the rescue »
Carl Doersch · Andrew Zisserman -
2018 Poster: Learning to Navigate in Cities Without a Map »
Piotr Mirowski · Matt Grimes · Mateusz Malinowski · Karl Moritz Hermann · Keith Anderson · Denis Teplyashin · Karen Simonyan · koray kavukcuoglu · Andrew Zisserman · Raia Hadsell -
2018 Poster: A flexible model for training action localization with varying levels of supervision »
Guilhem Chéron · Jean-Baptiste Alayrac · Ivan Laptev · Cordelia Schmid -
2018 Poster: On gradient regularizers for MMD GANs »
Michael Arbel · Danica J. Sutherland · Mikołaj Bińkowski · Arthur Gretton -
2018 Poster: Relational recurrent neural networks »
Adam Santoro · Ryan Faulkner · David Raposo · Jack Rae · Mike Chrzanowski · Theophane Weber · Daan Wierstra · Oriol Vinyals · Razvan Pascanu · Timothy Lillicrap -
2017 : Meta Unsupervised Learning »
Oriol Vinyals -
2017 : Evaluating the capacity to reason about beliefs »
Aida Nematzadeh -
2017 Workshop: Deep Learning: Bridging Theory and Practice »
Sanjeev Arora · Maithra Raghu · Russ Salakhutdinov · Ludwig Schmidt · Oriol Vinyals -
2017 : Distilling Expensive Simulations with Neural Networks »
Oriol Vinyals -
2017 Poster: Imagination-Augmented Agents for Deep Reinforcement Learning »
Sébastien Racanière · Theophane Weber · David Reichert · Lars Buesing · Arthur Guez · Danilo Jimenez Rezende · Adrià Puigdomènech Badia · Oriol Vinyals · Nicolas Heess · Yujia Li · Razvan Pascanu · Peter Battaglia · Demis Hassabis · David Silver · Daan Wierstra -
2017 Oral: Imagination-Augmented Agents for Deep Reinforcement Learning »
Sébastien Racanière · Theophane Weber · David Reichert · Lars Buesing · Arthur Guez · Danilo Jimenez Rezende · Adrià Puigdomènech Badia · Oriol Vinyals · Nicolas Heess · Yujia Li · Razvan Pascanu · Peter Battaglia · Demis Hassabis · David Silver · Daan Wierstra -
2017 Poster: Neural Discrete Representation Learning »
Aaron van den Oord · Oriol Vinyals · koray kavukcuoglu -
2017 Tutorial: Deep Learning: Practice and Trends »
Nando de Freitas · Scott Reed · Oriol Vinyals -
2016 Poster: Conditional Image Generation with PixelCNN Decoders »
Aaron van den Oord · Nal Kalchbrenner · Lasse Espeholt · koray kavukcuoglu · Oriol Vinyals · Alex Graves -
2016 Poster: An Online Sequence-to-Sequence Model Using Partial Conditioning »
Navdeep Jaitly · Quoc V Le · Oriol Vinyals · Ilya Sutskever · David Sussillo · Samy Bengio -
2016 Poster: Strategic Attentive Writer for Learning Macro-Actions »
Alexander (Sasha) Vezhnevets · Volodymyr Mnih · Simon Osindero · Alex Graves · Oriol Vinyals · John Agapiou · koray kavukcuoglu -
2016 Poster: Matching Networks for One Shot Learning »
Oriol Vinyals · Charles Blundell · Timothy Lillicrap · koray kavukcuoglu · Daan Wierstra -
2015 Poster: Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks »
Samy Bengio · Oriol Vinyals · Navdeep Jaitly · Noam Shazeer -
2015 Poster: Spatial Transformer Networks »
Max Jaderberg · Karen Simonyan · Andrew Zisserman · koray kavukcuoglu -
2015 Poster: Pointer Networks »
Oriol Vinyals · Meire Fortunato · Navdeep Jaitly -
2015 Spotlight: Pointer Networks »
Oriol Vinyals · Meire Fortunato · Navdeep Jaitly -
2015 Spotlight: Spatial Transformer Networks »
Max Jaderberg · Karen Simonyan · Andrew Zisserman · koray kavukcuoglu -
2015 Poster: Grammar as a Foreign Language »
Oriol Vinyals · Łukasz Kaiser · Terry Koo · Slav Petrov · Ilya Sutskever · Geoffrey Hinton -
2015 Tutorial: Large-Scale Distributed Systems for Training Neural Networks »
Jeff Dean · Oriol Vinyals -
2014 Poster: Two-Stream Convolutional Networks for Action Recognition in Videos »
Karen Simonyan · Andrew Zisserman -
2014 Spotlight: Two-Stream Convolutional Networks for Action Recognition in Videos »
Karen Simonyan · Andrew Zisserman -
2013 Poster: Deep Fisher Networks for Large-Scale Image Classification »
Karen Simonyan · Andrea Vedaldi · Andrew Zisserman -
2013 Spotlight: Deep Fisher Networks for Large-Scale Image Classification »
Karen Simonyan · Andrea Vedaldi · Andrew Zisserman -
2011 Poster: Pylon Model for Semantic Segmentation »
Victor Lempitsky · Andrea Vedaldi · Andrew Zisserman -
2010 Poster: Simultaneous Object Detection and Ranking with Weak Supervision »
Matthew B Blaschko · Andrea Vedaldi · Andrew Zisserman -
2010 Spotlight: Learning To Count Objects in Images »
Victor Lempitsky · Andrew Zisserman -
2010 Poster: Learning To Count Objects in Images »
Victor Lempitsky · Andrew Zisserman -
2009 Poster: Segmenting Scenes by Matching Image Composites »
Bryan C Russell · Alexei A Efros · Josef Sivic · Bill Freeman · Andrew Zisserman -
2009 Poster: Structured output regression for detection with partial truncation »
Andrea Vedaldi · Andrew Zisserman -
2008 Poster: SDL: Supervised Dictionary Learning »
Julien Mairal · Francis Bach · Jean A Ponce · Guillermo Sapiro · Andrew Zisserman -
2007 Spotlight: Learning Visual Attributes »
Vittorio Ferrari · Andrew Zisserman -
2007 Poster: Learning Visual Attributes »
Vittorio Ferrari · Andrew Zisserman -
2006 Poster: Bayesian Image Super-resolution, Continued »
Lyndsey C Pickup · David Capel · Stephen J Roberts · Andrew Zisserman -
2006 Spotlight: Bayesian Image Super-resolution, Continued »
Lyndsey C Pickup · David Capel · Stephen J Roberts · Andrew Zisserman