Timezone: »
Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts ("greet the pink brontosaurus by the ferris wheel"). Modern neural networks, by contrast, struggle to interpret novel compositions. In this paper, we introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding. Going beyond a related benchmark that focused on syntactic aspects of generalization, gSCAN defines a language grounded in the states of a grid world, facilitating novel evaluations of acquiring linguistically motivated rules. For example, agents must understand how adjectives such as 'small' are interpreted relative to the current world state or how adverbs such as 'cautiously' combine with new verbs. We test a strong multi-modal baseline model and a state-of-the-art compositional method finding that, in most cases, they fail dramatically when generalization requires systematic compositional rules.
Author Information
Laura Ruis (University of Amsterdam)
Jacob Andreas (MIT)
Marco Baroni (Facebook Artificial Intelligence Research)
Diane Bouchacourt (Facebook AI)
Brenden Lake (New York University)
More from the Same Authors
-
2020 Poster: Learning Compositional Rules via Neural Program Synthesis »
Maxwell Nye · Armando Solar-Lezama · Josh Tenenbaum · Brenden Lake -
2020 Poster: Mutual exclusivity as a challenge for deep neural networks »
Kanishk Gandhi · Brenden Lake -
2020 Poster: Self-supervised learning through the eyes of a child »
Emin Orhan · Vaibhav Gupta · Brenden Lake -
2020 Poster: Compositional Explanations of Neurons »
Jesse Mu · Jacob Andreas -
2020 Oral: Compositional Explanations of Neurons »
Jesse Mu · Jacob Andreas -
2019 Poster: Compositional generalization through meta sequence-to-sequence learning »
Brenden Lake -
2018 Poster: Speaker-Follower Models for Vision-and-Language Navigation »
Daniel Fried · Ronghang Hu · Volkan Cirik · Anna Rohrbach · Jacob Andreas · Louis-Philippe Morency · Taylor Berg-Kirkpatrick · Kate Saenko · Dan Klein · Trevor Darrell -
2017 Workshop: Cognitively Informed Artificial Intelligence: Insights From Natural Intelligence »
Michael Mozer · Brenden Lake · Angela Yu -
2017 Poster: Question Asking as Program Generation »
Anselm Rothe · Brenden Lake · Todd Gureckis -
2015 Poster: Softstar: Heuristic-Guided Probabilistic Inference »
Mathew Monfort · Brenden M Lake · Brenden Lake · Brian Ziebart · Patrick Lucey · Josh Tenenbaum -
2015 Poster: On the Accuracy of Self-Normalized Log-Linear Models »
Jacob Andreas · Maxim Rabinovich · Michael Jordan · Dan Klein -
2014 Poster: Unsupervised Transcription of Piano Music »
Taylor Berg-Kirkpatrick · Jacob Andreas · Dan Klein -
2014 Spotlight: Unsupervised Transcription of Piano Music »
Taylor Berg-Kirkpatrick · Jacob Andreas · Dan Klein