Timezone: »
Access to external knowledge is essential for many natural language processing tasks, such as question answering and dialogue. Existing methods often rely on a parametric model that stores knowledge in its parameters, or use a retrieval-augmented model that has access to an external knowledge source. Parametric and retrieval-augmented models have complementary strengths in terms of computational efficiency and predictive accuracy. To combine the strength of both approaches, we propose the Efficient Memory-Augmented Transformer (EMAT) – it encodes external knowledge into a key-value memory and exploits the fast maximum inner product search for memory querying. Experiments on various knowledge-intensive tasks such as question answering and dialogue datasets show that, simply augmenting parametric models (T5-base) using our method produces more accurate results while retaining a high throughput. Compared to retrieval-augmented models, EMAT runs substantially faster across the board and produces more accurate results on WoW and ELI5.
Author Information
Yuxiang Wu (University College London)
Yu Zhao (Harbin Institute of Technology, Shenzhen)
Baotian Hu (Harbin Institute of Technology, Shenzhen)
Pasquale Minervini (University College London)
Pontus Lars Erik Saito Stenetorp (University of Tokyo)
Sebastian Riedel (UCL)
More from the Same Authors
-
2022 : Discrete Learning Of DAGs Via Backpropagation »
Andrew Wren · Pasquale Minervini · Luca Franceschi · Valentina Zantedeschi -
2022 : Discrete Learning Of DAGs Via Backpropagation »
Andrew Wren · Pasquale Minervini · Luca Franceschi · Valentina Zantedeschi -
2023 Poster: On Efficient Training Algorithms For Transformer Language Models »
Jean Kaddour · Oscar Key · Piotr Nawrot · Pasquale Minervini · Matt Kusner -
2023 Poster: Improving Language Plasticity via Pretraining with Active Forgetting »
Yihong Chen · Mikel Artetxe · Kelly Marchisio · Roberta Raileanu · David Adelani · Pontus Lars Erik Saito Stenetorp · Sebastian Riedel -
2023 Poster: Adapting Neural Link Predictors for Efficient Complex Query Answering »
Erik Arakelyan · Pasquale Minervini · Daniel Daza · Michael Cochez · Isabelle Augenstein -
2022 : Discrete Learning Of DAGs Via Backpropagation »
Andrew Wren · Pasquale Minervini · Luca Franceschi · Valentina Zantedeschi -
2022 Poster: ReFactor GNNs: Revisiting Factorisation-based Models from a Message-Passing Perspective »
Yihong Chen · Pushkar Mishra · Luca Franceschi · Pasquale Minervini · Pontus Lars Erik Saito Stenetorp · Sebastian Riedel -
2021 Poster: Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions »
Mathias Niepert · Pasquale Minervini · Luca Franceschi -
2020 Workshop: HAMLETS: Human And Model in the Loop Evaluation and Training Strategies »
Divyansh Kaushik · Bhargavi Paranjape · Forough Arabshahi · Yanai Elazar · Yixin Nie · Max Bartolo · Polina Kirichenko · Pontus Lars Erik Saito Stenetorp · Mohit Bansal · Zachary Lipton · Douwe Kiela