Timezone: »
In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.
Author Information
Yi Tay (Google Brain)
Vinh Tran (Google)
Mostafa Dehghani (Google Brain)
Jianmo Ni (Google)
Dara Bahri (Google AI)
Harsh Mehta (Google Research)
Zhen Qin (Google)
Kai Hui (Google)
Zhe Zhao (Google)
Jai Gupta (Indian Institute of Technology Kharagpur)
Tal Schuster (Google Research)
William Cohen (Google AI)
Donald Metzler (Google)
More from the Same Authors
-
2021 : Programming Puzzles »
Tal Schuster · Ashwin Kalyan · Alex Polozov · Adam Kalai -
2021 Spotlight: Efficiently Identifying Task Groupings for Multi-Task Learning »
Chris Fifty · Ehsan Amid · Zhe Zhao · Tianhe Yu · Rohan Anil · Chelsea Finn -
2021 : Consistent Accelerated Inference via Confident Adaptive Transformers »
Tal Schuster · Adam Fisch · Tommi Jaakkola · Regina Barzilay -
2021 : An Empirical Study of Pre-trained Models on Out-of-distribution Generalization »
Yaodong Yu · Heinrich Jiang · Dara Bahri · Hossein Mobahi · Seungyeon Kim · Ankit Rawat · Andreas Veit · Yi Ma -
2023 Poster: Subject-driven Text-to-Image Generation via Apprenticeship Learning »
wenhu chen · Hexiang Hu · Yandong Li · Nataniel Ruiz · Xuhui Jia · Ming-Wei Chang · William Cohen -
2023 Poster: Mechanic: A Learning Rate Tuner »
Ashok Cutkosky · Aaron Defazio · Harsh Mehta -
2023 Poster: Learning List-Level Domain-Invariant Representations for Ranking »
Ruicheng Xian · Honglei Zhuang · Zhen Qin · Hamed Zamani · Jing Lu · Ji Ma · Kai Hui · Han Zhao · Xuanhui Wang · Michael Bendersky -
2023 Poster: Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution »
Mostafa Dehghani · Basil Mustafa · Josip Djolonga · Jonathan Heek · Matthias Minderer · Mathilde Caron · Andreas Steiner · Joan Puigcerver · Robert Geirhos · Ibrahim Alabdulmohsin · Avital Oliver · Piotr Padlewski · Alexey Gritsenko · Mario Lucic · Neil Houlsby -
2023 Poster: Recommender Systems with Generative Retrieval »
Shashank Rajput · Nikhil Mehta · Anima Singh · Raghunandan Hulikal Keshavan · Trung Vu · Lukasz Heldt · Lichan Hong · Yi Tay · Vinh Tran · Jonah Samost · Maciej Kula · Ed Chi · Maheswaran Sathiamoorthy -
2023 Poster: Sharpness-Aware Minimization Leads to Low-Rank Features »
Maksym Andriushchenko · Dara Bahri · Hossein Mobahi · Nicolas Flammarion -
2023 Poster: RD-Suite: A Benchmark for Ranking Distillation »
Zhen Qin · Rolf Jagerman · Rama Kumar Pasumarthi · Honglei Zhuang · He Zhang · Aijun Bai · Kai Hui · Le Yan · Xuanhui Wang -
2022 Spotlight: Improving Multi-Task Generalization via Regularizing Spurious Correlation »
Ziniu Hu · Zhe Zhao · Xinyang Yi · Tiansheng Yao · Lichan Hong · Yizhou Sun · Ed Chi -
2022 Panel: Panel 2C-6: Compositional Generalization in… & Confident Adaptive Language… »
Tal Schuster · Zhenlin Xu -
2022 Poster: Improving Multi-Task Generalization via Regularizing Spurious Correlation »
Ziniu Hu · Zhe Zhao · Xinyang Yi · Tiansheng Yao · Lichan Hong · Yizhou Sun · Ed Chi -
2022 Poster: Confident Adaptive Language Modeling »
Tal Schuster · Adam Fisch · Jai Gupta · Mostafa Dehghani · Dara Bahri · Vinh Tran · Yi Tay · Donald Metzler -
2021 : Consistent Accelerated Inference via Confident Adaptive Transformers »
Tal Schuster · Adam Fisch · Tommi Jaakkola · Regina Barzilay -
2021 Oral: High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails »
Ashok Cutkosky · Harsh Mehta -
2021 Poster: DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning »
Hussein Hazimeh · Zhe Zhao · Aakanksha Chowdhery · Maheswaran Sathiamoorthy · Yihua Chen · Rahul Mazumder · Lichan Hong · Ed Chi -
2021 : Programming Puzzles »
Tal Schuster · Ashwin Kalyan · Alex Polozov · Adam Kalai -
2021 Poster: High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails »
Ashok Cutkosky · Harsh Mehta -
2021 Poster: Efficiently Identifying Task Groupings for Multi-Task Learning »
Chris Fifty · Ehsan Amid · Zhe Zhao · Tianhe Yu · Rohan Anil · Chelsea Finn -
2020 Poster: Faithful Embeddings for Knowledge Base Queries »
Haitian Sun · Andrew Arnold · Tania Bedrax Weiss · Fernando Pereira · William Cohen -
2019 : Invited Talk (William W. Cohen) »
William Cohen -
2018 Poster: Diminishing Returns Shape Constraints for Interpretability and Regularization »
Maya Gupta · Dara Bahri · Andrew Cotter · Kevin Canini