Timezone: »

 
Poster
Transformer Memory as a Differentiable Search Index
Yi Tay · Vinh Tran · Mostafa Dehghani · Jianmo Ni · Dara Bahri · Harsh Mehta · Zhen Qin · Kai Hui · Zhe Zhao · Jai Gupta · Tal Schuster · William Cohen · Donald Metzler

@

In this paper, we demonstrate that information retrieval can be accomplished with a single Transformer, in which all information about the corpus is encoded in the parameters of the model. To this end, we introduce the Differentiable Search Index (DSI), a new paradigm that learns a text-to-text model that maps string queries directly to relevant docids; in other words, a DSI model answers queries directly using only its parameters, dramatically simplifying the whole retrieval process. We study variations in how documents and their identifiers are represented, variations in training procedures, and the interplay between models and corpus sizes. Experiments demonstrate that given appropriate design choices, DSI significantly outperforms strong baselines such as dual encoder models. Moreover, DSI demonstrates strong generalization capabilities, outperforming a BM25 baseline in a zero-shot setup.

Author Information

Yi Tay (Google Brain)
Vinh Tran (Google)
Mostafa Dehghani (Google Brain)
Jianmo Ni (Google)
Dara Bahri (Google AI)
Harsh Mehta (Google Research)
Zhen Qin (Google)
Kai Hui (Google)
Zhe Zhao (Google)
Jai Gupta (Indian Institute of Technology Kharagpur)
Tal Schuster (Google Research)
William Cohen (Google AI)
Donald Metzler (Google)

More from the Same Authors