Timezone: »
Given a user's input text, text-matching recommender systems output relevant items by comparing the input text to available items' description, such as product-to-product recommendation on e-commerce platforms. As users' interests and item inventory are expected to change, it is important for a text-matching system to generalize to data shifts, a task known as out-of-distribution (OOD) generalization. However, we find that the popular approach of fine-tuning a large, base language model on paired item relevance data (e.g., user clicks) can be counter-productive for OOD generalization. For a product recommendation task, fine-tuning obtains worse accuracy than the base model when recommending items in a new category or for a future time period. To explain this generalization failure, we consider an intervention-based importance metric, which shows that a fine-tuned model captures spurious correlations and fails to learn the causal features that determine the relevance between any two text inputs. Moreover, standard methods for causal regularization do not apply in this setting, because unlike in images, there exist no universally spurious features in a text-matching task (the same token may be spurious or causal depending on the text it is being matched to). For OOD generalization on text inputs, therefore, we highlight a different goal: avoiding high importance scores for certain features. We do so using an intervention-based regularizer that constraints the importance score of any token on the model's relevance score to be similar to the base model. Results on Amazon product and 3 question recommendation datasets show that our proposed regularizer improves generalization for both in-distribution and OOD evaluation, especially in difficult scenarios when the base model is not accurate.
Author Information
Parikshit Bansal (Microsoft Research)
Yashoteja Prabhu (Indian Institute of Technology Delhi)
Emre Kiciman (Microsoft Research)
Amit Sharma (Microsoft Research)
More from the Same Authors
-
2021 : Causality with Susan Athey, Konrad Kording, Amit Sharma »
Susan Athey · Konrad Kording · Amit Sharma · Emre Kiciman -
2022 Poster: Probing Classifiers are Unreliable for Concept Removal and Detection »
Abhinav Kumar · Chenhao Tan · Amit Sharma -
2022 : Causal Modeling of Soil Processes for Improved Generalization »
Somya Sharma · Swati Sharma · Emre Kiciman · Andy Neal · Ranveer Chandra · John Crawford · Sara Malvar · Eduardo Rodrigues -
2022 : Using Interventions to Improve Out-of-Distribution Generalization of Text-Matching Systems »
Parikshit Bansal · Yashoteja Prabhu · Emre Kiciman · Amit Sharma -
2022 : A Causal AI Suite for Decision-Making »
Emre Kiciman · Eleanor Dillon · Darren Edge · Adam Foster · Joel Jennings · Chao Ma · Robert Ness · Nick Pawlowski · Amit Sharma · Cheng Zhang -
2022 : Deep End-to-end Causal Inference »
Tomas Geffner · Javier AntorĂ¡n · Adam Foster · Wenbo Gong · Chao Ma · Emre Kiciman · Amit Sharma · Angus Lamb · Martin Kukla · Nick Pawlowski · Miltiadis Allamanis · Cheng Zhang -
2022 : Counterfactual Generation Under Confounding »
Abbavaram Gowtham Reddy · Saloni Dash · Amit Sharma · Vineeth N Balasubramanian -
2022 : The Counterfactual-Shapley Value: Attributing Change in System Metrics »
Amit Sharma · Hua Li · Jian Jiao -
2023 Poster: Causal Effect Regularization: Automated Detection and Removal of Spurious Attributes »
Abhinav Kumar · Amit Deshpande · Amit Sharma -
2022 Spotlight: Probing Classifiers are Unreliable for Concept Removal and Detection »
Abhinav Kumar · Chenhao Tan · Amit Sharma -
2022 Spotlight: Lightning Talks 1B-1 »
Qitian Wu · Runlin Lei · Rongqin Chen · Luca Pinchetti · Yangze Zhou · Abhinav Kumar · Hans Hao-Hsun Hsu · Wentao Zhao · Chenhao Tan · Zhen Wang · Shenghui Zhang · Yuesong Shen · Tommaso Salvatori · Gitta Kutyniok · Zenan Li · Amit Sharma · Leong Hou U · Yordan Yordanov · Christian Tomani · Bruno Ribeiro · Yaliang Li · David P Wipf · Daniel Cremers · Bolin Ding · Beren Millidge · Ye Li · Yuhang Song · Junchi Yan · Zhewei Wei · Thomas Lukasiewicz -
2022 : A Causal AI Suite for Decision-Making »
Emre Kiciman -
2020 Poster: AvE: Assistance via Empowerment »
Yuqing Du · Stas Tiomkin · Emre Kiciman · Daniel Polani · Pieter Abbeel · Anca Dragan -
2007 Workshop: Machine Learning for Systems Problems (Part 2) »
Archana Ganapathi · Sumit Basu · Fei Sha · Emre Kiciman -
2007 Workshop: Machine Learning for Systems Problems (Part 1) »
Archana Ganapathi · Sumit Basu · Fei Sha · Emre Kiciman -
2007 Poster: Fast Variational Inference for Large-scale Internet Diagnosis »
John C Platt · Emre Kiciman · David A Maltz