Timezone: »

 
Poster
CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior
Eldar D Abraham · Karel D'Oosterlinck · Amir Feder · Yair Gat · Atticus Geiger · Christopher Potts · Roi Reichart · Zhengxuan Wu

Thu Dec 01 02:00 PM -- 04:00 PM (PST) @ Hall J #107
in Poster Session 6 »

The increasing size and complexity of modern ML systems has improved their predictive capabilities but made their behavior harder to explain. Many techniques for model explanation have been developed in response, but we lack clear criteria for assessing these techniques. In this paper, we cast model explanation as the causal inference problem of estimating causal effects of real-world concepts on the output behavior of ML models given actual input data. We introduce CEBaB, a new benchmark dataset for assessing concept-based explanation methods in Natural Language Processing (NLP). CEBaB consists of short restaurant reviews with human-generated counterfactual reviews in which an aspect (food, noise, ambiance, service) of the dining experience was modified. Original and counterfactual reviews are annotated with multiply-validated sentiment ratings at the aspect-level and review-level. The rich structure of CEBaB allows us to go beyond input features to study the effects of abstract, real-world concepts on model behavior. We use CEBaB to compare the quality of a range of concept-based explanation methods covering different assumptions and conceptions of the problem, and we seek to establish natural metrics for comparative assessments of these methods.

Keywords: Causality · explainability · benchmark · Causal Explanation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Eldar D Abraham (Technion - Israel Institute of Technology)
Eldar D Abraham

I’m an M.Sc student at the Technion - Israel Institute of Technology, working on Natural Language Processing and Causal Inference. Sometimes I also play with general deep learning and even optimization algorithms. My advisor is Prof. Roi Reichart.

Karel D'Oosterlinck (Ghent University / Stanford University)
Karel D'Oosterlinck

PhD Student at Ghent University. Visiting Student Researcher at Stanford University. Computer Science, Explainable AI.

Amir Feder (Columbia University)
Amir Feder

Amir Feder is a Postdoctoral Research Scientist in the Data Science Institute, working with Professor David Blei on causal inference and natural language processing. His research seeks to develop methods that integrate causality into natural language processing, and use them to build linguistically-informed algorithms for predicting and understanding human behavior. Through the paradigm of causal machine learning, Amir aims to build bridges between machine learning and the social sciences. Before joining Columbia, Amir received his PhD from the Technion, where he was advised by Roi Reichart and worked closely with Uri Shalit. In a previous (academic) life, Amir was an economics, statistics and history student at Tel Aviv University, the Hebrew University of Jerusalem and Northwestern University. Amir was the organizer of the First Workshop on Causal Inference and NLP (CI+NLP) at EMNLP 2021.

Yair Gat (Technion)
Atticus Geiger (Stanford University)
Christopher Potts (Stanford University)
Roi Reichart (Technion, Israel Institute of Technology)
Zhengxuan Wu (Stanford University)

More from the Same Authors