Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of impressive technical advances. However, evaluation techniques for causal modeling algorithms have remained somewhat primitive, limiting what we can learn from experimental studies of algorithm performance, constraining the types of algorithms and model representations that researchers consider, and creating a gap between theory and practice. We argue for more frequent use of evaluation techniques that examine interventional measures rather than structural or observational measures, and that evaluate those measures on empirical data rather than synthetic data. We survey the current practice in evaluation and show that the techniques we recommend are rarely used in practice. We show that such techniques are feasible and that data sets are available to conduct such evaluations. We also show that these techniques produce substantially different results than using structural measures and synthetic data.
Amanda Gentzel (UMass Amherst)
Dan Garant (C&S Wholesale Grocers)
David Jensen (Univ. of Massachusetts)
David Jensen is a Professor of Computer Science at the University of Massachusetts Amherst. He directs the Knowledge Discovery Laboratory and currently serves as the Director of the Computational Social Science Institute, an interdisciplinary effort at UMass to study social phenomena using computational tools and concepts. From 1991 to 1995, he served as an analyst with the Office of Technology Assessment, an agency of the United States Congress. His current research focuses on methods for constructing accurate causal models from observational and experimental data. He regularly serves on program committees for several conferences, including the Conference on Neural Information Processing Systems, the International Conference on Machine Learning, and the Conference on Uncertainty in Artificial Intelligence. He has served on the Board of Directors of the ACM Special Interest Group on Knowledge Discovery and Data Mining (2005-2013), the Defense Science Study Group (2006-2007), and DARPA's Information Science and Technology Group (2007-2012). In 2011, he received the Outstanding Teacher Award from the UMass College of Natural Sciences.
More from the Same Authors
2022 : Improving the Efficiency of the PC Algorithm by Using Model-Based Conditional Independence Tests »
Erica Cai · Andrew McGregor · David Jensen
2020 : Prof. David Jensen (University of Massachusetts Amherst) - "Empirical Research in Machine Learning: Perspectives and Strategies" »
2018 : The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data »
2018 : Poster Session 1 (note there are numerous missing names here, all papers appear in all poster sessions) »
Akhilesh Gotmare · Kenneth Holstein · Jan Brabec · Michal Uricar · Kaleigh Clary · Cynthia Rudin · Sam Witty · Andrew Ross · Shayne O'Brien · Babak Esmaeili · Jessica Forde · Massimo Caccia · Ali Emami · Scott Jordan · Bronwyn Woods · D. Sculley · Rebekah Overdorf · Nicolas Le Roux · Peter Henderson · Brandon Yang · Tzu-Yu Liu · David Jensen · Niccolo Dalmasso · Weitang Liu · Paul Marc TRICHELAIR · Jun Ki Lee · Akanksha Atrey · Matt Groh · Yotam Hechtlinger · Emma Tosch