Timezone: »
Machine learning is rapidly becoming an integral component of sociotechnical systems. Predictions are increasingly used to grant beneficial resources or withhold opportunities, and the consequences of such decisions induce complex social dynamics by changing agent outcomes and prompting individuals to proactively respond to decision rules. This introduces challenges for standard machine learning methodology. Static measurements and training sets poorly capture the complexity of dynamic interactions between algorithms and humans. Strategic adaptation to decision rules can render statistical regularities obsolete. Correlations momentarily observed in data may not be robust enough to support interventions for long-term welfaremits of traditional, static approaches to decision-making, researchers in fields ranging from public policy to computer science to economics have recently begun to view consequential decision-making through a dynamic lens. This workshop will confront the use of machine learning to make consequential decisions in dynamic environments. Work in this area sits at the nexus of several different fields, and the workshop will provide an opportunity to better understand and synthesize social and technical perspectives on these issues and catalyze conversations between researchers and practitioners working across these diverse areas.
Sat 8:00 a.m. - 8:10 a.m.
|
Welcome and introduction
(
Live intro
)
|
🔗 |
Sat 8:10 a.m. - 8:30 a.m.
|
Invited Talk 1: What do we want? And when do we want it? Alternative objectives and their implications for experimental design.
(
Prerecorded talk
)
link »
SlidesLive Video » This talk will be based, in particular, on the following two papers: Adaptive treatment assignment in experiments for policy choice (joint with Anja Sautmann) Forthcoming, Econometrica, 2020 Manuscript: https://maxkasy.github.io/home/files/papers/adaptiveexperimentspolicy.pdf An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan (joint with Stefano Caria, Grant Gordon, Soha Osman, Simon Quinn and Alex Teytelboym) Working paper, 2020 Manuscript: https://maxkasy.github.io/home/files/papers/RefugeesWork.pdf |
Maximilian Kasy 🔗 |
Sat 8:30 a.m. - 8:50 a.m.
|
Invited Talk 2: Country-Scale Bandit Implementation for Targeted COVID-19 Testing
(
Prerecorded talk
)
link »
SlidesLive Video » In collaboration with the Greek government, we use machine learning to manage the threat of COVID-19. With tens of thousands of international visitors every day, Greece cannot test each visitor to ensure that they are not a carrier of COVID-19. We developed a bandit policy that balances allocating scarce tests to (i) continuously monitor the dynamic infection risk of passengers from different locations (exploration), and (ii) preferentially target risky tourist profiles for testing (exploitation). Our solution is currently deployed across all ports of entry to Greece. I will describe a number of technical challenges, including severely imbalanced outcomes, batched/delayed feedback, high-dimensional arms, port-specific testing constraints, and transferring knowledge from (unreliable) public epidemiological data. Joint work with Kimon Drakopoulos, Vishal Gupta and Jon Vlachogiannis. |
Hamsa Bastani 🔗 |
Sat 8:50 a.m. - 9:00 a.m.
|
Q&A for invited talks 1&2
(
Q&A session
)
|
🔗 |
Sat 9:00 a.m. - 10:00 a.m.
|
Poster Session 1
(
Poster session
)
|
🔗 |
Sat 10:00 a.m. - 10:20 a.m.
|
Break 1
|
🔗 |
Sat 10:20 a.m. - 10:30 a.m.
|
Introduction of invited speakers 3, 4
(
Live intro
)
|
🔗 |
Sat 10:30 a.m. - 10:50 a.m.
|
Invited Talk 3: Modeling the Dynamics of Poverty
(
Prerecorded talk
)
link »
SlidesLive Video » The dynamic nature of poverty presents a challenge in designing effective assistance policies. A significant gap in our understanding of poverty is related to the role of income shocks in triggering or perpetuating cycles of poverty. Such shocks can constitute unexpected expenses -- such as a medical bill or a parking ticket -- or an interruption to one’s income flow. Shocks have recently garnered increased public attention, in part due to prevalent evictions and food insecurity during the COVID-19 pandemic. However, shocks do not play a corresponding central role in the design and evaluation of poverty-alleviation programs. To bridge this gap, we present a model of economic welfare that incorporates dynamic experiences with shocks and pose a set of algorithmic questions related to subsidy allocations. We then computationally analyze the impact of shocks on poverty using a longitudinal, survey-based dataset. We reveal insights about the multi-faceted and dynamic nature of shocks and poverty. We discuss how these insights can inform the design of poverty-alleviation programs and highlight directions at this emerging interface of algorithms, economics, and social work. |
Rediet Abebe 🔗 |
Sat 10:50 a.m. - 11:10 a.m.
|
Invited Talk 4: From Moderate Deviations Theory to Distributionally Robust Optimization: Learning from Correlated Data
(
Prerecorded talk
)
link »
SlidesLive Video » We aim to learn a performance function of the invariant state distribution of an unknown linear dynamical system based on a single trajectory of correlated state observations. The function to be learned may represent, for example, an identification objective or a value function. To this end, we develop a distributionally robust estimation scheme that evaluates the worst- and best-case values of the given performance function across all stationary state distributions that are sufficiently likely to have generated the observed state trajectory. By leveraging new insights from moderate deviations theory, we prove that our estimation scheme offers consistent upper and lower confidence bounds whose exponential convergence rate can be actively tuned. In the special case of a quadratic cost, we show that the proposed confidence bounds can be computed efficiently by solving Riccati equations. |
Daniel Kuhn 🔗 |
Sat 11:10 a.m. - 11:20 a.m.
|
Q&A for invited talks 3, 4
(
Q&A session
)
|
🔗 |
Sat 11:20 a.m. - 11:25 a.m.
|
Contributed Talk 1: Fairness Under Partial Compliance
(
Prerecorded talk
)
link »
SlidesLive Video » |
Jessica Dai · Zachary Lipton 🔗 |
Sat 11:25 a.m. - 11:30 a.m.
|
Contributed Talk 2: Better Together? How Externalities of Size Complicate Notions of Solidarity and Actuarial Fairness
(
Prerecorded talk
)
SlidesLive Video » |
Kate Donahue · Solon Barocas 🔗 |
Sat 11:30 a.m. - 11:35 a.m.
|
Contributed Talk 3: Algorithmic Recourse: from Counterfactual Explanations to Interventions
(
Prerecorded talk
)
link »
SlidesLive Video » |
Amir-Hossein Karimi · Bernhard Schölkopf · Isabel Valera 🔗 |
Sat 11:35 a.m. - 11:45 a.m.
|
Q&A for contributed talks 1,2,3
(
Q&A session
)
|
🔗 |
Sat 11:45 a.m. - 12:20 p.m.
|
Break 2
|
🔗 |
Sat 12:20 p.m. - 12:30 p.m.
|
Introduction of invited speakers 5, 6, 7
(
Live intro
)
|
🔗 |
Sat 12:30 p.m. - 12:50 p.m.
|
Invited Talk 5: What are some hurdles before we can attempt machine learning? Examples from the Public and Non-Profit Sector
(
Prerecorded talk
)
link »
SlidesLive Video » Machine learning and predictive analytics are more accessible to the public and nonprofit space now more than ever. Local government and nonprofits strive to leverage these new technologies to improve outcomes, performance, and operations. While a willingness to collaborate and connect on common goals though a shared understanding of data needs serves to build towards a stronger culture around data, the complexities around defining critical terms in dynamic environments pose significant hurdles to be able to scale any machine learning for large cross-departmental initiatives in service of the public. I will share examples from my professional work in NYC government, and probe into challenges with data-driven processes, consensus-based motivations and outcomes. |
Mitsue Iwata 🔗 |
Sat 12:50 p.m. - 1:13 p.m.
|
Invited Talk 6: Unexpected Consequences of Algorithm-in-the-Loop Decision Making
(
Prerecorded talk
)
link »
SlidesLive Video » The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an “algorithm-in-the-loop” process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decision making. In this talk, I’ll discuss results from a set of controlled experiments on algorithm-in-the-loop human decision making in two contexts (pretrial release and financial lending). For example, when presented with algorithmic risk assessments, our study participants exhibited additional bias in their decisions and showed a change in their decision-making process by increasing risk aversion. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact. This talk is based on joint work with Ben Green. |
Yiling Chen 🔗 |
Sat 1:13 p.m. - 1:35 p.m.
|
Invited Talk 7: Prediction Dynamics
(
Prerecorded talk
)
link »
SlidesLive Video » |
Moritz Hardt 🔗 |
Sat 1:35 p.m. - 1:50 p.m.
|
Q&A for invited talks 5, 6, 7
(
Q&A session
)
|
🔗 |
Sat 1:50 p.m. - 2:20 p.m.
|
Break 3
|
🔗 |
Sat 2:20 p.m. - 2:25 p.m.
|
Contributed Talk 4: Strategic Recourse in Linear Classification
(
Prerecorded talk
)
link »
SlidesLive Video » |
Yatong Chen · Yang Liu 🔗 |
Sat 2:25 p.m. - 2:30 p.m.
|
Contributed Talk 5: Performative Prediction in a Stateful World
(
Prerecorded talk
)
link »
SlidesLive Video » |
Shlomi Hod 🔗 |
Sat 2:30 p.m. - 2:35 p.m.
|
Contributed Talk 6: Do Offline Metrics Predict Online Performance in Recommender Systems?
(
Prerecorded talk
)
link »
SlidesLive Video » |
Karl Krauth · Sarah Dean · Wenshuo Guo · Benjamin Recht · Michael Jordan 🔗 |
Sat 2:35 p.m. - 2:45 p.m.
|
Q&A for contributed talks 4, 5, 6
(
Q&A session
)
|
🔗 |
Sat 2:45 p.m. - 3:45 p.m.
|
Poster Session 2
(
Poster session
)
|
🔗 |
Sat 3:45 p.m. - 3:50 p.m.
|
Wrap up
(
Live wrap up
)
|
🔗 |
Author Information
Niki Kilbertus (Helmholtz AI)
Angela Zhou (Cornell University)
Ashia Wilson (UC Berkeley)
John Miller (University of California, Berkeley)
Lily Hu (Harvard University)
Lily Hu is a PhD candidate in Applied Mathematics and Philosophy at Harvard University. She works on topics in machine learning, algorithmic fairness, and (political) philosophy of technology. Her current time is divided between computer science-related research, where she studies theoretical properties and behaviors of machine learning systems as they bear on deployment in social and economic settings, and philosophical work, where she thinks about causal reasoning about categories like race, theories of discrimination, and what about current technological trends makes capitalism even more distressing.
Lydia T. Liu (University of California, Berk)
Nathan Kallus (Cornell University)
Shira Mitchell (Civis Analytics)
More from the Same Authors
-
2021 : It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks »
Michelle Bao · Angela Zhou · Samantha Zottola · Brian Brubach · Sarah Desmarais · Aaron Horowitz · Kristian Lum · Suresh Venkatasubramanian -
2021 : Stateful Offline Contextual Policy Evaluation and Learning »
Angela Zhou -
2022 : Discovering ordinary differential equations that govern time-series »
Sören Becker · Michal Klein · Alexander Neitz · Giambattista Parascandolo · Niki Kilbertus -
2022 : Modeling Single-Cell Dynamics Using Unbalanced Parameterized Monge Maps »
Luca Eyring · Dominik Klein · Giovanni Palla · Sören Becker · Philipp Weiler · Niki Kilbertus · Fabian Theis -
2022 : Partial identification without distributional assumptions »
Kirtan Padh · Jakob Zeitler · David Watson · Matt Kusner · Ricardo Silva · Niki Kilbertus -
2022 Panel: Panel 3C-5: Biologically-Plausible Determinant Maximization… & What's the Harm? ... »
Bariscan Bozkurt · Nathan Kallus -
2022 Workshop: A causal view on dynamical systems »
Sören Becker · Alexis Bellot · Cecilia Casolo · Niki Kilbertus · Sara Magliacane · Yuyang (Bernie) Wang -
2022 Poster: Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systems »
Masatoshi Uehara · Ayush Sekhari · Jason Lee · Nathan Kallus · Wen Sun -
2022 Poster: Sparsity in Continuous-Depth Neural Networks »
Hananeh Aliee · Till Richter · Mikhail Solonin · Ignacio Ibarra · Fabian Theis · Niki Kilbertus -
2022 Poster: The Implicit Delta Method »
Nathan Kallus · James McInerney -
2022 Poster: What's the Harm? Sharp Bounds on the Fraction Negatively Affected by Treatment »
Nathan Kallus -
2022 Poster: Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution »
Leon Hetzel · Simon Boehm · Niki Kilbertus · Stephan Günnemann · mohammad lotfollahi · Fabian Theis -
2021 Workshop: Causal Inference Challenges in Sequential Decision Making: Bridging Theory and Practice »
Aurelien Bibaut · Maria Dimakopoulou · Nathan Kallus · Xinkun Nie · Masatoshi Uehara · Kelly Zhang -
2021 : Stateful Offline Contextual Policy Evaluation and Learning »
Angela Zhou -
2021 Workshop: Machine Learning Meets Econometrics (MLECON) »
David Bruns-Smith · Arthur Gretton · Limor Gultchin · Niki Kilbertus · Krikamol Muandet · Evan Munro · Angela Zhou -
2021 Oral: Retiring Adult: New Datasets for Fair Machine Learning »
Frances Ding · Moritz Hardt · John Miller · Ludwig Schmidt -
2021 Poster: Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning »
Aurelien Bibaut · Nathan Kallus · Maria Dimakopoulou · Antoine Chambaz · Mark van der Laan -
2021 Poster: Retiring Adult: New Datasets for Fair Machine Learning »
Frances Ding · Moritz Hardt · John Miller · Ludwig Schmidt -
2021 Poster: Control Variates for Slate Off-Policy Evaluation »
Nikos Vlassis · Ashok Chandrashekar · Fernando Amat · Nathan Kallus -
2021 Poster: On Component Interactions in Two-Stage Recommender Systems »
Jiri Hron · Karl Krauth · Michael Jordan · Niki Kilbertus -
2021 Poster: Post-Contextual-Bandit Inference »
Aurelien Bibaut · Maria Dimakopoulou · Nathan Kallus · Antoine Chambaz · Mark van der Laan -
2021 : It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks »
Michelle Bao · Angela Zhou · Samantha Zottola · Brian Brubach · Sarah Desmarais · Aaron Horowitz · Kristian Lum · Suresh Venkatasubramanian -
2020 : Invited Talk: Does Causal Thinking About Discrimination Assume a Can Opener? »
Lily Hu -
2020 : Spotlight Talk 4: Fairness, Welfare, and Equity in Personalized Pricing »
Nathan Kallus · Angela Zhou -
2020 : Invited speaker: Stochastic Geodesic Optimization, Ashia Wilson »
Ashia Wilson -
2020 Poster: Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement Learning »
Nathan Kallus · Angela Zhou -
2020 Poster: A Class of Algorithms for General Instrumental Variable Models »
Niki Kilbertus · Matt Kusner · Ricardo Silva -
2020 Poster: Doubly Robust Off-Policy Value and Gradient Estimation for Deterministic Policies »
Nathan Kallus · Masatoshi Uehara -
2019 : Coffee Break and Poster Session »
Rameswar Panda · Prasanna Sattigeri · Kush Varshney · Karthikeyan Natesan Ramamurthy · Harvineet Singh · Vishwali Mhasawade · Shalmali Joshi · Laleh Seyyed-Kalantari · Matthew McDermott · Gal Yona · James Atwood · Hansa Srinivasan · Yonatan Halpern · D. Sculley · Behrouz Babaki · Margarida Carvalho · Josie Williams · Narges Razavian · Haoran Zhang · Amy Lu · Irene Y Chen · Xiaojie Mao · Angela Zhou · Nathan Kallus -
2019 : Opening Remarks »
Thorsten Joachims · Nathan Kallus · Michele Santacatterina · Adith Swaminathan · David Sontag · Angela Zhou -
2019 Workshop: “Do the right thing”: machine learning and causal inference for improved decision making »
Michele Santacatterina · Thorsten Joachims · Nathan Kallus · Adith Swaminathan · David Sontag · Angela Zhou -
2019 : Solutions »
Fitzroy Christian · Lily Hu · Risi Kondor · Brandeis Marshall · Fabian Rogers · Friederike Schuur · Emanuel Moss -
2019 : Nathan Kallus: Efficiently Breaking the Curse of Horizon with Double Reinforcement Learning »
Nathan Kallus -
2019 Poster: The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the XAUC Metric »
Nathan Kallus · Angela Zhou -
2019 Poster: Assessing Disparate Impact of Personalized Interventions: Identifiability and Bounds »
Nathan Kallus · Angela Zhou -
2019 Poster: Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions »
Ashia Wilson · Lester Mackey · Andre Wibisono -
2019 Poster: Intrinsically Efficient, Stable, and Bounded Off-Policy Evaluation for Reinforcement Learning »
Nathan Kallus · Masatoshi Uehara -
2019 Poster: Model Similarity Mitigates Test Set Overuse »
Horia Mania · John Miller · Ludwig Schmidt · Moritz Hardt · Benjamin Recht -
2019 Poster: A Meta-Analysis of Overfitting in Machine Learning »
Becca Roelofs · Vaishaal Shankar · Benjamin Recht · Sara Fridovich-Keil · Moritz Hardt · John Miller · Ludwig Schmidt -
2019 Poster: Policy Evaluation with Latent Confounders via Optimal Balance »
Andrew Bennett · Nathan Kallus -
2019 Poster: Deep Generalized Method of Moments for Instrumental Variable Analysis »
Andrew Bennett · Nathan Kallus · Tobias Schnabel -
2018 Workshop: Privacy Preserving Machine Learning »
Adria Gascon · Aurélien Bellet · Niki Kilbertus · Olga Ohrimenko · Mariana Raykova · Adrian Weller -
2018 Workshop: Challenges and Opportunities for AI in Financial Services: the Impact of Fairness, Explainability, Accuracy, and Privacy »
Manuela Veloso · Nathan Kallus · Sameena Shah · Senthil Kumar · Isabelle Moulinier · Jiahao Chen · John Paisley -
2018 Poster: Causal Inference with Noisy and Missing Covariates via Matrix Factorization »
Nathan Kallus · Xiaojie Mao · Madeleine Udell -
2018 Poster: Removing Hidden Confounding by Experimental Grounding »
Nathan Kallus · Aahlad Puli · Uri Shalit -
2018 Spotlight: Removing Hidden Confounding by Experimental Grounding »
Nathan Kallus · Aahlad Puli · Uri Shalit -
2018 Poster: On the Local Minima of the Empirical Risk »
Chi Jin · Lydia T. Liu · Rong Ge · Michael Jordan -
2018 Poster: Confounding-Robust Policy Improvement »
Nathan Kallus · Angela Zhou -
2018 Spotlight: On the Local Minima of the Empirical Risk »
Chi Jin · Lydia T. Liu · Rong Ge · Michael Jordan -
2018 Poster: Balanced Policy Evaluation and Learning »
Nathan Kallus -
2017 Workshop: From 'What If?' To 'What Next?' : Causal Inference and Machine Learning for Intelligent Decision Making »
Ricardo Silva · Panagiotis Toulis · John Shawe-Taylor · Alexander Volfovsky · Thorsten Joachims · Lihong Li · Nathan Kallus · Adith Swaminathan -
2017 Poster: Avoiding Discrimination through Causal Reasoning »
Niki Kilbertus · Mateo Rojas Carulla · Giambattista Parascandolo · Moritz Hardt · Dominik Janzing · Bernhard Schölkopf