Timezone: »

Consequential Decisions in Dynamic Environments
Niki Kilbertus · Angela Zhou · Ashia Wilson · John Miller · Lily Hu · Lydia T. Liu · Nathan Kallus · Shira Mitchell

Sat Dec 12 08:00 AM -- 03:50 PM (PST) @ None
Event URL: https://dynamicdecisions.github.io/ »

Machine learning is rapidly becoming an integral component of sociotechnical systems. Predictions are increasingly used to grant beneficial resources or withhold opportunities, and the consequences of such decisions induce complex social dynamics by changing agent outcomes and prompting individuals to proactively respond to decision rules. This introduces challenges for standard machine learning methodology. Static measurements and training sets poorly capture the complexity of dynamic interactions between algorithms and humans. Strategic adaptation to decision rules can render statistical regularities obsolete. Correlations momentarily observed in data may not be robust enough to support interventions for long-term welfaremits of traditional, static approaches to decision-making, researchers in fields ranging from public policy to computer science to economics have recently begun to view consequential decision-making through a dynamic lens. This workshop will confront the use of machine learning to make consequential decisions in dynamic environments. Work in this area sits at the nexus of several different fields, and the workshop will provide an opportunity to better understand and synthesize social and technical perspectives on these issues and catalyze conversations between researchers and practitioners working across these diverse areas.

Sat 8:00 a.m. - 8:10 a.m.
Welcome and introduction (Live intro)
Sat 8:10 a.m. - 8:30 a.m.

This talk will be based, in particular, on the following two papers:

Adaptive treatment assignment in experiments for policy choice (joint with Anja Sautmann) Forthcoming, Econometrica, 2020 Manuscript: https://maxkasy.github.io/home/files/papers/adaptiveexperimentspolicy.pdf

An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan (joint with Stefano Caria, Grant Gordon, Soha Osman, Simon Quinn and Alex Teytelboym) Working paper, 2020 Manuscript: https://maxkasy.github.io/home/files/papers/RefugeesWork.pdf

Maximilian Kasy
Sat 8:30 a.m. - 8:50 a.m.

In collaboration with the Greek government, we use machine learning to manage the threat of COVID-19. With tens of thousands of international visitors every day, Greece cannot test each visitor to ensure that they are not a carrier of COVID-19. We developed a bandit policy that balances allocating scarce tests to (i) continuously monitor the dynamic infection risk of passengers from different locations (exploration), and (ii) preferentially target risky tourist profiles for testing (exploitation). Our solution is currently deployed across all ports of entry to Greece. I will describe a number of technical challenges, including severely imbalanced outcomes, batched/delayed feedback, high-dimensional arms, port-specific testing constraints, and transferring knowledge from (unreliable) public epidemiological data. Joint work with Kimon Drakopoulos, Vishal Gupta and Jon Vlachogiannis.

Hamsa Bastani
Sat 8:50 a.m. - 9:00 a.m.
Q&A for invited talks 1&2 (Q&A session)
Sat 9:00 a.m. - 10:00 a.m.
Poster Session 1 (Poster session)
Sat 10:00 a.m. - 10:20 a.m.
Break 1 (Break)
Sat 10:20 a.m. - 10:30 a.m.
Introduction of invited speakers 3, 4 (Live intro)
Sat 10:30 a.m. - 10:50 a.m.

The dynamic nature of poverty presents a challenge in designing effective assistance policies. A significant gap in our understanding of poverty is related to the role of income shocks in triggering or perpetuating cycles of poverty. Such shocks can constitute unexpected expenses -- such as a medical bill or a parking ticket -- or an interruption to one’s income flow. Shocks have recently garnered increased public attention, in part due to prevalent evictions and food insecurity during the COVID-19 pandemic. However, shocks do not play a corresponding central role in the design and evaluation of poverty-alleviation programs.

To bridge this gap, we present a model of economic welfare that incorporates dynamic experiences with shocks and pose a set of algorithmic questions related to subsidy allocations. We then computationally analyze the impact of shocks on poverty using a longitudinal, survey-based dataset. We reveal insights about the multi-faceted and dynamic nature of shocks and poverty. We discuss how these insights can inform the design of poverty-alleviation programs and highlight directions at this emerging interface of algorithms, economics, and social work.

Rediet Abebe
Sat 10:50 a.m. - 11:10 a.m.

We aim to learn a performance function of the invariant state distribution of an unknown linear dynamical system based on a single trajectory of correlated state observations. The function to be learned may represent, for example, an identification objective or a value function. To this end, we develop a distributionally robust estimation scheme that evaluates the worst- and best-case values of the given performance function across all stationary state distributions that are sufficiently likely to have generated the observed state trajectory. By leveraging new insights from moderate deviations theory, we prove that our estimation scheme offers consistent upper and lower confidence bounds whose exponential convergence rate can be actively tuned. In the special case of a quadratic cost, we show that the proposed confidence bounds can be computed efficiently by solving Riccati equations.

Daniel Kuhn
Sat 11:10 a.m. - 11:20 a.m.
Q&A for invited talks 3, 4 (Q&A session)
Sat 11:20 a.m. - 11:25 a.m.
Contributed Talk 1: Fairness Under Partial Compliance (Prerecorded talk) Jessica Dai, Zachary Lipton
Sat 11:25 a.m. - 11:30 a.m.
Contributed Talk 2: Better Together? How Externalities of Size Complicate Notions of Solidarity and Actuarial Fairness (Prerecorded talk)
Kate Donahue, Solon Barocas
Sat 11:30 a.m. - 11:35 a.m.
Contributed Talk 3: Algorithmic Recourse: from Counterfactual Explanations to Interventions (Prerecorded talk) Amir Karimi, Bernhard Schölkopf, Isabel Valera
Sat 11:35 a.m. - 11:45 a.m.
Q&A for contributed talks 1,2,3 (Q&A session)
Sat 11:45 a.m. - 12:20 p.m.
Break 2 (Break)
Sat 12:20 p.m. - 12:30 p.m.
Introduction of invited speakers 5, 6, 7 (Live intro)
Sat 12:30 p.m. - 12:50 p.m.

Machine learning and predictive analytics are more accessible to the public and nonprofit space now more than ever. Local government and nonprofits strive to leverage these new technologies to improve outcomes, performance, and operations. While a willingness to collaborate and connect on common goals though a shared understanding of data needs serves to build towards a stronger culture around data, the complexities around defining critical terms in dynamic environments pose significant hurdles to be able to scale any machine learning for large cross-departmental initiatives in service of the public. I will share examples from my professional work in NYC government, and probe into challenges with data-driven processes, consensus-based motivations and outcomes.

Mitsue Iwata
Sat 12:50 p.m. - 1:13 p.m.

The rise of machine learning has fundamentally altered decision making: rather than being made solely by people, many important decisions are now made through an “algorithm-in-the-loop” process where machine learning models inform people. Yet insufficient research has considered how the interactions between people and models actually influence human decision making. In this talk, I’ll discuss results from a set of controlled experiments on algorithm-in-the-loop human decision making in two contexts (pretrial release and financial lending). For example, when presented with algorithmic risk assessments, our study participants exhibited additional bias in their decisions and showed a change in their decision-making process by increasing risk aversion. These results highlight the urgent need to expand our analyses of algorithmic decision making aids beyond evaluating the models themselves to investigating the full sociotechnical contexts in which people and algorithms interact.

This talk is based on joint work with Ben Green.

Yiling Chen
Sat 1:13 p.m. - 1:35 p.m.
Invited Talk 7: Prediction Dynamics (Prerecorded talk) Moritz Hardt
Sat 1:35 p.m. - 1:50 p.m.
Q&A for invited talks 5, 6, 7 (Q&A session)
Sat 1:50 p.m. - 2:20 p.m.
Break 3 (Break)
Sat 2:20 p.m. - 2:25 p.m.
Contributed Talk 4: Strategic Recourse in Linear Classification (Prerecorded talk) Yatong Chen, Yang Liu
Sat 2:25 p.m. - 2:30 p.m.
Contributed Talk 5: Performative Prediction in a Stateful World (Prerecorded talk) Shlomi Hod
Sat 2:30 p.m. - 2:35 p.m.
Contributed Talk 6: Do Offline Metrics Predict Online Performance in Recommender Systems? (Prerecorded talk) Karl Krauth, Sarah Dean, Wenshuo Guo, Benjamin Recht, Michael Jordan
Sat 2:35 p.m. - 2:45 p.m.
Q&A for contributed talks 4, 5, 6 (Q&A session)
Sat 2:45 p.m. - 3:45 p.m.
Poster Session 2 (Poster session)
Sat 3:45 p.m. - 3:50 p.m.
Wrap up (Live wrap up)

Author Information

Niki Kilbertus (Helmholtz AI)
Angela Zhou (Cornell University)
Ashia Wilson (UC Berkeley)
John Miller (University of California, Berkeley)
Lily Hu (Harvard University)

Lily Hu is a PhD candidate in Applied Mathematics and Philosophy at Harvard University. She works on topics in machine learning, algorithmic fairness, and (political) philosophy of technology. Her current time is divided between computer science-related research, where she studies theoretical properties and behaviors of machine learning systems as they bear on deployment in social and economic settings, and philosophical work, where she thinks about causal reasoning about categories like race, theories of discrimination, and what about current technological trends makes capitalism even more distressing.

Lydia T. Liu (University of California, Berk)
Nathan Kallus (Cornell University)
Shira Mitchell (Civis Analytics)

More from the Same Authors