Balancing Competing Objectives for Welfare-Aware Machine Learning with Imperfect Data

From financial loans and humanitarian aid, to medical diagnosis and criminal justice, consequential decisions in society increasingly rely on machine learning. In most cases, the machine learning algorithms used in these contexts are trained to optimize a single metric of performance; however, most real-world decisions exist in a multi-objective setting that requires the balance of multiple incentives and outcomes. To this end, we develop a methodology for optimizing multi-objective decisions. Building on the traditional notion of Pareto optimality, we focus on understanding how to balance multiple objectives when those objectives are measured noisily or not directly observed. We believe this regime of imperfect information is far more common in real-world decisions, where one cannot easily measure the social consequences of an algorithmic decision. To show how the multi-objective framework can be used in practice, we present results using data from roughly 40,000 videos promoted by YouTube’s recommendation algorithm. This illustrates the empirical trade-off between maximizing user engagement and promoting high-quality videos. We show that multi-objective optimization could produce substantial increases in average video quality at the expense of almost negligible reductions in user engagement.

Speaker bio: Esther Rolf is a 4th year Ph.D. student in the Computer Science department at the University of California, Berkeley, advised by Benjamin Recht and Michael I. Jordan. She is an NSF Graduate Research Fellow and is a fellow in the Global Policy Lab in the Goldman School of Public Policy at UC Berkeley. Esther’s research targets machine learning algorithms that interact with society. Her current focus lies in two main domains: the field of algorithmic fairness, which aims to design and audit black-box decision algorithms to ensure equity and benefit for all individuals, and in machine learning for environmental monitoring, where abundant sources of temporally recurrent data provide an exciting opportunity to make inferences and predictions about our planet.