Timezone: »
Inappropriate design and deployment of machine learning (ML) systems leads to negative downstream social and ethical impact -- described here as social and ethical risks -- for users, society and the environment. Despite the growing need to regulate ML systems, current processes for assessing and mitigating risks are disjointed and inconsistent. We interviewed 30 industry practitioners on their current social and ethical risk management practices, and collected their first reactions on adapting safety engineering frameworks into their practice -- namely, System Theoretic Process Analysis (STPA) and Failure Mode and Effects Analysis (FMEA). Our findings suggest STPA/FMEA can provide appropriate structure toward social and ethical risk assessment and mitigation processes. However, we also find nontrivial challenges in integrating such frameworks in the fast-paced culture of the ML industry. We call on the ML research community to strengthen existing frameworks and assess their efficacy, ensuring that ML systems are safer for all people.
Author Information
Shalaleh Rismani (McGill University)
Renee Shelby (Google)
Andrew Smart (Google)
Edgar Jatho (Naval Postgraduate School)
Commander Jatho is a member of the Permanent Military Professor Community. He is currently pursuing a PhD in Computer Science at the Naval Postgraduate School in Monterey where his research specializes in trustworthy artificial intelligence and developing general approaches to eliminate or mitigate the problem of adversarial examples in deep neural network classification systems. Prior to May 2020, CDR Jatho served in the Cryptologic Warfare community. Previous tours included Navy Cyber Defense Operations Command as the N9 Defensive Cyber Operations Afloat Department Head, CARRIER STRIKE GROUP TEN as Cryptologic Resource Coordinator, National Security Agency as Special Access Program Central Office and Special Technical Operations Deputy Chief. His awards include two Defense Meritorious Service Medals and two Navy Meritorious Service Medals.
Joshua Kroll (Naval Postgraduate School)
AJung Moon (McGill University)
AJung Moon is an experimental roboticist. She investigates how robots and AI systems influence the way people move, behave, and make decisions in order to inform how we can design and deploy such autonomous intelligent systems more responsibly. At McGill University, she is the Director of the McGill Responsible Autonomy & Intelligent System Ethics (RAISE) lab. The RAISE lab is an interdisciplinary group that investigates the social and ethical implications of robots and AI systems and explores what it means for engineers to be designing and deploying such systems responsibly for a better, technological future.
Negar Rostamzadeh (Google)
More from the Same Authors
-
2021 : Artsheets for Art Datasets »
Ramya Srinivasan · Remi Denton · Jordan Famularo · Negar Rostamzadeh · Fernando Diaz · Beth Coleman -
2021 : Thinking Beyond Distributions in Testing Machine Learned Models »
Negar Rostamzadeh · Ben Hutchinson · Vinodkumar Prabhakaran -
2022 : System Safety Engineering for Social and Ethical ML Risks: A Case Study »
Edgar Jatho · Logan Mailloux · Shalaleh Rismani · Eugene Williams · Joshua Kroll -
2023 : Assessing LLMs for Moral Value Pluralism - (Spoiler Alert: They’re not There Yet) »
Sonja Schmer-Galunder · Noam Benkler · Drisana Mosaphir · Andrew Smart · Scott Friedman -
2023 : Improving Generalization in Reinforcement Learning Training Regimes for Social Robot Navigation »
Adam Sigal · Hsiu-Chin Lin · AJung Moon -
2023 : Breaking Barriers to Creative Expression: Co-Designing and Implementing an Accessible Text-to-Image Interface »
Atieh Taheri · Mohammad Izadi · Gururaj Shriram · Negar Rostamzadeh · Shaun Kane -
2022 : Q & A »
Cheng-Zhi Anna Huang · Negar Rostamzadeh · Mark Riedl -
2022 Tutorial: Creative Culture and Machine Learning »
Negar Rostamzadeh · Cheng-Zhi Anna Huang · Mark Riedl -
2022 : Tutorial part 1 »
Negar Rostamzadeh · Mark Riedl · Cheng-Zhi Anna Huang -
2022 : Ethics Roundtable »
Negar Rostamzadeh · Sina Fazelpour · Nyalleng Moorosi -
2019 : Poster Session »
Jonathan Scarlett · Piotr Indyk · Ali Vakilian · Adrian Weller · Partha P Mitra · Benjamin Aubin · Bruno Loureiro · Florent Krzakala · Lenka Zdeborová · Kristina Monakhova · Joshua Yurtsever · Laura Waller · Hendrik Sommerhoff · Michael Moeller · Rushil Anirudh · Shuang Qiu · Xiaohan Wei · Zhuoran Yang · Jayaraman Thiagarajan · Salman Asif · Michael Gillhofer · Johannes Brandstetter · Sepp Hochreiter · Felix Petersen · Dhruv Patel · Assad Oberai · Akshay Kamath · Sushrut Karmalkar · Eric Price · Ali Ahmed · Zahra Kadkhodaie · Sreyas Mohan · Eero Simoncelli · Carlos Fernandez-Granda · Oscar Leong · Wesam Sakla · Rebecca Willett · Stephan Hoyer · Jascha Sohl-Dickstein · Sam Greydanus · Gauri Jagatap · Chinmay Hegde · Michael Kellman · Jonathan Tamir · Nouamane Laanait · Ousmane Dia · Mirco Ravanelli · Jonathan Binas · Negar Rostamzadeh · Shirin Jalali · Tiantian Fang · Alex Schwing · Sébastien Lachapelle · Philippe Brouillard · Tristan Deleu · Simon Lacoste-Julien · Stella Yu · Arya Mazumdar · Ankit Singh Rawat · Yue Zhao · Jianshu Chen · Xiaoyang Li · Hubert Ramsauer · Gabrio Rizzuti · Nikolaos Mitsakos · Dingzhou Cao · Thomas Strohmer · Yang Li · Pei Peng · Gregory Ongie -
2019 : Responsibilities »
Been Kim · Liz O'Sullivan · Friederike Schuur · Andrew Smart · Jacob Metcalf -
2019 Poster: Adaptive Cross-Modal Few-shot Learning »
Chen Xing · Negar Rostamzadeh · Boris Oreshkin · Pedro O. Pinheiro -
2019 Poster: Neural Multisensory Scene Inference »
Jae Hyun Lim · Pedro O. Pinheiro · Negar Rostamzadeh · Chris Pal · Sungjin Ahn -
2018 : Coffee Break and Poster Session I »
Pim de Haan · Bin Wang · Dequan Wang · Aadil Hayat · Ibrahim Sobh · Muhammad Asif Rana · Thibault Buhet · Nicholas Rhinehart · Arjun Sharma · Alex Bewley · Michael Kelly · Lionel Blondé · Ozgur S. Oguz · Vaibhav Viswanathan · Jeroen Vanbaar · Konrad Żołna · Negar Rostamzadeh · Rowan McAllister · Sanjay Thakur · Alexandros Kalousis · Chelsea Sidrane · Sujoy Paul · Daphne Chen · Michal Garmulewicz · Henryk Michalewski · Coline Devin · Hongyu Ren · Jiaming Song · Wen Sun · Hanzhang Hu · Wulong Liu · Emilie Wirbel