Reva Schwartz: Real World Matters: What Actually Happens When People Use AI? The NIST Assessing Risks and Impacts of AI (ARIA) Program

Abstract: ARIA (Assessing Risks and Impacts of AI) is a NIST evaluation-driven research program to develop measurement methods that can account for AI’s risks and impacts in the real world. The program establishes an experimentation environment to gather evidence about AI’s risks to the public under controlled real-world conditions. In contrast to current approaches that rely on probabilities and predictions of what might happen, ARIA will enable direct observation of AI system behaviors and impacts on users. ARIA pairs people with AI applications in experimental scenarios designed around specific AI risks, and studies the results. Applications are submitted to NIST from around the globe and evaluated based on whether risks materialized in the resulting interactions, and the magnitude and degree of resulting impacts. Participating teams will learn whether their applications can maintain functionality across the varying contexts of the test environment.

Bio: Reva Schwartz is a research scientist at the National Institute of Standards and Technology’s (NIST) Information Technology Laboratory (ITL) where she serves as Principal Investigator on Bias in Artificial Intelligence and leads ARIA, a research program which advances the measurement of AI’s risks to people and society. She previously served as forensic scientist for almost 15 years at the United States Secret Service and adjunct researcher at the Johns Hopkins University Human Language Technology Center of Excellence. \n Reva’s background is in linguistics and human language technology. A socio-technical researcher, she focuses on real world testing methodologies and people-centered approaches for AI measurement. She has advised federal agencies about how experts interact with automation to make sense of information in high-stakes settings.