Timezone: »

 
Metric Elicitation; Moving from Theory to Practice
Safinah Ali · Sohini Upadhyay · Gaurush Hiranandani · Elena Glassman · Sanmi Koyejo

@
in HCAI@NeurIPS 2022, Human Centered AI »
Event URL: https://openreview.net/forum?id=QS-pUJuzjh »

Metric Elicitation (ME) is a framework for eliciting classification metrics that better align with implicit user preferences based on the task and context. The existing ME strategy so far is based on the assumption that users can most easily provide preference feedback over classifier statistics such as confusion matrices. This work examines ME, by providing a first ever implementation of the ME strategy. Specifically, we create a web-based ME interface and conduct a user study that elicits users' preferred metrics in a binary classification setting. We discuss the study findings and present guidelines for future research in this direction.

Keywords: user study · Confusion Matrix · Visualization · pairwise comparison · binary search · practical · user interface
[ Poster]  [ OpenReview]  [ Topia] 

Author Information

Safinah Ali
Sohini Upadhyay (IBM Research)
Gaurush Hiranandani (Amazon)
Elena Glassman (Harvard University)

I design, build and evaluate systems for comprehending and interacting with population-level structure and trends in large code and data corpora. I am an Assistant Professor of Computer Science at the Harvard Paulson School of Engineering & Applied Sciences and the Stanley A. Marks & William H. Marks Professor at the Radcliffe Institute for Advanced Study, specializing in human-computer interaction. At MIT, I earned a PhD and MEng in Electrical Engineering and Computer Science and a BS in Electrical Science and Engineering. Before joining Harvard, I was a postdoctoral scholar in Electrical Engineering and Computer Science at the University of California, Berkeley, where I received the Berkeley Institute for Data Science Moore/Sloan Data Science Fellowship.

Sanmi Koyejo (Stanford, Google Research)
Sanmi Koyejo

Sanmi Koyejo is an Assistant Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign and a research scientist at Google AI in Accra. Koyejo's research interests are in developing the principles and practice of adaptive and robust machine learning. Additionally, Koyejo focuses on applications to biomedical imaging and neuroscience. Koyejo co-founded the Black in AI organization and currently serves on its board.

More from the Same Authors