Timezone: »
Performance metric elicitation is a type of inverse decision problem where the goal is to learn a loss function for a classification problem using expert comparisons between candidate classifiers. However, for many practical tasks, such an expert can be noisy. We present an approach for learning performance metrics in this setting that can handle general noise models. Our approach takes advantage of the problem's similarity to probabilistic bisection search and uses pairwise comparisons to update a pseudo-belief distribution for the performance metric. Our theoretical results guarantee convergence in practical settings and extend beyond previous results to include multi-expert elicitation. Quantitative comparisons against prior work demonstrate the superiority of our approach.
Author Information
Zachary Robertson (University of Illinois Urbana Champaign)
Hantao Zhang (University of Illinois Urbana Champaign)
Sanmi Koyejo (University of Illinois at Urbana-Champaign & Google Research)

Sanmi Koyejo is an Assistant Professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign and a research scientist at Google AI in Accra. Koyejo's research interests are in developing the principles and practice of adaptive and robust machine learning. Additionally, Koyejo focuses on applications to biomedical imaging and neuroscience. Koyejo co-founded the Black in AI organization and currently serves on its board.
More from the Same Authors
-
2021 : Robust and Personalized Federated Learning with Spurious Features: an Adversarial Approach »
Xiaoyang Wang · Han Zhao · Klara Nahrstedt · Sanmi Koyejo -
2021 : RVFR: Robust Vertical Federated Learning via Feature Subspace Recovery »
Jing Liu · Chulin Xie · Krishnaram Kenthapadi · Sanmi Koyejo · Bo Li -
2021 : Secure Byzantine-Robust Distributed Learning via Clustering »
Raj Kiriti Velicheti · Sanmi Koyejo -
2021 : Exploiting Causal Chains for Domain Generalization »
Olawale Salaudeen · Sanmi Koyejo -
2021 : Distribution Preserving Bayesian Coresets using Set Constraints »
Shovik Guha · Rajiv Khanna · Sanmi Koyejo -
2019 Tutorial: Representation Learning and Fairness »
Moustapha Cisse · Sanmi Koyejo