Timezone: »
We propose an efficient method to estimate the accuracy of classifiers using only unlabeled data. We consider a setting with multiple classification problems where the target classes may be tied together through logical constraints. For example, a set of classes may be mutually exclusive, meaning that a data instance can belong to at most one of them. The proposed method is based on the intuition that: (i) when classifiers agree, they are more likely to be correct, and (ii) when the classifiers make a prediction that violates the constraints, at least one classifier must be making an error. Experiments on four real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs. The results emphasize the utility of logical constraints in estimating accuracy, thus validating our intuition.
Author Information
Emmanouil Platanios (Carnegie Mellon University)
Hoifung Poon (Microsoft Research)
Hoifung Poon is Senior Director at Microsoft Health Futures. His research interests lie in advancing biomedical AI for precision health. His past work has been recognized with Best Paper Awards from premier NLP and machine learning venues such as the Conference of the North American Chapter of the Association for Computational Linguistics, the Conference of Empirical Methods in Natural Language Processing, and the Conference of Uncertainty in AI.
Tom M Mitchell (Carnegie Mellon University)
Eric Horvitz (Microsoft Research)
More from the Same Authors
-
2021 : Bursting Scientific Filter Bubbles: Boosting Innovation via Novel Author Discovery »
Jason Portenoy · Jevin West · Eric Horvitz · Daniel Weld · Tom Hope -
2021 : A Search Engine for Discovery of Scientific Challenges and Directions »
Dan Lahav · Jon Saad-Falcon · Duen Horng Chau · Diyi Yang · Eric Horvitz · Daniel Weld · Tom Hope -
2022 : Panel »
Tim Althoff · Chun-Nan Hsu · Hoifung Poon · Alison Moore · Rada Mihalcea -
2020 : Closing Remarks: Eric Horvitz (Microsoft) »
Eric Horvitz -
2020 Workshop: Cooperative AI »
Thore Graepel · Dario Amodei · Vincent Conitzer · Allan Dafoe · Gillian Hadfield · Eric Horvitz · Sarit Kraus · Kate Larson · Yoram Bachrach -
2019 : Tom Mitchell »
Tom M Mitchell -
2019 Poster: Efficient Forward Architecture Search »
Hanzhang Hu · John Langford · Rich Caruana · Saurajit Mukherjee · Eric Horvitz · Debadeepta Dey -
2019 Poster: Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting »
Aditya Grover · Jiaming Song · Ashish Kapoor · Kenneth Tran · Alekh Agarwal · Eric Horvitz · Stefano Ermon -
2019 Poster: Graph Agreement Models for Semi-Supervised Learning »
Otilia Stretcu · Krishnamurthy Viswanathan · Dana Movshovitz-Attias · Emmanouil Platanios · Sujith Ravi · Andrew Tomkins -
2019 Poster: Staying up to Date with Online Content Changes Using Reinforcement Learning for Scheduling »
Andrey Kolobov · Yuval Peres · Cheng Lu · Eric Horvitz -
2019 Poster: Game Design for Eliciting Distinguishable Behavior »
Fan Yang · Liu Leqi · Yifan Wu · Zachary Lipton · Pradeep Ravikumar · Tom M Mitchell · William Cohen -
2012 Poster: Patient Risk Stratification for Hospital-Associated C. Diff as a Time-Series Classification Task »
Jenna Wiens · John Guttag · Eric Horvitz -
2012 Spotlight: Patient Risk Stratification for Hospital-Associated C. Diff as a Time-Series Classification Task »
Jenna Wiens · John Guttag · Eric Horvitz -
2009 Poster: Breaking Boundaries Between Induction Time and Diagnosis Time Active Information Acquisition »
Ashish Kapoor · Eric Horvitz