Timezone: »

 
Poster
Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach
Emmanouil Platanios · Hoifung Poon · Tom M Mitchell · Eric Horvitz

Mon Dec 04 06:30 PM -- 10:30 PM (PST) @ Pacific Ballroom #32
in Posters Mon »

We propose an efficient method to estimate the accuracy of classifiers using only unlabeled data. We consider a setting with multiple classification problems where the target classes may be tied together through logical constraints. For example, a set of classes may be mutually exclusive, meaning that a data instance can belong to at most one of them. The proposed method is based on the intuition that: (i) when classifiers agree, they are more likely to be correct, and (ii) when the classifiers make a prediction that violates the constraints, at least one classifier must be making an error. Experiments on four real-world data sets produce accuracy estimates within a few percent of the true accuracy, using solely unlabeled data. Our models also outperform existing state-of-the-art solutions in both estimating accuracies, and combining multiple classifier outputs. The results emphasize the utility of logical constraints in estimating accuracy, thus validating our intuition.

Keywords: Semi-Supervised Learning · Unsupervised Learning · Relational Learning · Boosting and Ensemble Methods
[ Code]  [ Paper] 

Author Information

Emmanouil Platanios (Carnegie Mellon University)
Hoifung Poon (Microsoft Research)

Hoifung Poon is Senior Director at Microsoft Health Futures. His research interests lie in advancing biomedical AI for precision health. His past work has been recognized with Best Paper Awards from premier NLP and machine learning venues such as the Conference of the North American Chapter of the Association for Computational Linguistics, the Conference of Empirical Methods in Natural Language Processing, and the Conference of Uncertainty in AI.

Tom M Mitchell (Carnegie Mellon University)
Eric Horvitz (Microsoft Research)

More from the Same Authors