Timezone: »
An increasingly common use case for machine learning models is augmenting the abilities of human decision makers. For classification tasks where neither the human nor model are perfectly accurate, a key step in obtaining high performance is combining their individual predictions in a manner that leverages their relative strengths. In this work, we develop a set of algorithms that combine the probabilistic output of a model with the class-level output of a human. We show theoretically that the accuracy of our combination model is driven not only by the individual human and model accuracies, but also by the model's confidence. Empirical results on image classification with CIFAR-10 and a subset of ImageNet demonstrate that such human-model combinations consistently have higher accuracies than the model or human alone, and that the parameters of the combination method can be estimated effectively with as few as ten labeled datapoints.
Author Information
Gavin Kerrigan (UC Irvine)
Padhraic Smyth (University of California, Irvine)
Mark Steyvers (UC Irvine)
More from the Same Authors
-
2022 : Probabilistic Querying of Continuous-Time Sequential Events »
Alex Boyd · Yuxin Chang · Stephan Mandt · Padhraic Smyth -
2023 Poster: Zero-Shot Batch-Level Anomaly Detection »
Aodong Li · Chen Qiu · Marius Kloft · Padhraic Smyth · Maja Rudolph · Stephan Mandt -
2023 Poster: Large Language Models Are Implicitly Topic Models: Explaining and Finding Good Demonstrations for In-Context Learning »
Xinyi Wang · Wanrong Zhu · Michael Saxon · Mark Steyvers · William Yang Wang -
2022 Poster: Predictive Querying for Autoregressive Neural Sequence Models »
Alex Boyd · Samuel Showalter · Stephan Mandt · Padhraic Smyth -
2021 Poster: Detecting and Adapting to Irregular Distribution Shifts in Bayesian Online Learning »
Aodong Li · Alex Boyd · Padhraic Smyth · Stephan Mandt -
2020 Poster: Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference »
Disi Ji · Padhraic Smyth · Mark Steyvers -
2020 Poster: User-Dependent Neural Sequence Models for Continuous-Time Event Data »
Alex Boyd · Robert Bamler · Stephan Mandt · Padhraic Smyth -
2017 : Coffee break and Poster Session II »
Mohamed Kane · Albert Haque · Vagelis Papalexakis · John Guibas · Peter Li · Carlos Arias · Eric Nalisnick · Padhraic Smyth · Frank Rudzicz · Xia Zhu · Theodore Willke · Noemie Elhadad · Hans Raffauf · Harini Suresh · Paroma Varma · Yisong Yue · Ognjen (Oggi) Rudovic · Luca Foschini · Syed Rameel Ahmad · Hasham ul Haq · Valerio Maggio · Giuseppe Jurman · Sonali Parbhoo · Pouya Bashivan · Jyoti Islam · Mirco Musolesi · Chris Wu · Alexander Ratner · Jared Dunnmon · Cristóbal Esteban · Aram Galstyan · Greg Ver Steeg · Hrant Khachatrian · Marc Górriz · Mihaela van der Schaar · Anton Nemchenko · Manasi Patwardhan · Tanay Tandon -
2016 Workshop: Towards an Artificial Intelligence for Data Science »
Charles Sutton · James Geddes · Zoubin Ghahramani · Padhraic Smyth · Chris Williams -
2013 Poster: Scoring Workers in Crowdsourcing: How Many Control Questions are Enough? »
Qiang Liu · Alexander Ihler · Mark Steyvers -
2013 Spotlight: Scoring Workers in Crowdsourcing: How Many Control Questions are Enough? »
Qiang Liu · Alexander Ihler · Mark Steyvers -
2012 Workshop: Algorithmic and Statistical Approaches for Large Social Network Data Sets »
Michael Goodrich · Pavel N Krivitsky · David M Mount · Christopher DuBois · Padhraic Smyth -
2011 Oral: Continuous-Time Regression Models for Longitudinal Networks »
Duy Q Vu · Arthur Asuncion · David Hunter · Padhraic Smyth -
2011 Poster: Continuous-Time Regression Models for Longitudinal Networks »
Duy Q Vu · Arthur Asuncion · David Hunter · Padhraic Smyth -
2010 Spotlight: Learning concept graphs from text with stick-breaking priors »
America Chambers · Padhraic Smyth · Mark Steyvers -
2010 Poster: Learning concept graphs from text with stick-breaking priors »
America Chambers · Padhraic Smyth · Mark Steyvers -
2009 Poster: Particle-based Variational Inference for Continuous Systems »
Alexander Ihler · Andrew Frank · Padhraic Smyth -
2009 Poster: The Wisdom of Crowds in the Recollection of Order Information »
Mark Steyvers · Michael D Lee · Brent Miller · Pernille Hemmer -
2008 Poster: Asynchronous Distributed Learning of Topic Models »
Arthur Asuncion · Padhraic Smyth · Max Welling -
2007 Spotlight: Distributed Inference for Latent Dirichlet Allocation »
David Newman · Arthur Asuncion · Padhraic Smyth · Max Welling -
2007 Poster: Distributed Inference for Latent Dirichlet Allocation »
David Newman · Arthur Asuncion · Padhraic Smyth · Max Welling -
2006 Poster: Modeling General and Specific Aspects of Documents with a Probabilistic Topic Model »
Chaitanya Chemudugunta · Padhraic Smyth · Mark Steyvers -
2006 Poster: Learning Time-Intensity Profiles of Human Activity using Non-Parametric Bayesian Models »
Alexander Ihler · Padhraic Smyth -
2006 Poster: Hierarchical Dirichlet Processes with Random Effects »
Seyoung Kim · Padhraic Smyth