Timezone: »
We study the problem of estimating continuous quantities, such as prices, probabilities, and point spreads, using a crowdsourcing approach. A challenging aspect of combining the crowd's answers is that workers' reliabilities and biases are usually unknown and highly diverse. Control items with known answers can be used to evaluate workers' performance, and hence improve the combined results on the target items with unknown answers. This raises the problem of how many control items to use when the total number of items each workers can answer is limited: more control items evaluates the workers better, but leaves fewer resources for the target items that are of direct interest, and vice versa. We give theoretical results for this problem under different scenarios, and provide a simple rule of thumb for crowdsourcing practitioners. As a byproduct, we also provide theoretical analysis of the accuracy of different consensus methods.
Author Information
Qiang Liu (UC Irvine)
Alexander Ihler (UC Irvine)
Mark Steyvers (UC Irvine)
Related Events (a corresponding poster, oral, or spotlight)
-
2013 Spotlight: Scoring Workers in Crowdsourcing: How Many Control Questions are Enough? »
Sat. Dec 7th 06:14 -- 06:18 PM Room Harvey's Convention Center Floor, CC
More from the Same Authors
-
2021 : Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates »
Litian Liang · Yaosheng Xu · Stephen McAleer · Dailin Hu · Alexander Ihler · Pieter Abbeel · Roy Fox -
2023 Poster: Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning »
Xinyi Wang · Wanrong Zhu · Michael Saxon · Mark Steyvers · William Yang Wang -
2021 Poster: Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration »
Gavin Kerrigan · Padhraic Smyth · Mark Steyvers -
2020 Poster: Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference »
Disi Ji · Padhraic Smyth · Mark Steyvers -
2018 Poster: Lifted Weighted Mini-Bucket »
Nicholas Gallo · Alexander Ihler -
2017 Workshop: NIPS Highlights (MLTrain), Learn How to code a paper with state of the art frameworks »
Alex Dimakis · Nikolaos Vasiloglou · Guy Van den Broeck · Alexander Ihler · Assaf Araki -
2017 Poster: Dynamic Importance Sampling for Anytime Bounds of the Partition Function »
Qi Lou · Rina Dechter · Alexander Ihler -
2016 Poster: Learning Infinite RBMs with Frank-Wolfe »
Wei Ping · Qiang Liu · Alexander Ihler -
2015 Poster: Probabilistic Variational Bounds for Graphical Models »
Qiang Liu · John Fisher III · Alexander Ihler -
2015 Poster: Decomposition Bounds for Marginal MAP »
Wei Ping · Qiang Liu · Alexander Ihler -
2014 Poster: Distributed Estimation, Information Loss and Exponential Families »
Qiang Liu · Alexander Ihler -
2013 Workshop: Crowdsourcing: Theory, Algorithms and Applications »
Jennifer Wortman Vaughan · Greg Stoddard · Chien-Ju Ho · Adish Singla · Michael Bernstein · Devavrat Shah · Arpita Ghosh · Evgeniy Gabrilovich · Denny Zhou · Nikhil Devanur · Xi Chen · Alexander Ihler · Qiang Liu · Genevieve Patterson · Ashwinkumar Badanidiyuru Varadaraja · Hossein Azari Soufiani · Jacob Whitehill -
2013 Poster: Variational Planning for Graph-based MDPs »
Qiang Cheng · Qiang Liu · Feng Chen · Alexander Ihler -
2012 Poster: Variational Inference for Crowdsourcing »
Qiang Liu · Jian Peng · Alexander Ihler -
2010 Spotlight: Learning concept graphs from text with stick-breaking priors »
America Chambers · Padhraic Smyth · Mark Steyvers -
2010 Poster: Learning concept graphs from text with stick-breaking priors »
America Chambers · Padhraic Smyth · Mark Steyvers -
2009 Poster: Particle-based Variational Inference for Continuous Systems »
Alexander Ihler · Andrew Frank · Padhraic Smyth -
2009 Poster: The Wisdom of Crowds in the Recollection of Order Information »
Mark Steyvers · Michael D Lee · Brent Miller · Pernille Hemmer -
2006 Poster: Learning Time-Intensity Profiles of Human Activity using Non-Parametric Bayesian Models »
Alexander Ihler · Padhraic Smyth