Workshop
Crowdsourcing: Theory, Algorithms and Applications
Jennifer Wortman Vaughan · Greg Stoddard · Chien-Ju Ho · Adish Singla · Michael Bernstein · Devavrat Shah · Arpita Ghosh · Evgeniy Gabrilovich · Denny Zhou · Nikhil Devanur · Xi Chen · Alexander Ihler · Qiang Liu · Genevieve Patterson · Ashwinkumar Badanidiyuru Varadaraja · Hossein Azari Soufiani · Jacob Whitehill

Mon Dec 9th 07:30 AM -- 06:30 PM @ Harrah's Tahoe A+B
Event URL: http://www.ics.uci.edu/~qliu1/nips13_workshop/ »

All machine learning systems are an integration of data that store human or physical knowledge, and algorithms that discover knowledge patterns and make predictions to new instances. Even though most research attention has been focused on developing more efficient learning algorithms, it is the quality and amount of training data that predominately govern the performance of real-world systems. This is only amplified by the recent popularity of large scale and complicated learning systems such as deep networks, which require millions to billions of training data to perform well. Unfortunately, the traditional methods of collecting data from specialized workers are usually expensive and slow. In recent years, however, the situation has dramatically changed with the emergence of crowdsourcing, where huge amounts of labeled data are collected from large groups of (usually online) workers for low or no cost. Many machine learning tasks, such as computer vision and natural language processing are increasingly benefitting from data crowdsourced platforms such as Amazon Mechanical Turk and CrowdFlower. On the other hand, tools in machine learning, game theory and mechanism design can help to address many challenging problems in crowdsourcing systems, such as making them more reliable, efficient and less expensive.

In this workshop, we call attention back to sources of data, discussing cheap and fast data collection methods based on crowdsourcing, and how it could impact subsequent machine learning stages.
Furthermore, we will emphasize how the data sourcing paradigm interacts with the most recent emerging trends of machine learning in NIPS community.

Examples of topics of potential interest in the workshop include (but are not limited to):

Application of crowdsourcing to machine learning.

Reliable crowdsourcing, e.g., label aggregation, quality control.

Optimal budget allocation or active learning in crowdsourcing.

Workflow design and answer aggregation for complex tasks (e.g., machine translation, proofreading).

Pricing and incentives in crowdsourcing markets.

Prediction markets / information markets and its connection to learning.

Theoretical analysis for crowdsourcing algorithms, e.g., error rates and sample complexities for label aggregation and budget allocation algorithms.

Author Information

Jenn Wortman Vaughan (Microsoft Research)

Jenn Wortman Vaughan is a Senior Researcher at Microsoft Research, New York City, where she studies algorithmic economics, machine learning, and social computing, with a frequent focus on prediction markets and crowdsourcing. Jenn came to MSR in 2012 from UCLA, where she was an assistant professor in the computer science department. She completed her Ph.D. at the University of Pennsylvania in 2009, and subsequently spent a year as a Computing Innovation Fellow at Harvard. She is the recipient of Penn's 2009 Rubinoff dissertation award for innovative applications of computer technology, a National Science Foundation CAREER award, a Presidential Early Career Award for Scientists and Engineers (PECASE), and a handful of best paper or best student paper awards. In her "spare" time, Jenn is involved in a variety of efforts to provide support for women in computer science; most notably, she co-founded the Annual Workshop for Women in Machine Learning, which has been held each year since 2006.

Greg Stoddard (Northwestern University)
Chien-Ju Ho (UCLA)
Adish Singla (MPI-SWS)
Michael Bernstein (Stanford University)
Devavrat Shah (Massachusetts Institute of Technology)

Devavrat Shah is a professor of Electrical Engineering & Computer Science and Director of Statistics and Data Science at MIT. He received PhD in Computer Science from Stanford. He received Erlang Prize from Applied Probability Society of INFORMS in 2010 and NeuIPS best paper award in 2008.

Arpita Ghosh (Cornell University)
Evgeniy Gabrilovich (Google)
Denny Zhou (Microsoft Research Redmond)
Nikhil Devanur (Microsoft Research)
Xi Chen (NYU)
Alexander Ihler (UC Irvine)
Qiang Liu (UC Irvine)
Genevieve Patterson (TRASH Inc)
Ashwinkumar Badanidiyuru Varadaraja (Google Research)
Hossein Azari Soufiani (Harvard University)
Jacob Whitehill (University of California, San Diego)

More from the Same Authors