Skip to yearly menu bar Skip to main content


( events)   Timezone:  
Workshop
Fri Dec 11 08:00 AM -- 04:00 PM (PST)
Crowd Science Workshop: Remoteness, Fairness, and Mechanisms as Challenges of Data Supply by Humans for Automation
Daria Baidakova · Fabio Casati · Alexey Drutsa · Dmitry Ustalov





Workshop Home Page

Despite the obvious advantages, automation driven by machine learning and artificial intelligence carries pitfalls for the lives of millions of people: disappearance of many well-established mass professions and consumption of labeled data that are produced by humans managed by out of time approach with full-time office work and pre-planned task types. Crowdsourcing methodology can be considered as an effective way to overcome these issues since it provides freedom for task executors in terms of place, time and which task type they want to work on. However, many potential participants of crowdsourcing processes hesitate to use this technology due to a series of doubts (that have not been removed during the past decade).

This workshop brings together people studying research questions on

(a) quality and effectiveness in remote crowd work;
(b) fairness and quality of life at work, tackling issues such as fair task assignment, fair work conditions, and on providing opportunities for growth; and
(c) economic mechanisms that incentivize quality and effectiveness for requester while maintaining a high level of quality and fairness for crowd performers (also known as workers).

Because quality, fairness and opportunities for crowd workers are central to our workshop, we will invite a diverse group of crowd workers from a global public crowdsourcing platform to our panel-led discussion.

Workshop web site: [ protected link dropped ]

Introduction & Icebreakers (Live Intro)
Data Excellence: Better Data for Better AI (by Lora Aroyo) (Invited Talk)
Q&A with Lora Aroyo "Data Excellence: Better Data for Better AI " (Live Q&A)
A Gamified Crowdsourcing Framework for Data-Driven Co-creation of Policy Making and Social Foresight (by Andrea Tocchetti and Marco Brambilla) (Talk)
Q&A with Andrea Tocchetti and Marco Brambilla "A Gamified Crowdsourcing Framework for Data-Driven Co-creation of Policy Making and Social Foresight" (Live Q&A)
Conversational Crowdsourcing (by Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben) (Talk)
Q&A with Sihang Qiu, Ujwal Gadiraju, Alessandro Bozzon and Geert-Jan Houben "Conversational Crowdsourcing" (Live Q&A)
Coffee Break
Quality Control in Crowdsourcing (by Seid Muhie Yimam) (Invited Talk)
Q&A with Seid Muhie Yimam "Quality Control in Crowdsourcing" (Live Q&A)
What Can Crowd Computing Do for the Next Generation of AI Technology? (by Ujwal Gadiraju and Jie Yang) (Talk)
Q&A with Ujwal Gadiraju and Jie Yang "What Can Crowd Computing Do for the Next Generation of AI Technology?" (Live Q&A)
Real-Time Crowdsourcing of Health Data in a Low-Income country: A case study of Human Data Supply on Malaria first-line treatment policy tracking in Nigeria (by Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo) (Talk)
Q&A with Olubayo Adekanmbi, Wuraola Fisayo Oyewusi and Ezekiel Ogundepo: "Real-Time Crowdsourcing of Health Data in a Low-Income country: A case study of Human Data Supply on Malaria first-line treatment policy tracking in Nigeria" (Live Q&A)
Coffee Break
Panel Discussion "Successes and failures in crowdsourcing: experiences from work providers, performers and platforms" (Panel Discussion)
Lunch Break
Modeling and Aggregation of Complex Annotations Via Annotation Distance (by Matt Lease) (Invited Talk)
Q&A with Matt Lease: "Modeling and Aggregation of Complex Annotations Via Annotation Distance" (Live Q&A)
Active Learning from Crowd in Item Screening (by Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik) (Talk)
Q&A with Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik: "Active Learning from Crowd in Item Screening" (Live Q&A)
Human Computation Requires and Enables a New Approach to Ethics (by Libuse Veprek, Patricia Seymour and Pietro Michelucci) (Talk)
Q&A with Libuse Veprek, Patricia Seymour and Pietro Michelucci: "Human computation requires and enables a new approach to ethics" (Live Q&A)
Coffee Break
Bias in Human-in-the-Loop Artificial Intelligence (by Gianluca Demartini) (Invited Talk)
Q&A with Gianluca Demartini: "Bias in Human-in-the-loop Artificial Intelligence" (Live Q&A)
VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts (by Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan) (Talk)
Q&A with Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan: " VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts" (Live Q&A)
Achieving Data Excellence (by Praveen Paritosh) (Invited Talk)
Q&A with Praveen Paritosh "Achieving Data Excellence" (Live Q&A)
Closing