Active Learning from Crowd in Item Screening (by Evgeny Krivosheev, Burcu Sayin, Alessandro Bozzon and Zoltán Szlávik)

In this paper, we explore how to efficiently combine crowdsourcing and machine intelligence for the problem of document screening, where we need to screen a finite number of documents with a set of machine-learning filters. Specifically, we focus on building a set of machine learning classifiers that evaluate documents, and then screen them efficiently. It is a challenging task since the budget is limited and there are countless number of ways to spend the given budget on the problem. We propose a multi-label active learning screening specific sampling technique -objective-aware sampling- for querying unlabelled documents for annotating. Our algorithm takes a decision on which machine filter needs more training data and how to choose unlabeled items to annotate in order to minimize the risk of overall classification errors rather than minimizing a single filter error. Our results demonstrate that objective-aware sampling significantly outperforms the state of the art sampling strategies on multi-filter classification problems.