Workshop
CiML 2018 - Machine Learning competitions "in the wild": Playing in the real world or in real time
Isabelle Guyon · Evelyne Viegas · Sergio Escalera · Jacob D Abernethy

Sat Dec 8th 08:00 AM -- 06:30 PM @ Room 511 ABDE
Event URL: http://ciml.chalearn.org/ »

Challenges in machine learning and data science are competitions running over several weeks or months to resolve problems using provided datasets or simulated environments. The playful nature of challenges naturally attracts students, making challenge a great teaching resource. For this fifth edition of the CiML workshop at NIPS we want to go beyond simple data science challenges using canned data. We will explore the possibilities offered by challenges in which code submitted by participants are evaluated "in the wild", directly interacting in real time with users or with real or simulated systems. Organizing challenges "in the wild" is not new. One of the most impactful such challenge organized relatively recently is the DARPA grant challenge 2005 on autonomous navigation, which accelerated research on autonomous vehicles, leading to self-driving cars. Other high profile challenge series with live competitions include RoboCup, which has been running from the past 22 years. Recently, the machine learning community has started being interested in such interactive challenges, with last year at NIPS the learning to run challenge, an reinforcement learning challenge in which a human avatar had to be controlled with simulated muscular contractions, and the ChatBot challenge in which humans and robots had to engage into an intelligent conversation. Applications are countless for machine learning and artificial intelligence programs to solve problems in real time in the real world, by interacting with the environment. But organizing such challenges is far from trivial
The workshop will give a large part to discussions around two principal axes: (1) Design principles and implementation issues; (2) Opportunities to organize new impactful challenges.
Our objectives include bringing together potential partner to organize new such challenges and stimulating "machine learning for good", i.e. the organization of challenges for the benefit of society.
CiML is a forum that brings together workshop organizers, platform providers, and participants to discuss best practices in challenge organization and new methods and application opportunities to design high impact challenges. Following the success of previous years' workshops, we propose to reconvene and discuss new opportunities for challenges "in the wild", one of the hottest topics in challenge organization. We have invited prominent speakers having experience in this domain.
The audience of this workshop is targeted to workshop organizers, participants, and anyone with scientific problem involving machine learning, which may be formulated as a challenge. The emphasis of the workshop is on challenge design. Hence it complements nicely the workshop on the NIPS 2018 competition track and will help paving the way toward next year's competition program.
Submit abstract (up to 2 pages) before October 10 by sending email to nips2018@chalearn.org. See http://ciml.chalearn.org/ciml2018#CALL.

08:00 AM Morning Welcome - - Isabelle Guyon and Evelyne Viegas (Announcement) Evelyne Viegas
08:10 AM Esteban Arcaute and Umut Ozertem - Facebook project on developing benchmarks of algorithms in realistic settings (Invited talk) Esteban Arcaute
08:40 AM Laura Seaman - Project Alloy – Machine Learning Challenges for Researching Human-Machine Teaming (Invited talk) Laura Seaman
09:10 AM Live competiton 1: Pommerman (Competition)
10:40 AM Beyond the Leaderboard, Adrienne M. Mendrik, Stephen R. Aylward (Poster) Adrienne Mendrik, Stephen R Aylward
10:40 AM Grand-Challenge.org, James A. Meakin Bram van Ginneken (Poster) James Meakin
10:40 AM Break, poster viewing 1 (Poster)
10:40 AM ML Benchmark Tools Package, Ryan Turner, Uber AI Labs (Poster) Ryan Turner
10:40 AM Corpus for AutoML Pipelines, Richard Lippmann, Swoop Vattam, Pooya Khorrami, and Cagri Dagli (Poster) Richard Lippmann, Pooya Khorrami
10:40 AM How to fail hosting data science contests with images, Evgeny Nizhibitsky, Artur Kuzin (Poster) n01z3 Kuzin, Evgeny Nizhibitskiy
11:00 AM Julien Hay, Bich-Liên Doan, Fabrice Popineau - Renewal news recommendation platform (Invited talk) Julien Hay, Bich-Liên DOAN
11:30 AM Panel discussion. Design principles and implementation issues. (Discussion) Sergio Escalera
12:00 PM Lunch Break, poster viewing 2 (Poster)
01:00 PM Afternoon Welcome - Isabelle Guyon and Evelyne Viegas (Announcement) Isabelle Guyon
01:10 PM Mikhail Burtsev and Varvara Logacheva - Wild evaluation of chat-bots (Invited talk) Mikhail Burtsev, Varvara Logacheva
01:40 PM Larry Jackel - Measuring Progress in Robotics (Invited talk) Lawrence Jackel
02:10 PM Daniel Polani - Competitions to Challenge Artificial Intelligence: from the L-Game to RoboCup (Invited talk) Daniel Polani
02:40 PM Antoine Marot - Learning to run a power network (Invited talk) Antoine Marot
03:10 PM Break, poster viewing 3 (Poster)
03:10 PM L2RPN: Learning To Run a Power Network Competition, Antoine Marot, Balthazar Donon, Isabelle Guyon, Benjamin Donnot. (Poster) Benjamin Donnot
03:10 PM AutoDL challenge design and beta tests, Zhengying Liu, Olivier Bousquet, Andre Elisseeff, Isabelle Guyon, Adrien Pavao, Lisheng Sun-Hosoya, and Sebastien Treguer (Poster) Zhengying Liu, Sébastien Treguer
03:10 PM Multi-Agent RL in MalmO (MARLO) Competition, Diego Perez-Liebana Katja Hofmann Sharada Prasanna Mohanty Noburu Kuno Andre Kramer Sam Devlin Raluca D. Gaina Daniel Ionita (Poster) Sharada Mohanty
03:10 PM NASA Frontier Development Lab 2018, A. Bella, A. Chopra, W. Fawcett, R. Talebi, D. Angerhausen, A. Berea, N.A. Cabrol, C. Kempes, M. Mascaro (Poster) Aaron C Bell
03:10 PM TrackML, a Particle Physics Tracking Machine Learning Challenge, Jean-Roch Vlimant (Caltech), Vincenzo Innocente, Andreas Salzburger (CERN), Isabelle Guyon (ChaLearn), Sabrina Amrouche, Tobias Golling, Moritz Kiehn (Geneva University),David Rousseau∗, Yet (Poster) Andrey Ustyuzhanin, jean-roch vlimant
03:30 PM Live competition 2: The driving AI Olympics (Competition)
05:30 PM Panel discussion: Opportunities to organize new impactful challenges. (Discussion) Jacob Abernethy

Author Information

Isabelle Guyon (U. Paris-Saclay & ChaLearn)
Evelyne Viegas (Microsoft Research)
Sergio Escalera (University of Barcelona and Computer Vision Center)

Sergio Escalera obtained the P.h.D. degree on Multi-class visual categorization systems at Computer Vision Center, UAB. He obtained the 2008 best Thesis award on Computer Science at Universitat Autònoma de Barcelona. He leads the Human Pose Recovery and Behavior Analysis Group at UB, CVC, and the Barcelona Graduate School of Mathematics. He is an associate professor at the Department of Mathematics and Informatics, Universitat de Barcelona. He is an adjunct professor at Universitat Oberta de Catalunya, Aalborg University, and Dalhousie University. He has been visiting professor at TU Delft and Aalborg Universities. He is a member of the Visual and Computational Learning consolidated research group of Catalonia. He is also a member of the Computer Vision Center at Campus UAB. He is Editor-in-Chief of American Journal of Intelligent Systems and editorial board member of more than 5 international journals. He is advisor, director, and vice-president of ChaLearn Challenges in Machine Learning, leading ChaLearn Looking at People events. He is co-founder of PhysicalTech and Care Respite companies. He is also member of the AERFAI Spanish Association on Pattern Recognition, ACIA Catalan Association of Artificial Intelligence, and he is vice-chair of IAPR TC-12: Multimedia and visual information systems. He has different patents and registered models. He has published more than 150 research papers and participated in the organization of scientific events, including CCIA04, CCIA14, ICCV11, AMDO2016, FG2017, and workshops at ICCV11, ICMI13, ECCV14, CVPR15, ICCV15, CVPR16, ECCV16, ICPR16, NIPS16. He has been guest editor at JMLR, TPAMI, IJCV, TAC, and Neural Comp. and App. He has been area chair at WACV16, NIPS16, and FG17. His research interests include, between others, statistical pattern recognition, visual object recognition, and HCI systems, with special interest in human pose recovery and behavior analysis from multi-modal data.

Jacob D Abernethy (University of Michigan)

More from the Same Authors