Timezone: »
Challenges in machine learning and data science are competitions running over several weeks or months to resolve problems using provided datasets or simulated environments. The playful nature of challenges naturally attracts students, making challenge a great teaching resource. For this fifth edition of the CiML workshop at NIPS we want to go beyond simple data science challenges using canned data. We will explore the possibilities offered by challenges in which code submitted by participants are evaluated "in the wild", directly interacting in real time with users or with real or simulated systems. Organizing challenges "in the wild" is not new. One of the most impactful such challenge organized relatively recently is the DARPA grant challenge 2005 on autonomous navigation, which accelerated research on autonomous vehicles, leading to self-driving cars. Other high profile challenge series with live competitions include RoboCup, which has been running from the past 22 years. Recently, the machine learning community has started being interested in such interactive challenges, with last year at NIPS the learning to run challenge, an reinforcement learning challenge in which a human avatar had to be controlled with simulated muscular contractions, and the ChatBot challenge in which humans and robots had to engage into an intelligent conversation. Applications are countless for machine learning and artificial intelligence programs to solve problems in real time in the real world, by interacting with the environment. But organizing such challenges is far from trivial
The workshop will give a large part to discussions around two principal axes: (1) Design principles and implementation issues; (2) Opportunities to organize new impactful challenges.
Our objectives include bringing together potential partner to organize new such challenges and stimulating "machine learning for good", i.e. the organization of challenges for the benefit of society.
CiML is a forum that brings together workshop organizers, platform providers, and participants to discuss best practices in challenge organization and new methods and application opportunities to design high impact challenges. Following the success of previous years' workshops, we propose to reconvene and discuss new opportunities for challenges "in the wild", one of the hottest topics in challenge organization. We have invited prominent speakers having experience in this domain.
The audience of this workshop is targeted to workshop organizers, participants, and anyone with scientific problem involving machine learning, which may be formulated as a challenge. The emphasis of the workshop is on challenge design. Hence it complements nicely the workshop on the NIPS 2018 competition track and will help paving the way toward next year's competition program.
Submit abstract (up to 2 pages) before October 10 by sending email to nips2018@chalearn.org. See http://ciml.chalearn.org/ciml2018#CALL.
Sat 5:00 a.m. - 5:10 a.m.
|
Morning Welcome - - Isabelle Guyon and Evelyne Viegas
(
Announcement
)
|
Evelyne Viegas 🔗 |
Sat 5:10 a.m. - 5:40 a.m.
|
Esteban Arcaute and Umut Ozertem - Facebook project on developing benchmarks of algorithms in realistic settings
(
Invited talk
)
|
Esteban Arcaute 🔗 |
Sat 5:40 a.m. - 6:10 a.m.
|
Laura Seaman - Project Alloy – Machine Learning Challenges for Researching Human-Machine Teaming
(
Invited talk
)
|
Laura Seaman 🔗 |
Sat 6:10 a.m. - 7:40 a.m.
|
Live competiton 1: Pommerman
(
Competition
)
|
🔗 |
Sat 7:40 a.m. - 8:00 a.m.
|
Break, poster viewing 1
(
Poster
)
|
🔗 |
Sat 7:40 a.m. - 8:00 a.m.
|
Beyond the Leaderboard, Adrienne M. Mendrik, Stephen R. Aylward
(
Poster
)
|
Adrienne Mendrik · Stephen R Aylward 🔗 |
Sat 7:40 a.m. - 8:00 a.m.
|
Grand-Challenge.org, James A. Meakin Bram van Ginneken
(
Poster
)
|
James Meakin 🔗 |
Sat 7:40 a.m. - 8:00 a.m.
|
Corpus for AutoML Pipelines, Richard Lippmann, Swoop Vattam, Pooya Khorrami, and Cagri Dagli
(
Poster
)
|
Richard Lippmann · Pooya Khorrami 🔗 |
Sat 7:40 a.m. - 8:00 a.m.
|
How to fail hosting data science contests with images, Evgeny Nizhibitsky, Artur Kuzin
(
Poster
)
|
Artur Kuzin · Evgeny Nizhibitskiy 🔗 |
Sat 7:40 a.m. - 8:00 a.m.
|
ML Benchmark Tools Package, Ryan Turner, Uber AI Labs
(
Poster
)
|
Ryan Turner 🔗 |
Sat 8:00 a.m. - 8:30 a.m.
|
Julien Hay, Bich-Liên Doan, Fabrice Popineau - Renewal news recommendation platform
(
Invited talk
)
|
Julien Hay · Bich-Liên DOAN 🔗 |
Sat 8:30 a.m. - 9:00 a.m.
|
Panel discussion. Design principles and implementation issues.
(
Discussion
)
|
Sergio Escalera 🔗 |
Sat 9:00 a.m. - 10:00 a.m.
|
Lunch Break, poster viewing 2
(
Poster
)
|
🔗 |
Sat 10:00 a.m. - 10:10 a.m.
|
Afternoon Welcome - Isabelle Guyon and Evelyne Viegas
(
Announcement
)
|
Isabelle Guyon 🔗 |
Sat 10:10 a.m. - 10:40 a.m.
|
Mikhail Burtsev and Varvara Logacheva - Wild evaluation of chat-bots
(
Invited talk
)
|
Mikhail Burtsev · Varvara Logacheva 🔗 |
Sat 10:40 a.m. - 11:10 a.m.
|
Larry Jackel - Measuring Progress in Robotics
(
Invited talk
)
|
Lawrence Jackel 🔗 |
Sat 11:10 a.m. - 11:40 a.m.
|
Daniel Polani - Competitions to Challenge Artificial Intelligence: from the L-Game to RoboCup
(
Invited talk
)
|
Daniel Polani 🔗 |
Sat 11:40 a.m. - 12:10 p.m.
|
Antoine Marot - Learning to run a power network
(
Invited talk
)
|
Antoine Marot 🔗 |
Sat 12:10 p.m. - 12:30 p.m.
|
Break, poster viewing 3
(
Poster
)
|
🔗 |
Sat 12:10 p.m. - 12:30 p.m.
|
NASA Frontier Development Lab 2018, A. Bella, A. Chopra, W. Fawcett, R. Talebi, D. Angerhausen, A. Berea, N.A. Cabrol, C. Kempes, M. Mascaro
(
Poster
)
|
Aaron C Bell 🔗 |
Sat 12:10 p.m. - 12:30 p.m.
|
AutoDL challenge design and beta tests, Zhengying Liu, Olivier Bousquet, Andre Elisseeff, Isabelle Guyon, Adrien Pavao, Lisheng Sun-Hosoya, and Sebastien Treguer
(
Poster
)
|
Zhengying Liu · Sébastien Treguer 🔗 |
Sat 12:10 p.m. - 12:30 p.m.
|
TrackML, a Particle Physics Tracking Machine Learning Challenge, Jean-Roch Vlimant (Caltech), Vincenzo Innocente, Andreas Salzburger (CERN), Isabelle Guyon (ChaLearn), Sabrina Amrouche, Tobias Golling, Moritz Kiehn (Geneva University),David Rousseau∗, Yet
(
Poster
)
|
Andrey Ustyuzhanin · jean-roch vlimant 🔗 |
Sat 12:10 p.m. - 12:30 p.m.
|
L2RPN: Learning To Run a Power Network Competition, Antoine Marot, Balthazar Donon, Isabelle Guyon, Benjamin Donnot.
(
Poster
)
|
Benjamin Donnot 🔗 |
Sat 12:10 p.m. - 12:30 p.m.
|
Multi-Agent RL in MalmO (MARLO) Competition, Diego Perez-Liebana Katja Hofmann Sharada Prasanna Mohanty Noburu Kuno Andre Kramer Sam Devlin Raluca D. Gaina Daniel Ionita
(
Poster
)
|
Sharada Mohanty 🔗 |
Sat 12:30 p.m. - 2:00 p.m.
|
Live competition 2: The driving AI Olympics
(
Competition
)
|
🔗 |
Sat 2:30 p.m. - 2:30 p.m.
|
Panel discussion: Opportunities to organize new impactful challenges.
(
Discussion
)
|
Jacob Abernethy 🔗 |
Author Information
Isabelle Guyon (U. Paris-Saclay & ChaLearn)
Isabelle Guyon recently joined Google Brain as a research scientist. She is also professor of artificial intelligence at Université Paris-Saclay (Orsay). Her areas of expertise include computer vision, bioinformatics, and power systems. She is best known for being a co-inventor of Support Vector Machines. Her recent interests are in automated machine learning, meta-learning, and data-centric AI. She has been a strong promoter of challenges and benchmarks, and is president of ChaLearn, a non-profit dedicated to organizing machine learning challenges. She is community lead of Codalab competitions, a challenge platform used both in academia and industry. She co-organized the “Challenges in Machine Learning Workshop” @ NeurIPS between 2014 and 2019, launched the "NeurIPS challenge track" in 2017 while she was general chair, and pushed the creation of the "NeurIPS datasets and benchmark track" in 2021, as a NeurIPS board member.
Evelyne Viegas (Microsoft Research)
Sergio Escalera (University of Barcelona and Computer Vision Center)
Sergio Escalera obtained the P.h.D. degree on Multi-class visual categorization systems at Computer Vision Center, UAB. He obtained the 2008 best Thesis award on Computer Science at Universitat Autònoma de Barcelona. He leads the Human Pose Recovery and Behavior Analysis Group at UB, CVC, and the Barcelona Graduate School of Mathematics. He is an associate professor at the Department of Mathematics and Informatics, Universitat de Barcelona. He is an adjunct professor at Universitat Oberta de Catalunya, Aalborg University, and Dalhousie University. He has been visiting professor at TU Delft and Aalborg Universities. He is a member of the Visual and Computational Learning consolidated research group of Catalonia. He is also a member of the Computer Vision Center at Campus UAB. He is Editor-in-Chief of American Journal of Intelligent Systems and editorial board member of more than 5 international journals. He is advisor, director, and vice-president of ChaLearn Challenges in Machine Learning, leading ChaLearn Looking at People events. He is co-founder of PhysicalTech and Care Respite companies. He is also member of the AERFAI Spanish Association on Pattern Recognition, ACIA Catalan Association of Artificial Intelligence, and he is vice-chair of IAPR TC-12: Multimedia and visual information systems. He has different patents and registered models. He has published more than 150 research papers and participated in the organization of scientific events, including CCIA04, CCIA14, ICCV11, AMDO2016, FG2017, and workshops at ICCV11, ICMI13, ECCV14, CVPR15, ICCV15, CVPR16, ECCV16, ICPR16, NIPS16. He has been guest editor at JMLR, TPAMI, IJCV, TAC, and Neural Comp. and App. He has been area chair at WACV16, NIPS16, and FG17. His research interests include, between others, statistical pattern recognition, visual object recognition, and HCI systems, with special interest in human pose recovery and behavior analysis from multi-modal data.
Jacob D Abernethy (University of Michigan)
More from the Same Authors
-
2022 : Fifteen-minute Competition Overview Video »
Dustin Carrión-Ojeda · Ihsan Ullah · Sergio Escalera · Isabelle Guyon · Felix Mohr · Manh Hung Nguyen · Joaquin Vanschoren -
2022 Competition: Cross-Domain MetaDL: Any-Way Any-Shot Learning Competition with Novel Datasets from Practical Domains »
Dustin Carrión-Ojeda · Ihsan Ullah · Sergio Escalera · Isabelle Guyon · Felix Mohr · Manh Hung Nguyen · Joaquin Vanschoren -
2022 Poster: Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification »
Ihsan Ullah · Dustin Carrión-Ojeda · Sergio Escalera · Isabelle Guyon · Mike Huisman · Felix Mohr · Jan N. van Rijn · Haozhe Sun · Joaquin Vanschoren · Phan Anh Vu -
2022 : Isabelle Guyon »
Isabelle Guyon -
2022 Invited Talk: The Data-Centric Era: How ML is Becoming an Experimental Science »
Isabelle Guyon -
2022 : NeurIPS Competitions – Evolution and Opportunities »
Isabelle Guyon · Evelyne Viegas -
2021 Panel: The Role of Benchmarks in the Scientific Progress of Machine Learning »
Lora Aroyo · Samuel Bowman · Isabelle Guyon · Joaquin Vanschoren -
2020 : Keynote talk by Isabelle Guyon and Evelyne Viegas - "AI Competitions and the Science Behind Contests" »
Isabelle Guyon · Evelyne Viegas -
2020 Poster: Deep Statistical Solvers »
Balthazar Donon · Zhengying Liu · Wenzhuo LIU · Isabelle Guyon · Antoine Marot · Marc Schoenauer -
2019 : Open Space Topic “The Organization of Challenges for the Benefit of More Diverse Communities” »
Adrienne Mendrik · Isabelle Guyon · Wei-Wei Tu · Evelyne Viegas · Ming LI -
2019 Workshop: CiML 2019: Machine Learning Competitions for All »
Adrienne Mendrik · Wei-Wei Tu · Wei-Wei Tu · Isabelle Guyon · Evelyne Viegas · Ming LI -
2019 : Welcome and Opening Remarks »
Adrienne Mendrik · Wei-Wei Tu · Isabelle Guyon · Evelyne Viegas · Ming LI -
2018 : Afternoon Welcome - Isabelle Guyon and Evelyne Viegas »
Isabelle Guyon -
2018 : Panel discussion. Design principles and implementation issues. »
Sergio Escalera -
2018 : Panel discussion. Design principles and implementation issues »
Sergio Escalera -
2018 Workshop: NeurIPS 2018 Competition Track Day 2 »
Ralf Herbrich · Sergio Escalera -
2018 : Morning Welcome - - Isabelle Guyon and Evelyne Viegas »
Evelyne Viegas -
2018 : Datasets and Benchmarks for Causal Learning »
Csaba Szepesvari · Isabelle Guyon · Nicolai Meinshausen · David Blei · Elias Bareinboim · Bernhard Schölkopf · Pietro Perona -
2018 : Building Algorithms by Playing Games »
Jacob D Abernethy -
2018 : AutoML3 - LifeLong ML with concept drift Challenge: Overview and award ceremony »
Hugo Jair Escalante · Isabelle Guyon · Daniel Silver · Evelyne Viegas · Wei-Wei Tu -
2018 : Evaluating Causation Coefficients »
Isabelle Guyon -
2018 Workshop: NeurIPS 2018 Competition Track Day 1 »
Sergio Escalera · Ralf Herbrich -
2018 : Opening and Best Demo Award announcement »
Sergio Escalera · Ralf Herbrich -
2017 Workshop: Machine Learning Challenges as a Research Tool »
Isabelle Guyon · Evelyne Viegas · Sergio Escalera · Jacob D Abernethy -
2017 : Introduction - Isabelle Guyon and Evelyne Viegas »
Isabelle Guyon -
2017 : Closing »
Sergio Escalera · Markus Weimer -
2017 : Opening »
Sergio Escalera · Markus Weimer -
2017 Workshop: Competition track »
Sergio Escalera · Markus Weimer -
2017 Poster: On Frank-Wolfe and Equilibrium Computation »
Jacob D Abernethy · Jun-Kun Wang -
2017 Spotlight: On Frank-Wolfe and Equilibrium Computation »
Jacob D Abernethy · Jun-Kun Wang -
2016 Workshop: Machine Learning for Spatiotemporal Forecasting »
Florin Popescu · Sergio Escalera · Xavier Baró · Stephane Ayache · Isabelle Guyon -
2016 : Gaming challenges and encouraging collaborations »
Sergio Escalera · Isabelle Guyon -
2016 Workshop: Challenges in Machine Learning: Gaming and Education »
Isabelle Guyon · Evelyne Viegas · Balázs Kégl · Ben Hamner · Sergio Escalera -
2016 : Welcome »
Evelyne Viegas -
2016 Demonstration: Biometric applications of CNNs: get a job at "Impending Technologies"! »
Sergio Escalera · Isabelle Guyon · Baiyu Chen · Marc Quintana · Umut Güçlü · Yağmur Güçlütürk · Xavier Baró · Rob van Lier · Carlos Andujar · Marcel A. J. van Gerven · Bernhard E Boser · Luke Wang -
2016 Demonstration: Project Malmo - Minecraft for AI Research »
Katja Hofmann · Matthew A Johnson · Fernando Diaz · Alekh Agarwal · Tim Hutton · David Bignell · Evelyne Viegas -
2016 Poster: Threshold Bandits, With and Without Censored Feedback »
Jacob D Abernethy · Kareem Amin · Ruihao Zhu -
2015 Workshop: Challenges in Machine Learning (CiML 2015): "Open Innovation" and "Coopetitions" »
Isabelle Guyon · Evelyne Viegas · Ben Hamner · Balázs Kégl -
2015 Poster: Fighting Bandits with a New Kind of Smoothness »
Jacob D Abernethy · Chansoo Lee · Ambuj Tewari -
2015 Poster: A Market Framework for Eliciting Private Data »
Bo Waggoner · Rafael Frongillo · Jacob D Abernethy -
2015 Demonstration: CodaLab Worksheets for Reproducible, Executable Papers »
Percy Liang · Evelyne Viegas -
2014 Workshop: High-energy particle physics, machine learning, and the HiggsML data challenge (HEPML) »
Glen Cowan · Balázs Kégl · Kyle Cranmer · Gábor Melis · Tim Salimans · Vladimir Vava Gligorov · Daniel Whiteson · Lester Mackey · Wojciech Kotlowski · Roberto Díaz Morales · Pierre Baldi · Cecile Germain · David Rousseau · Isabelle Guyon · Tianqi Chen -
2014 Workshop: Challenges in Machine Learning workshop (CiML 2014) »
Isabelle Guyon · Evelyne Viegas · Percy Liang · Olga Russakovsky · Rinat Sergeev · Gábor Melis · Michele Sebag · Gustavo Stolovitzky · Jaume Bacardit · Michael S Kim · Ben Hamner -
2014 Workshop: NIPS Workshop on Transactional Machine Learning and E-Commerce »
David Parkes · David H Wolpert · Jennifer Wortman Vaughan · Jacob D Abernethy · Amos Storkey · Mark Reid · Ping Jin · Nihar Bhadresh Shah · Mehryar Mohri · Luis E Ortiz · Robin Hanson · Aaron Roth · Satyen Kale · Sebastien Lahaie -
2013 Workshop: NIPS 2013 Workshop on Causality: Large-scale Experiment Design and Inference of Causal Mechanisms »
Isabelle Guyon · Leon Bottou · Bernhard Schölkopf · Alexander Statnikov · Evelyne Viegas · james m robins -
2013 Poster: Minimax Optimal Algorithms for Unconstrained Linear Optimization »
Brendan McMahan · Jacob D Abernethy -
2013 Poster: Adaptive Market Making via Online Learning »
Jacob D Abernethy · Satyen Kale -
2013 Poster: How to Hedge an Option Against an Adversary: Black-Scholes Pricing is Minimax Optimal »
Jacob D Abernethy · Peter Bartlett · Rafael Frongillo · Andre Wibisono -
2013 Spotlight: How to Hedge an Option Against an Adversary: Black-Scholes Pricing is Minimax Optimal »
Jacob D Abernethy · Peter Bartlett · Rafael Frongillo · Andre Wibisono -
2013 Oral: Adaptive Market Making via Online Learning »
Jacob D Abernethy · Satyen Kale -
2012 Demonstration: Gesture recognition with Kinect »
Isabelle Guyon -
2011 Poster: A Collaborative Mechanism for Crowdsourcing Prediction Problems »
Jacob D Abernethy · Rafael Frongillo -
2011 Oral: A Collaborative Mechanism for Crowdsourcing Prediction Problems »
Jacob D Abernethy · Rafael Frongillo -
2010 Poster: Repeated Games against Budgeted Adversaries »
Jacob D Abernethy · Manfred K. Warmuth -
2009 Workshop: Clustering: Science or art? Towards principled approaches »
Margareta Ackerman · Shai Ben-David · Avrim Blum · Isabelle Guyon · Ulrike von Luxburg · Robert Williamson · Reza Zadeh -
2009 Mini Symposium: Causality and Time Series Analysis »
Florin Popescu · Isabelle Guyon · Guido Nolte -
2009 Demonstration: Causality Workbench »
Isabelle Guyon -
2008 Workshop: Causality: objectives and assessment »
Isabelle Guyon · Dominik Janzing · Bernhard Schölkopf -
2007 Demonstration: CLOP: a Matlab Learning Object Package »
Amir Reza Saffari Azar Alamdari · Isabelle Guyon · Hugo Jair Escalante · Gökhan H Bakir · Gavin Cawley -
2006 Workshop: Multi-level Inference Workshop and Model Selection Game »
Isabelle Guyon