Timezone: »

 
Workshop
Challenges in Machine Learning (CiML 2015): "Open Innovation" and "Coopetitions"
Isabelle Guyon · Evelyne Viegas · Ben Hamner · Balázs Kégl

Sat Dec 12 05:30 AM -- 03:30 PM (PST) @ 512 e
Event URL: http://ciml.chalearn.org/ »

Challenges in Machine Learning have proven to be efficient and cost-effective ways to quickly bring to industry solutions that may have been confined to research. In addition, the playful nature of challenges naturally attracts students, making challenge a great teaching resource. Challenge participants range from undergraduate students to retirees, joining forces in a rewarding environment allowing them to learn, perform research, and demonstrate excellence. Therefore challenges can be used as a means of directing research, advancing the state-of-the-art or venturing in completely new domains.

Because challenges have become stream line in the execution of Machine Learning projects, it has become increasingly important to regularly bring together workshop organizers, platform providers, and participants to discuss best practices in challenge organization and new methods and application opportunities to design high impact challenges. Following the success of last year's workshop (http://ciml.chalearn.org/), in which a fruitful exchange led to many innovations, we propose to reconvene and discuss the new avenues that have been explored and lay the basis for further developments. We are particularly interested in following progresses made in two conceptually important directions:
1) Open innovation: Organization of contests in which data are made available and the contestants must both formalize and solve a problem (with some constraints), leaving more freedom to creativity, while giving more difficulty to the organizers to objectively assess the results.
2) Coopetitions: Organization of contests encouraging both collaboration and competition, in an effort to make possible the contributions of many towards a the grand goal of solving the overall problem; this poses to the organizers the problem of rewarding partial contributions.

We also want to closely follow more technical, albeit important aspects:
3) Platforms: New developments including "code submission" (platforms and protocols permitting code submission, as opposed to result submission, allowing fairer standardized comparisons in terms of hardware utilization and easier reproducibility) and " worksheets" or "scripts" facilitating code sharing.
4) Sharing, dissemination, and recognition: Facilitate sharing resources, including data, means of data collection and annotation, challenge announcements, best practices, challenge templates, publication channels, etc.; creation of awards to recognize academic services rendered by the various actors of challenge organization.

Author Information

Isabelle Guyon (Clopinet)

Isabelle Guyon recently joined Google Brain as a research scientist. She is also professor of artificial intelligence at Université Paris-Saclay (Orsay). Her areas of expertise include computer vision, bioinformatics, and power systems. She is best known for being a co-inventor of Support Vector Machines. Her recent interests are in automated machine learning, meta-learning, and data-centric AI. She has been a strong promoter of challenges and benchmarks, and is president of ChaLearn, a non-profit dedicated to organizing machine learning challenges. She is community lead of Codalab competitions, a challenge platform used both in academia and industry. She co-organized the “Challenges in Machine Learning Workshop” @ NeurIPS between 2014 and 2019, launched the "NeurIPS challenge track" in 2017 while she was general chair, and pushed the creation of the "NeurIPS datasets and benchmark track" in 2021, as a NeurIPS board member.

Evelyne Viegas (Microsoft Research)
Ben Hamner (Kaggle)
Balázs Kégl (Université Paris Saclay/CNRS)

More from the Same Authors