Datasets and Benchmarks
Dataset and Benchmark Poster Session 1
Joaquin Vanschoren 路 Serena Yeung
Moderator s: Viorica Patraucean 路 Ludwig Schmidt
The Datasets and Benchmarks track serves as a novel venue for high-quality publications, talks, and posters on highly valuable machine learning datasets and benchmarks, as well as a forum for discussions on how to improve dataset development. Datasets and benchmarks are crucial for the development of machine learning methods, but also require their own publishing and reviewing guidelines. For instance, datasets can often not be reviewed in a double-blind fashion, and hence full anonymization will not be required. On the other hand, they do require additional specific checks, such as a proper description of how the data was collected, whether they show intrinsic bias, and whether they will remain accessible.
Schedule
-
|
Programming Puzzles
(
Poster
)
>
link
SlidesLive Video |
Tal Schuster 路 Ashwin Kalyan 路 Alex Polozov 路 Adam Kalai 馃敆 |
-
|
FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information
(
Poster
)
>
SlidesLive Video |
Rami Aly 路 Zhijiang Guo 路 Michael Schlichtkrull 路 James Thorne 路 Andreas Vlachos 路 Christos Christodoulopoulos 路 Oana Cocarascu 路 Arpit Mittal 馃敆 |
-
|
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling
(
Poster
)
>
SlidesLive Video |
Zhaojiang Lin 路 Andrea Madotto 路 Genta Winata 路 Peng Xu 路 Feijun Jiang 路 Yuxiang Hu 路 Chen Shi 路 Pascale N Fung 馃敆 |
-
|
Towards a robust experimental framework and benchmark for lifelong language learning
(
Poster
)
>
link
SlidesLive Video |
Aman Hussain 路 Nithin Holla 路 Pushkar Mishra 路 Helen Yannakoudakis 路 Ekaterina Shutova 馃敆 |
-
|
The People鈥檚 Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage
(
Poster
)
>
SlidesLive Video |
Daniel Galvez 路 Greg Diamos 路 Juan Torres 路 Juan Cer贸n 路 Keith Achorn 路 Anjali Gopi 路 David Kanter 路 Max Lam 路 Mark Mazumder 路 Vijay Janapa Reddi 馃敆 |
-
|
CrowdSpeech and Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription
(
Poster
)
>
SlidesLive Video |
Nikita Pavlichenko 路 Ivan Stelmakh 路 Dmitry Ustalov 馃敆 |
-
|
ReaSCAN: Compositional Reasoning in Language Grounding
(
Poster
)
>
SlidesLive Video |
Zhengxuan Wu 路 Elisa Kreiss 路 Desmond Ong 路 Christopher Potts 馃敆 |
-
|
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
(
Poster
)
>
SlidesLive Video |
22 presentersShuai Lu 路 Daya Guo 路 Shuo Ren 路 Junjie Huang 路 Alexey Svyatkovskiy 路 Ambrosio Blanco 路 Colin Clement 路 Dawn Drain 路 Daxin Jiang 路 Duyu Tang 路 Ge Li 路 Lidong Zhou 路 Linjun Shou 路 Long Zhou 路 Michele Tufano 路 MING GONG 路 Ming Zhou 路 Nan Duan 路 Neel Sundaresan 路 Shao Kun Deng 路 Shengyu Fu 路 Shujie LIU |
-
|
Variance-Aware Machine Translation Test Sets
(
Poster
)
>
SlidesLive Video |
Runzhe Zhan 路 Xuebo Liu 路 Derek Wong 路 Lidia Chao 馃敆 |
-
|
Timers and Such: A Practical Benchmark for Spoken Language Understanding with Numbers
(
Poster
)
>
|
Loren Lugosch 路 Piyush Papreja 路 Mirco Ravanelli 路 Abdelwahab HEBA 路 Titouan Parcollet 馃敆 |
-
|
LiRo: Benchmark and leaderboard for Romanian language tasks
(
Poster
)
>
SlidesLive Video |
20 presentersStefan Dumitrescu 路 Petru Rebeja 路 Beata Lorincz 路 Mihaela Gaman 路 Andrei Avram 路 Mihai Ilie 路 Andrei Pruteanu 路 Adriana Stan 路 Lorena Rosia 路 Cristina Iacobescu 路 Luciana Morogan 路 George Dima 路 Gabriel Marchidan 路 Traian Rebedea 路 Madalina Chitez 路 Dani Yogatama 路 Sebastian Ruder 路 Radu Tudor Ionescu 路 Razvan Pascanu 路 Viorica Patraucean |
-
|
A Spoken Language Dataset of Descriptions for Speech-Based Grounded Language Learning
(
Poster
)
>
SlidesLive Video |
11 presentersGaoussou Kebe 路 Padraig Higgins 路 Patrick Jenkins 路 Kasra Darvish 路 Rishabh Sachdeva 路 Ryan Barron 路 John Winder 路 Donald Engel 路 Edward Raff 路 Francis Ferraro 路 Cynthia Matuszek |
-
|
NaturalProofs: Mathematical Theorem Proving in Natural Language
(
Poster
)
>
SlidesLive Video |
Sean Welleck 路 Jiacheng Liu 路 Ronan Le Bras 路 Hanna Hajishirzi 路 Yejin Choi 路 Kyunghyun Cho 馃敆 |
-
|
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
(
Poster
)
>
SlidesLive Video |
Dan Hendrycks 路 Collin Burns 路 Anya Chen 路 Spencer Ball 馃敆 |
-
|
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
(
Poster
)
>
SlidesLive Video |
17 presentersRuchir Puri 路 David Kung 路 Geert Janssen 路 Wei Zhang 路 Giacomo Domeniconi 路 Vladimir Zolotov 路 Julian T Dolby 路 Jie Chen 路 Mihir Choudhury 路 Lindsey Decker 路 Veronika Thost 路 Luca Buratti 路 Saurabh Pujar 路 Shyam Ramji 路 Ulrich Finkler 路 Susan Malaika 路 Frederick Reiss |
-
|
DUE: End-to-End Document Understanding Benchmark
(
Poster
)
>
link
SlidesLive Video |
艁ukasz Borchmann 路 Micha艂 Pietruszka 路 Tomasz Stanislawek 路 Dawid Jurkiewicz 路 Micha艂 Turski 路 Karolina Szyndler 路 Filip Grali艅ski 馃敆 |
-
|
COVID-19 Sounds: A Large-Scale Audio Dataset for Digital Respiratory Screening
(
Poster
)
>
SlidesLive Video |
12 presentersTong Xia 路 Dimitrios Spathis 路 Chlo{\"e} Brown 路 J Ch 路 Andreas Grammenos 路 Jing Han 路 Apinan Hasthanasombat 路 Erika Bondareva 路 Ting Dang 路 Andres Floto 路 Pietro Cicuta 路 Cecilia Mascolo |
-
|
WaveFake: A Data Set to Facilitate Audio Deepfake Detection
(
Poster
)
>
SlidesLive Video |
Joel Frank 路 Lea Sch枚nherr 馃敆 |
-
|
RP-Mod & RP-Crowd: Moderator- and Crowd-Annotated German News Comment Datasets
(
Poster
)
>
SlidesLive Video |
Dennis Assenmacher 路 Marco Niemann 路 Kilian M眉ller 路 Moritz Seiler 路 Dennis Riehle 路 Heike Trautmann 馃敆 |
-
|
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
(
Poster
)
>
SlidesLive Video |
Boxin Wang 路 Chejian Xu 路 Shuohang Wang 路 Zhe Gan 路 Yu Cheng 路 Jianfeng Gao 路 Ahmed Awadallah 路 Bo Li 馃敆 |
-
|
Multilingual Spoken Words Corpus
(
Poster
)
>
SlidesLive Video |
14 presentersMark Mazumder 路 Sharad Chitlangia 路 Colby Banbury 路 Yiping Kang 路 Juan Ciro 路 Keith Achorn 路 Daniel Galvez 路 Mark Sabini 路 Peter Mattson 路 David Kanter 路 Greg Diamos 路 Pete Warden 路 Josh Meyer 路 Vijay Janapa Reddi |
-
|
Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension
(
Poster
)
>
SlidesLive Video |
Shusheng Xu 路 Yichen Liu 路 Xiaoyu Yi 路 Siyuan Zhou 路 Huizi Li 路 Yi Wu 馃敆 |
-
|
Measuring Coding Challenge Competence With APPS
(
Poster
)
>
SlidesLive Video |
11 presentersDan Hendrycks 路 Steven Basart 路 Saurav Kadavath 路 Mantas Mazeika 路 Akul Arora 路 Ethan Guo 路 Collin Burns 路 Samir Puranik 路 Horace He 路 Dawn Song 路 Jacob Steinhardt |
-
|
NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
(
Poster
)
>
SlidesLive Video |
David Alfonso-Hermelo 路 Ahmad Rashid 路 Abbas Ghaddar 路 Philippe Langlais 路 Mehdi Rezagholizadeh 馃敆 |
-
|
CSFCube - A Test Collection of Computer Science Research Articles for Faceted Query by Example
(
Poster
)
>
link
SlidesLive Video |
Sheshera Mysore 路 Tim O'Gorman 路 Andrew McCallum 路 Hamed Zamani 馃敆 |
-
|
RAFT: A Real-World Few-Shot Text Classification Benchmark
(
Poster
)
>
SlidesLive Video |
12 presentersNeel Alex 路 Eli Lifland 路 Lewis Tunstall 路 Abhishek Thakur 路 Pegah Maham 路 C. Riedel 路 Emmie Hine 路 Carolyn Ashurst 路 Paul Sedille 路 Alexis Carlier 路 Michael Noetel 路 Andreas Stuhlm眉ller |
-
|
A Dataset for Answering Time-Sensitive Questions
(
Poster
)
>
SlidesLive Video |
Wenhu Chen 路 Xinyi Wang 路 William Yang Wang 馃敆 |
-
|
DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates
(
Poster
)
>
SlidesLive Video |
John Pougu茅-Biyong 路 Valentina Semenova 路 Alexandre Matton 路 Rachel Han 路 Aerin Kim 路 Renaud Lambiotte 路 Doyne Farmer 馃敆 |
-
|
Task Agnostic and Task Specific Self-Supervised Learning from Speech with LeBenchmark
(
Poster
)
>
SlidesLive Video |
18 presentersSol猫ne Evain 路 Ha Nguyen 路 Hang Le 路 Marcely Zanon Boito 路 Salima Mdhaffar 路 Sina Alisamir 路 Ziyi Tong 路 Natalia Tomashenko 路 Marco Dinarelli 路 Titouan Parcollet 路 Alexandre Allauzen 路 Yannick Est猫ve 路 Benjamin Lecouteux 路 Fran莽ois Portet 路 Solange Rossato 路 Fabien Ringeval 路 Didier Schwab 路 laurent besacier |
-
|
SynthBio: A Case Study in Faster Curation of Text Datasets
(
Poster
)
>
SlidesLive Video |
Ann Yuan 路 Daphne Ippolito 路 Vitaly Nikolaev 路 Chris Callison-Burch 路 Andy Coenen 路 Sebastian Gehrmann 馃敆 |
-
|
Benchmarking the Combinatorial Generalizability of Complex Query Answering on Knowledge Graphs
(
Poster
)
>
SlidesLive Video |
Zihao Wang 路 Hang Yin 路 Yangqiu Song 馃敆 |
-
|
KLUE: Korean Language Understanding Evaluation
(
Poster
)
>
SlidesLive Video |
31 presentersSungjoon Park 路 Jihyung Moon 路 Sungdong Kim 路 Won Ik Cho 路 Ji Yoon Han 路 Jangwon Park 路 Chisung Song 路 Junseong Kim 路 Youngsook Song 路 Taehwan Oh 路 Joohong Lee 路 Juhyun Oh 路 Sungwon Lyu 路 Younghoon Jeong 路 Inkwon Lee 路 Sangwoo Seo 路 Dongjun Lee 路 Hyunwoo Kim 路 Myeonghwa Lee 路 Seongbo Jang 路 Seungwon Do 路 Sunkyoung Kim 路 Kyungtae Lim 路 Jongwon Lee 路 Kyumin Park 路 Jamin Shin 路 Seonghyun Kim 路 Lucy Park 路 Alice Oh 路 Jung-Woo Ha 路 Kyunghyun Cho |
-
|
CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge
(
Poster
)
>
SlidesLive Video |
Yasumasa Onoe 路 Michael Zhang 路 Eunsol Choi 路 Greg Durrett 馃敆 |
-
|
Few-Shot Learning Evaluation in Natural Language Understanding
(
Poster
)
>
link
SlidesLive Video |
Subhabrata Mukherjee 路 Xiaodong Liu 路 Guoqing Zheng 路 Saghar Hosseini 路 Hao Cheng 路 Ge Yang 路 Christopher Meek 路 Ahmed Awadallah 路 Jianfeng Gao 馃敆 |
-
|
SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables
(
Poster
)
>
SlidesLive Video |
Nafise Moosavi 路 Andreas R眉ckl茅 路 Dan Roth 路 Iryna Gurevych 馃敆 |
-
|
HumBugDB: A Large-scale Acoustic Mosquito Dataset
(
Poster
)
>
SlidesLive Video |
16 presentersIvan Kiskin 路 Marianne Sinka 路 Adam Cobb 路 Waqas Rafique 路 Lawrence Wang 路 Davide Zilli 路 Benjamin Gutteridge 路 Rinita Dam 路 Theodoros Marinos 路 Yunpeng Li 路 Dickson Msaky 路 Emmanuel Kaindoa 路 Gerard Killeen 路 Eva Herreros-Moya 路 Kathy Willis 路 Stephen J Roberts |
-
|
KeSpeech: An Open Source Speech Dataset of Mandarin and Its Eight Subdialects
(
Poster
)
>
SlidesLive Video |
15 presentersZhiyuan Tang 路 Dong Wang 路 Yanguang Xu 路 Jianwei Sun 路 Xiaoning Lei 路 Shuaijiang Zhao 路 cheng wen 路 Xingjun Tan 路 Chuandong Xie 路 Shuran Zhou 路 Rui Yan 路 Chenjia Lv 路 Yang Han 路 Wei Zou 路 Xiangang Li |
-
|
Measuring Mathematical Problem Solving With the MATH Dataset
(
Poster
)
>
SlidesLive Video |
Dan Hendrycks 路 Collin Burns 路 Saurav Kadavath 路 Akul Arora 路 Steven Basart 路 Eric Tang 路 Dawn Song 路 Jacob Steinhardt 馃敆 |