Timezone: »
Human involvement in AI system design, development, and evaluation is critical to ensure that the insights being derived are practical, and the systems built are meaningful, reliable, and relatable to those who need them. Humans play an integral role in all stages of machine learning development, be it during data generation, interactively teaching machines, or interpreting, evaluating and debugging models. With growing interest in such “human in the loop” learning, we aim to highlight new and emerging research opportunities for the ML community that arise from the evolving needs to design evaluation and training strategies for humans and models in the loop. The specific focus of this workshop is on emerging and under-explored areas of human- and model-in-the-loop learning, such as employing humans to seek richer forms of feedback for data than labels alone, learning from dynamic adversarial data collection with humans employed to find weaknesses in models, learning from human teachers instructing computers through conversation and/or demonstration, investigating the role of humans in model interpretability, and assessing social impact of ML systems. This workshop aims to bring together interdisciplinary researchers from academia and industry to discuss major challenges, outline recent advances, and facilitate future research in these areas.
Sat 8:15 a.m. - 8:30 a.m.
|
Opening Remarks
|
Divyansh Kaushik · Bhargavi Paranjape · Douwe Kiela 🔗 |
Sat 8:30 a.m. - 8:55 a.m.
|
Invited Talk by Tom Mitchell (Carnegie Mellon)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 8:55 a.m. - 9:00 a.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 9:00 a.m. - 9:25 a.m.
|
Invited Talk by Jenn Wortman Vaughan (Microsoft Research)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 9:25 a.m. - 9:30 a.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 9:30 a.m. - 9:55 a.m.
|
Invited Talk by Sanjoy Dasgupta (UCSD)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 9:55 a.m. - 10:00 a.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 10:00 a.m. - 10:25 a.m.
|
Invited Talk by Finale Doshi-Velez (Harvard)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 10:25 a.m. - 11:30 a.m.
|
Q & A and Panel Session with Tom Mitchell, Jenn Wortman Vaughan, Sanjoy Dasgupta, and Finale Doshi-Velez
(
Discussion Panel
)
|
Tom Mitchell · Jennifer Wortman Vaughan · Sanjoy Dasgupta · Finale Doshi-Velez · Zachary Lipton 🔗 |
Sat 11:30 a.m. - 12:45 p.m.
|
Poster Session 1 ( GatherTown Posters ) link » | 🔗 |
Sat 12:45 p.m. - 1:55 p.m.
|
Break (Or meet on GatherTown) link » | 🔗 |
Sat 1:55 p.m. - 2:00 p.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 2:00 p.m. - 2:25 p.m.
|
Invited Talk by Dan Weld (University of Washington)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 2:25 p.m. - 2:30 p.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 2:30 p.m. - 2:55 p.m.
|
Invited Talk by Kristen Grauman (UT Austin, FAIR)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 2:55 p.m. - 3:00 p.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 3:00 p.m. - 3:25 p.m.
|
Invited Talk by Scott Yih (FAIR)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 3:25 p.m. - 3:30 p.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 3:30 p.m. - 3:55 p.m.
|
Invited Talk by Emma Brunskill (Stanford)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 3:55 p.m. - 4:00 p.m.
|
Break
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 4:00 p.m. - 4:25 p.m.
|
Invited Talk by Alex Ratner (Snorkel, University of Washington)
(
Talk
)
SlidesLive Video » |
Divyansh Kaushik 🔗 |
Sat 4:25 p.m. - 5:30 p.m.
|
Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner
(
Discussion Panel
)
|
Kristen Grauman · Wen-tau Yih · Alexander Ratner · Emma Brunskill · Douwe Kiela · Daniel S. Weld 🔗 |
Sat 5:30 p.m. - 6:45 p.m.
|
Poster Session 2 ( GatherTown Posters ) link » | 🔗 |
Sat 6:45 p.m. - 7:00 p.m.
|
Closing Remarks
(
Closing
)
|
🔗 |
Sat 7:00 p.m. - 8:00 p.m.
|
Social/Get-Together on GatherTown ( GatherTown ) link » | 🔗 |
Author Information
Divyansh Kaushik (Carnegie Mellon University)
Bhargavi Paranjape (University of Washington)
Forough Arabshahi (Facebook)
Yanai Elazar (Bar-Ilan University)
Yixin Nie (UNC-Chapel Hill)
Max Bartolo (UCL)
Polina Kirichenko (New York University)
Pontus Lars Erik Saito Stenetorp (University of Tokyo)
Mohit Bansal (UNC Chapel Hill)
Zachary Lipton (Carnegie Mellon University)
Douwe Kiela (Facebook AI Research)
More from the Same Authors
-
2021 : VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation »
Linjie Li · Jie Lei · Zhe Gan · Licheng Yu · Yen-Chun Chen · Rohit Pillai · Yu Cheng · Luowei Zhou · Xin Wang · William Yang Wang · Tamara L Berg · Mohit Bansal · Jingjing Liu · Lijuan Wang · Zicheng Liu -
2021 : Model-Free Learning for Continuous Timing as an Action »
Helen Zhou · David Childers · Zachary Lipton -
2022 : Downstream Datasets Make Surprisingly Good Pretraining Corpora »
Kundan Krishna · Saurabh Garg · Jeffrey Bigham · Zachary Lipton -
2022 : LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning »
Yi-Lin Sung · Jaemin Cho · Mohit Bansal -
2022 : Disentangling the Mechanisms Behind Implicit Regularization in SGD »
Zachary Novack · Simran Kaur · Tanya Marwah · Saurabh Garg · Zachary Lipton -
2022 : An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks »
Yuxiang Wu · Yu Zhao · Baotian Hu · Pasquale Minervini · Pontus Lars Erik Saito Stenetorp · Sebastian Riedel -
2022 : RLSBench: A Large-Scale Empirical Study of Domain Adaptation Under Relaxed Label Shift »
Saurabh Garg · Nick Erickson · James Sharpnack · Alexander Smola · Sivaraman Balakrishnan · Zachary Lipton -
2022 : Local Causal Discovery for Estimating Causal Effects »
Shantanu Gupta · David Childers · Zachary Lipton -
2022 : On the Maximum Hessian Eigenvalue and Generalization »
Simran Kaur · Jeremy M Cohen · Zachary Lipton -
2023 Poster: Visual Programming for Step-by-Step Text-to-Image Generation and Evaluation »
Jaemin Cho · Abhay Zala · Mohit Bansal -
2023 Poster: Understanding the detrimental class-level effects of data augmentation »
Polina Kirichenko · Mark Ibrahim · Randall Balestriero · Diane Bouchacourt · Shanmukha Ramakrishna Vedantam · Hamed Firooz · Andrew Wilson -
2023 Poster: Resolving Interference When Merging Models »
Prateek Yadav · Derek Tam · Leshem Choshen · Colin Raffel · Mohit Bansal -
2023 Poster: Deep Equilibrium Based Neural Operators for Steady-State PDEs »
Tanya Marwah · Ashwini Pokle · J. Zico Kolter · Zachary Lipton · Jianfeng Lu · Andrej Risteski -
2023 Poster: Complementary Benefits of Contrastive Learning and Self-Training Under Distribution Shift »
Saurabh Garg · Amrith Setlur · Zachary Lipton · Sivaraman Balakrishnan · Virginia Smith · Aditi Raghunathan -
2023 Poster: Improving Language Plasticity via Pretraining with Active Forgetting »
Yihong Chen · Mikel Artetxe · Kelly Marchisio · Roberta Raileanu · David Adelani · Pontus Lars Erik Saito Stenetorp · Sebastian Riedel -
2023 Poster: PanoGen: Text-Conditioned Panoramic Environment Generation for Vision-and-Language Navigation »
Jialu Li · Mohit Bansal -
2023 Poster: Self-Chained Image-Language Model for Video Localization and Question Answering »
Shoubin Yu · Jaemin Cho · Prateek Yadav · Mohit Bansal -
2023 Poster: Paxion: Patching Action Knowledge in Video-Language Foundation Models »
Zhenhailong Wang · Ansel Blume · Sha Li · Genglin Liu · Jaemin Cho · Zineng Tang · Mohit Bansal · Heng Ji -
2023 Poster: Online Label Shift: Optimal Dynamic Regret meets Practical Algorithms »
Dheeraj Baby · Saurabh Garg · Tzu-Ching Yen · Sivaraman Balakrishnan · Zachary Lipton · Yu-Xiang Wang -
2023 Poster: Can Language Models Teach? Teacher Explanations Improve Student Performance via Theory of Mind »
Swarnadeep Saha · Peter Hase · Mohit Bansal -
2023 Poster: Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models »
Peter Hase · Mohit Bansal · Been Kim · Asma Ghandeharioun -
2023 Poster: Adaptive Contextual Perception: How To Generalize To New Backgrounds and Ambiguous Objects »
Zhuofan Ying · Peter Hase · Mohit Bansal -
2023 Poster: Any-to-Any Generation via Composable Diffusion »
Zineng Tang · Ziyi Yang · Chenguang Zhu · Michael Zeng · Mohit Bansal -
2022 : Panel on Technical Challenges Associated with Reliable Human Evaluations of Generative Models »
Long Ouyang · Tongshuang Wu · Zachary Lipton -
2022 Workshop: Human Evaluation of Generative Models »
Divyansh Kaushik · Jennifer Hsia · Jessica Huynh · Yonadav Shavit · Samuel Bowman · Ting-Hao Huang · Douwe Kiela · Zachary Lipton · Eric Michael Smith -
2022 : Opening Remarks »
Divyansh Kaushik -
2022 Poster: Characterizing Datapoints via Second-Split Forgetting »
Pratyush Maini · Saurabh Garg · Zachary Lipton · J. Zico Kolter -
2022 Poster: Chroma-VAE: Mitigating Shortcut Learning with Generative Classifiers »
Wanqian Yang · Polina Kirichenko · Micah Goldblum · Andrew Wilson -
2022 Poster: TVLT: Textless Vision-Language Transformer »
Zineng Tang · Jaemin Cho · Yixin Nie · Mohit Bansal -
2022 Poster: Unsupervised Learning under Latent Label Shift »
Manley Roberts · Pranav Mani · Saurabh Garg · Zachary Lipton -
2022 Poster: Domain Adaptation under Open Set Label Shift »
Saurabh Garg · Sivaraman Balakrishnan · Zachary Lipton -
2022 Poster: ReFactor GNNs: Revisiting Factorisation-based Models from a Message-Passing Perspective »
Yihong Chen · Pushkar Mishra · Luca Franceschi · Pasquale Minervini · Pontus Lars Erik Saito Stenetorp · Sebastian Riedel -
2022 Poster: Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners »
Zhenhailong Wang · Manling Li · Ruochen Xu · Luowei Zhou · Jie Lei · Xudong Lin · Shuohang Wang · Ziyi Yang · Chenguang Zhu · Derek Hoiem · Shih-Fu Chang · Mohit Bansal · Heng Ji -
2022 Poster: LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning »
Yi-Lin Sung · Jaemin Cho · Mohit Bansal -
2022 Poster: On Feature Learning in the Presence of Spurious Correlations »
Pavel Izmailov · Polina Kirichenko · Nate Gruver · Andrew Wilson -
2022 Poster: Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning »
Haokun Liu · Derek Tam · Mohammed Muqeeth · Jay Mohta · Tenghao Huang · Mohit Bansal · Colin Raffel -
2022 Poster: VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives »
Zhuofan Ying · Peter Hase · Mohit Bansal -
2022 Poster: WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models »
Yonatan Bitton · Nitzan Bitton Guetta · Ron Yosef · Yuval Elovici · Mohit Bansal · Gabriel Stanovsky · Roy Schwartz -
2021 : Facebook - Data Centric Infrastructure »
Douwe Kiela -
2021 Poster: The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations »
Peter Hase · Harry Xie · Mohit Bansal -
2021 Poster: True Few-Shot Learning with Language Models »
Ethan Perez · Douwe Kiela · Kyunghyun Cho -
2021 : Invited talk - Douwe Kiela »
Douwe Kiela -
2021 Poster: Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking »
Zhiyi Ma · Kawin Ethayarajh · Tristan Thrush · Somya Jain · Ledell Wu · Robin Jia · Christopher Potts · Adina Williams · Douwe Kiela -
2021 Poster: Does Knowledge Distillation Really Work? »
Samuel Stanton · Pavel Izmailov · Polina Kirichenko · Alexander Alemi · Andrew Wilson -
2021 : Introduction Competion Day 1 »
Douwe Kiela -
2021 Poster: Human-Adversarial Visual Question Answering »
Sasha Sheng · Amanpreet Singh · Vedanuj Goswami · Jose Magana · Tristan Thrush · Wojciech Galuba · Devi Parikh · Douwe Kiela -
2021 Poster: VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer »
Zineng Tang · Jaemin Cho · Hao Tan · Mohit Bansal -
2021 Poster: Detecting Moments and Highlights in Videos via Natural Language Queries »
Jie Lei · Tamara L Berg · Mohit Bansal -
2020 : Q & A and Panel Session with Dan Weld, Kristen Grauman, Scott Yih, Emma Brunskill, and Alex Ratner »
Kristen Grauman · Wen-tau Yih · Alexander Ratner · Emma Brunskill · Douwe Kiela · Daniel S. Weld -
2020 : Invited Talk by Alex Ratner (Snorkel, University of Washington) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Emma Brunskill (Stanford) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Scott Yih (FAIR) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Kristen Grauman (UT Austin, FAIR) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Dan Weld (University of Washington) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Contributed Talk 1: Fairness Under Partial Compliance »
Jessica Dai · Zachary Lipton -
2020 : Q & A and Panel Session with Tom Mitchell, Jenn Wortman Vaughan, Sanjoy Dasgupta, and Finale Doshi-Velez »
Tom Mitchell · Jennifer Wortman Vaughan · Sanjoy Dasgupta · Finale Doshi-Velez · Zachary Lipton -
2020 : Invited Talk by Finale Doshi-Velez (Harvard) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Sanjoy Dasgupta (UCSD) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Jenn Wortman Vaughan (Microsoft Research) »
Divyansh Kaushik -
2020 : Break »
Divyansh Kaushik -
2020 : Invited Talk by Tom Mitchell (Carnegie Mellon) »
Divyansh Kaushik -
2020 : Opening Remarks »
Divyansh Kaushik · Bhargavi Paranjape · Douwe Kiela -
2020 : The Hateful Memes Challenge: Live award ceremony and winner presentations »
Douwe Kiela -
2020 : The Hateful Memes Challenge: Competition Overview »
Douwe Kiela -
2020 Poster: A Unified View of Label Shift Estimation »
Saurabh Garg · Yifan Wu · Sivaraman Balakrishnan · Zachary Lipton -
2020 Poster: The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes »
Douwe Kiela · Hamed Firooz · Aravind Mohan · Vedanuj Goswami · Amanpreet Singh · Pratik Ringshia · Davide Testuggine -
2020 Poster: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks »
Patrick Lewis · Ethan Perez · Aleksandra Piktus · Fabio Petroni · Vladimir Karpukhin · Naman Goyal · Heinrich Küttler · Mike Lewis · Wen-tau Yih · Tim Rocktäschel · Sebastian Riedel · Douwe Kiela -
2020 Poster: Why Normalizing Flows Fail to Detect Out-of-Distribution Data »
Polina Kirichenko · Pavel Izmailov · Andrew Wilson -
2020 Poster: Learning Optimal Representations with the Decodable Information Bottleneck »
Yann Dubois · Douwe Kiela · David Schwab · Ramakrishna Vedantam -
2020 Spotlight: Learning Optimal Representations with the Decodable Information Bottleneck »
Yann Dubois · Douwe Kiela · David Schwab · Ramakrishna Vedantam -
2019 : Audrey Durand, Douwe Kiela, Kamalika Chaudhuri moderated by Yann Dauphin »
Audrey Durand · Kamalika Chaudhuri · Yann Dauphin · Orhan Firat · Dilan Gorur · Douwe Kiela -
2019 : Douwe Kiela - Benchmarking Progress in AI: A New Benchmark for Natural Language Understanding »
Douwe Kiela -
2019 Workshop: Emergent Communication: Towards Natural Language »
Abhinav Gupta · Michael Noukhovitch · Cinjon Resnick · Natasha Jaques · Angelos Filos · Marie Ossenkopf · Angeliki Lazaridou · Jakob Foerster · Ryan Lowe · Douwe Kiela · Kyunghyun Cho -
2019 Poster: Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift »
Stephan Rabanser · Stephan Günnemann · Zachary Lipton -
2019 Poster: Learning Robust Global Representations by Penalizing Local Predictive Power »
Haohan Wang · Songwei Ge · Zachary Lipton · Eric Xing -
2019 Poster: Hyperbolic Graph Neural Networks »
Qi Liu · Maximilian Nickel · Douwe Kiela -
2019 Poster: Game Design for Eliciting Distinguishable Behavior »
Fan Yang · Liu Leqi · Yifan Wu · Zachary Lipton · Pradeep Ravikumar · Tom M Mitchell · William Cohen -
2018 : Invited Talk 1 »
Zachary Lipton -
2018 Workshop: Emergent Communication Workshop »
Jakob Foerster · Angeliki Lazaridou · Ryan Lowe · Igor Mordatch · Douwe Kiela · Kyunghyun Cho -
2018 : Panel Discussion »
Antonio Torralba · Douwe Kiela · Barbara Landau · Angeliki Lazaridou · Joyce Chai · Christopher Manning · Stevan Harnad · Roozbeh Mottaghi -
2018 : Panel on research process »
Zachary Lipton · Charles Sutton · Finale Doshi-Velez · Hanna Wallach · Suchi Saria · Rich Caruana · Thomas Rainforth -
2018 : Douwe Kiela - Learning Multimodal Embeddings »
Douwe Kiela -
2018 : Zachary Lipton »
Zachary Lipton -
2018 Poster: Does mitigating ML's impact disparity require treatment disparity? »
Zachary Lipton · Julian McAuley · Alexandra Chouldechova -
2017 Workshop: Emergent Communication Workshop »
Jakob Foerster · Igor Mordatch · Angeliki Lazaridou · Kyunghyun Cho · Douwe Kiela · Pieter Abbeel -
2017 Demonstration: Interactive-Length Multi-Task Video Captioning with Cooperative Feedback »
Han Guo · Ramakanth Pasunuru · Mohit Bansal -
2017 Poster: Poincaré Embeddings for Learning Hierarchical Representations »
Maximilian Nickel · Douwe Kiela -
2017 Spotlight: Poincaré Embeddings for Learning Hierarchical Representations »
Maximilian Nickel · Douwe Kiela