Skip to yearly menu bar Skip to main content

Workshop

6th Robot Learning Workshop: Pretraining, Fine-Tuning, and Generalization with Large Scale Models

Dhruv Shah · Paula Wulkop · Claas Voelcker · Georgia Chalvatzaki · Alex Bewley · Hamidreza Kasaei · Ransalu Senanayake · Julien PEREZ · Jonathan Tompson

Project Page

Abstract

The proposed workshop focuses on the intersection of machine learning (ML) and robotics, under this year’s focus topic: “Pretraining, Fine-Tuning, and Generalization with Large Scale Models.” Embodied AI and robotics pose unique challenges and opportunities for utilizing large pre-trained models. We seek to host a diverse set of views and approaches from across the robotics domain and dive deep into questions such as: What sources of data can be used for training large models in robotics? What role should pre-training play in robotics pipelines? How far can pre-trained models generalize when faced with novel tasks and environments? What is currently missing to the pre-training paradigm for embodied systems?

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles

6:15 AM

Opening Remarks Presentation

Video

6:20 AM

Keynote Masha Itkina Talk

Video

6:45 AM

Keynote Jesse Thomason Talk

Video

7:10 AM

Keynote Dhruv Batra and Arjun Majumdar Talk

Video

7:35 AM

Keynote Deepak Pathak Talk

Video

8:00 AM

Poster Session + Robot Demos I Poster Session

9:00 AM

Oral Spotlights Presentation

Video

9:40 AM

Panel: How much are physical robots still needed in current robot learning research? Panel

Video

10:10 AM

Lunch Break

11:30 AM

Keynote Suraj Nair Talk

Video

11:55 AM

Keynote Matt Barnes Talk

Video

12:20 PM

Keynote Keerthana Gopalakrishnan and Montserrat Gonzalez Arenas Talk

Video

12:45 PM

Poster Session + Robot demos II Poster Session

2:15 PM

Debate: Scaling models and data size is sufficient for deploying robots in the real world Debate

Video

3:15 PM

Closing remarks & Awards Presentation

Video

Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation Poster

Annie Xie · Lisa Lee · Ted Xiao · Chelsea Finn

Pre-Trained Binocular ViTs for Image-Goal Navigation Poster

Guillaume Bono · Leonid Antsfeld · Boris Chidlovskii · Philippe Weinzaepfel · Christian Wolf

Sample-Efficient Online Imitation Learning using Pretrained Behavioural Cloning Policies Poster

Joe Watson · Jan Peters

DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models Poster

Norman Di Palo · Edward Johns

MultiReAct: Multimodal Tools Augmented Reasoning-Acting Traces for Embodied Agent Planning Poster

Zhouliang Yu · Jie Fu · Yao Mu · Chenguang Wang · Lin Shao · Yaodong Yang

MultiReAct: Multimodal Tools Augmented Reasoning-Acting Traces for Embodied Agent Planning Spotlight

Zhouliang Yu · Jie Fu · Yao Mu · Chenguang Wang · Lin Shao · Yaodong Yang

Reinforcement-learning robotic sailboats: simulator and preliminary results Poster

Eduardo Vasconcellos · Ronald M. Sampaio · ANDRE PAULO ARAUJO · Esteban CLUA · philippe preux · Luiz Marcos Garcia Goncalves · Luis Martí

Learning to Act from Actionless Videos through Dense Correspondences Poster

Po-Chen Ko · Jiayuan Mao · Yilun Du · Shao-Hua Sun · Josh Tenenbaum

EvIL: Evolution Strategies for Generalisable Imitation Learning Poster

Silvia Sapora · Chris Lu · Gokul Swamy · Yee Whye Teh · Jakob Foerster

A Statistical Guarantee for Representation Transfer in Multitask Imitation Learning Poster

Bryan Chan · James Bergstra · Karime Pereida

CAJun: Continuous Adaptive Jumping using a Learned Centroidal Controller Poster

Yuxiang Yang · Guanya Shi · Xiangyun Meng · Wenhao Yu · Tingnan Zhang · Jie Tan · Byron Boots

Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks Poster

Murtaza Dalal · Tarun Chiruvolu · Devendra Singh Chaplot · Russ Salakhutdinov

TD-MPC2: Scalable, Robust World Models for Continuous Control Poster

Nicklas Hansen · Hao Su · Xiaolong Wang

Reasoning with Latent Diffusion in Offline Reinforcement Learning Poster

Siddarth Venkatraman · Shivesh Khaitan · Ravi Tej Akella · John Dolan · Jeff Schneider · Glen Berseth

Knolling bot 2.0: Enhancing Object Organization with Self-supervised Graspability Estimation Poster

Yuhang Hu · Zhizhuo Zhang · Hod Lipson

Open X-Embodiment: Robotic Learning Datasets and RT-X Models Poster

Quan Vuong · Ajinkya Jain · Alex Bewley · Alexander Irpan · Alexander Khazatsky · Anant Rai · Anikait Singh · Antonin Raffin · Ayzaan Wahid · Beomjoon Kim · Bernhard Schölkopf · brian ichter · Cewu Lu · Charles Xu · Chelsea Finn · Chenfeng Xu · Cheng Chi · Chenguang Huang · Chuer Pan · Chuyuan Fu · Coline Devin · Danny Driess · Deepak Pathak · Dhruv Shah · Dieter Büchler · Dmitry Kalashnikov · Dorsa Sadigh · Edward Johns · Federico Ceola · Fei Xia · Freek Stulp · Gaoyue Zhou · Gaurav Sukhatme · Gautam Salhotra · Ge Yan · Giulio Schiavi · Hao Su · Hao-Shu Fang · Haochen Shi · Heni Ben Amor · Henrik Christensen · Hiroki Furuta · Homer Walke · Hongjie Fang · Igor Mordatch · Ilija Radosavovic · Isabel Leal · Jacky Liang · Jaehyung Kim · Jan Schneider · Jasmine Hsu · Jeannette Bohg · Jiajun Wu · Jialin Wu · Jianlan Luo · Jiayuan Gu · Jie Tan · Jitendra Malik · Jonathan Tompson · Jonathan Yang · Joseph Lim · João Silvério · Junhyek Han · Kanishka Rao · Karl Pertsch · Karol Hausman · Keegan Go · Keerthana Gopalakrishnan · Ken Goldberg · Kevin Zhang · Keyvan Majd · Krishan Rana · Krishnan Srinivasan · Lawrence Yunliang Chen · Lerrel Pinto · Liam Tan · Lionel Ott · Lisa Lee · Masayoshi TOMIZUKA · Michael Ahn · Mingyu Ding · Mohan Kumar Srirama · Mohit Sharma · Moo J Kim · Nicklas Hansen · Nicolas Heess · Nikhil Joshi · Niko Suenderhauf · Norman Di Palo · Nur Muhammad Shafiullah · Oier Mees · Oliver Kroemer · Pannag Sanketi · Paul Wohlhart · Peng Xu · Pierre Sermanet · Priya Sundaresan · Rafael Rafailov · Ran Tian · Ria Doshi · Roberto Martín-Martín · Russell Mendonca · Rutav Shah · Ryan Hoque · Ryan Julian · Samuel Bustamante · Sean Kirmani · Sergey Levine · Sherry Q Moore · Shikhar Bahl · Shivin Dass · Shuran Song · Sichun Xu · Siddhant Haldar · Simeon Adebola · Simon Guist · Soroush Nasiriany · Stefan Schaal · Stefan Welker · Stephen Tian · Sudeep Dasari · Suneel Belkhale · Takayuki Osa · Tatsuya Harada · Tatsuya Matsushima · Ted Xiao · Tianhe Yu · Tianli Ding · Todor Davchev · Tony Zhao · Trevor Darrell · Vidhi Jain · Vincent Vanhoucke · Wei Zhan · Wenxuan Zhou · Wolfram Burgard · Xi Chen · Xiaolong Wang · Xinghao Zhu · Xuanlin Li · Yao Lu · Yevgen Chebotar · Yifan Zhou · Yifeng Zhu · Yonatan Bisk · Yoonyoung Cho · Youngwoon Lee · Yuchen Cui · Yueh-Hua Wu · Yujin Tang · Yuke Zhu · Yunzhu Li · Yusuke Iwasawa · Yutaka Matsuo · Zhuo Xu · Zichen Cui · Alexander Herzog · Abhishek Padalkar · Acorn Pooley · Anthony Brohan · Ben Burgess-Limerick · Christine Chan · Jeffrey Bingham · Jihoon Oh · Kendra Byrne · Kenneth Oslund · Kento Kawaharazuka · Maximilian Du · Mingtong Zhang · Naoaki Kanazawa · Travis Armstrong · Ying Xu · Yixuan Wang · Jan Peters

T3GDT: Three-Tier Tokens to Guide Decision Transformer for Offline Meta Reinforcement Learning Poster

Zhe Wang · Haozhu Wang · Yanjun Qi

Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models Poster

Ivan Kapelyukh · Yifei Ren · Ignacio Alzugaray · Edward Johns

Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models Spotlight

Ivan Kapelyukh · Yifei Ren · Ignacio Alzugaray · Edward Johns

Policy-Guided Diffusion Poster

Matthew T Jackson · Michael Matthews · Cong Lu · Jakob Foerster · Shimon Whiteson

$\texttt{PREMIER-TACO}$ is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Loss Poster

Ruijie Zheng · Yongyuan Liang · Xiyao Wang · Shuang Ma · Hal Daumé III · Huazhe Xu · John Langford · Praveen Palanisamy · Kalyan Basu · Furong Huang

IG-Net: Image-Goal Network for Offline Visual Navigation on A Large-Scale Game Map Poster

Pushi Zhang · Baiting Zhu · Xin-Qiang Cai · Li Zhao · Masashi Sugiyama · Jiang Bian

Robotic Task Generalization via Hindsight Trajectory Sketches Poster

Jiayuan Gu · Sean Kirmani · Paul Wohlhart · Yao Lu · Montserrat Gonzalez Arenas · Kanishka Rao · Wenhao Yu · Chuyuan Fu · Keerthana Gopalakrishnan · Zhuo Xu · Priya Sundaresan · Peng Xu · Hao Su · Karol Hausman · Chelsea Finn · Quan Vuong · Ted Xiao

Hybrid Inverse Reinforcement Learning Poster

Juntao Ren · Gokul Swamy · Steven Wu · J. Bagnell · Sanjiban Choudhury

RoboAgent: Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking Poster

Homanga Bharadhwaj · Jay Vakil · Mohit Sharma · Abhinav Gupta · Shubham Tulsiani · Vikash Kumar

Adapt On-the-Go: Behavior Modulation for Single-Life Robot Deployment Poster

Annie Chen · Govind Chada · Laura Smith · Archit Sharma · Zipeng Fu · Sergey Levine · Chelsea Finn

D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Robotic Manipulation Poster

Yixuan Wang · Zhuoran Li · Mingtong Zhang · Katherine Driggs-Campbell · Jiajun Wu · Fei-Fei Li · Yunzhu Li

How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies Poster

Montserrat Gonzalez Arenas · Ted Xiao · Sumeet Singh · Vidhi Jain · Allen Z. Ren · Quan Vuong · Jake Varley · Alexander Herzog · Isabel Leal · Sean Kirmani · Dorsa Sadigh · Vikas Sindhwani · Kanishka Rao · Jacky Liang · Andy Zeng

How to Prompt Your Robot: A PromptBook for Manipulation Skills with Code as Policies Spotlight

Montserrat Gonzalez Arenas · Ted Xiao · Sumeet Singh · Vidhi Jain · Allen Z. Ren · Quan Vuong · Jake Varley · Alexander Herzog · Isabel Leal · Sean Kirmani · Dorsa Sadigh · Vikas Sindhwani · Kanishka Rao · Jacky Liang · Andy Zeng

World Model Based Sim2Real Transfer for Visual Navigation Poster

Kiran Lekkala · Chen Liu · Laurent Itti

Robotic Offline RL from Internet Videos via Value-Function Pre-Training Poster

Chethan Bhateja · Derek Guo · Dibya Ghosh · Anikait Singh · Manan Tomar · Quan Vuong · Yevgen Chebotar · Sergey Levine · Aviral Kumar

RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot Poster

Hao-Shu Fang · Hongjie Fang · Zhenyu Tang · Jirong Liu · Chenxi Wang · Junbo Wang · Haoyi Zhu · Cewu Lu

RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot Spotlight

Hao-Shu Fang · Hongjie Fang · Zhenyu Tang · Jirong Liu · Chenxi Wang · Junbo Wang · Haoyi Zhu · Cewu Lu

Exploitation-Guided Exploration for Semantic Embodied Navigation Poster

Justin Wasserman · Girish Chowdhary · Abhinav Gupta · Unnat Jain

Exploitation-Guided Exploration for Semantic Embodied Navigation Spotlight

Justin Wasserman · Girish Chowdhary · Abhinav Gupta · Unnat Jain

A$^2$Nav: Action-Aware Zero-Shot Robot Navigation Using Vision-Language Ability of Foundation Models Poster

Peihao Chen · Xinyu Sun · Hongyan Zhi · Runhao Zeng · Thomas Li · Mingkui Tan · Chuang Gan

LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotion Poster

Firas Al-Hafez · Davide Tateo · Jan Peters

Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design Poster

Aoran Jiao · Tanmay Patel · Sanjmi Khurana · Anna-Mariya Korol · Lukas Brunke · Vivek Adajania · Utku Culha · SiQi Zhou · Angela Schoellig

Human Scene Transformer Poster

Tim Salzmann · Hao-Tien Lewis Chiang · Markus Ryll · Dorsa Sadigh · Carolina Parada · Alex Bewley

Low-Cost Exoskeletons for Learning Whole-Arm Manipulation in the Wild Poster

Hongjie Fang · Hao-Shu Fang · Yiming Wang · Jieji Ren · Jingjing Chen · Ruo Zhang · Weiming Wang · Cewu Lu

Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models Poster

Tsun-Hsuan Johnson Wang · Alaa Maalouf · Wei Xiao · Yutong Ban · Alexander Amini · Guy Rosman · Sertac Karaman · Daniela Rus

LLM Augmented Hierarchical Agents Poster

Bharat Prakash · Tim Oates · Tinoosh Mohsenin

Formalizing Lines of Research on Generalization in Deep Reinforcement Learning Poster

Ezgi Korkmaz

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning Poster

Jingyun Yang · Max Sobol Mark · Brandon Vu · Archit Sharma · Jeannette Bohg · Chelsea Finn

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning Spotlight

Jingyun Yang · Max Sobol Mark · Brandon Vu · Archit Sharma · Jeannette Bohg · Chelsea Finn

Vision-Language Models Provide Promptable Representations for Reinforcement Learning Poster

William Chen · Oier Mees · Aviral Kumar · Sergey Levine

Trajeglish: Learning the Language of Driving Scenarios Poster

Jonah Philion · Xue Bin Peng · Sanja Fidler

Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models Poster

Kevin Black · Mitsuhiko Nakamoto · Pranav Atreya · Homer Walke · Chelsea Finn · Aviral Kumar · Sergey Levine

Causal Influence Aware Counterfactual Data Augmentation Poster

Núria Armengol Urpí · Georg Martius

LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation Poster

Shengqiang Zhang · Philipp Wicke · Lütfi Kerem Senel · Luis Figueredo · Abdeldjallil Naceri · Sami Haddadin · Barbara Plank · Hinrich Schuetze