Timezone: »
Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., motion trajectories and perceptual data). No adequate communication channel exists between the human expert and the robot to describe critical aspects of the task, such as the properties of the target object or the intended shape of the motion. Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent (e.g., "go to the large green bowl"). The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity. We demonstrate in a set of simulation experiments how our approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compare the results to a variety of alternative methods.
Author Information
Simon Stepputtis (Arizona State University)
Joseph Campbell (Arizona State University)
Mariano Phielipp (Intel AI Labs)
Dr. Mariano Phielipp works at the Intel AI Lab inside the Intel Artificial Intelligence Products Group. His work includes research and development in deep learning, deep reinforcement learning, machine learning, and artificial intelligence. Since joining Intel, Dr. Phielipp has developed and worked on Computer Vision, Face Recognition, Face Detection, Object Categorization, Recommendation Systems, Online Learning, Automatic Rule Learning, Natural Language Processing, Knowledge Representation, Energy Based Algorithms, and other Machine Learning and AI-related efforts. Dr. Phielipp has also contributed to different disclosure committees, won an Intel division award related to Robotics, and has a large number of patents and pending patents. He has published on NeuriPS, ICML, ICLR, AAAI, IROS, IEEE, SPIE, IASTED, and EUROGRAPHICS-IEEE Conferences and Workshops.
Stefan Lee (Oregon State University)
Chitta Baral (Arizona State University)
Heni Ben Amor (Arizona State University)
Related Events (a corresponding poster, oral, or spotlight)
-
2020 Poster: Language-Conditioned Imitation Learning for Robot Manipulation Tasks »
Wed. Dec 9th 05:00 -- 07:00 AM Room Poster Session 2 #723
More from the Same Authors
-
2021 : The Reflective Explorer: Online Meta-Exploration from Offline Data in Realistic Robotic Tasks »
Rafael Rafailov · · Tianhe Yu · Avi Singh · Mariano Phielipp · Chelsea Finn -
2021 : The Reflective Explorer: Online Meta-Exploration from Offline Data in Realistic Robotic Tasks »
Rafael Rafailov · · Tianhe Yu · Avi Singh · Mariano Phielipp · Chelsea Finn -
2022 : Benchmarking Counterfactual Reasoning Abilities about Implicit Physical Properties »
Maitreya Patel · Tejas Gokhale · Chitta Baral · 'YZ' Yezhou Yang -
2022 : LILA: A Unified Benchmark for Mathematical Reasoning »
Swaroop Mishra · Matthew Finlayson · Pan Lu · Leonard Tang · Sean Welleck · Chitta Baral · Tanmay Rajpurohit · Oyvind Tafjord · Ashish Sabharwal · Peter Clark · Ashwin Kalyan -
2022 : Certifiably-correct Control Policies for Safe Learning and Adaptation in Assistive Robotics »
Keyvan Majd · GEOFFEY CLARK · Tanmay Khandait · · Sriram Sankaranarayanan · Georgios Fainekos · Heni Ben Amor -
2022 : Offline Policy Comparison with Confidence: Benchmarks and Baselines »
Anurag Koul · Mariano Phielipp · Alan Fern -
2022 : Group SELFIES: A Robust Fragment-Based Molecular String Representation »
Austin Cheng · Andy Cai · Santiago Miret · Gustavo Malkomes · Mariano Phielipp · Alan Aspuru-Guzik -
2022 : Conformer Search Using SE3-Transformers and Imitation Learning »
Luca Thiede · Santiago Miret · Krzysztof Sadowski · Haoping Xu · Mariano Phielipp · Alan Aspuru-Guzik -
2022 : Certifiably-correct Control Policies for Safe Learning and Adaptation in Assistive Robotics »
Keyvan Majd · GEOFFEY CLARK · Tanmay Khandait · · Sriram Sankaranarayanan · Georgios Fainekos · Heni Ben Amor -
2022 : Certifiably-correct Control Policies for Safe Learning and Adaptation in Assistive Robotics »
Keyvan Majd · GEOFFEY CLARK · Tanmay Khandait · Heni Ben Amor -
2021 : Neuroevolution-Enhanced Multi-Objective Optimization for Mixed-Precision Quantization »
Santiago Miret · Vui Seng Chua · Mattias Marder · Mariano Phielipp · Nilesh Jain · Somdeb Majumdar -
2021 Poster: SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation »
Abhinav Moudgil · Arjun Majumdar · Harsh Agrawal · Stefan Lee · Dhruv Batra -
2020 : VAIDA: An Educative Benchmark Creation Paradigm using Visual Analytics for Interactively Discouraging Artifacts (by Anjana Arunkumar, Swaroop Mishra, Bhavdeep Sachdeva, Chitta Baral and Chris Bryan) »
Anjana Arunkumar · Swaroop Mishra · Chitta Baral -
2020 Poster: Instance-based Generalization in Reinforcement Learning »
Martin Bertran · Natalia Martinez · Mariano Phielipp · Guillermo Sapiro -
2020 Poster: Dialog without Dialog Data: Learning Visual Dialog Agents from VQA Data »
Michael Cogswell · Jiasen Lu · Rishabh Jain · Stefan Lee · Devi Parikh · Dhruv Batra -
2019 Poster: Goal-conditioned Imitation Learning »
Yiming Ding · Carlos Florensa · Pieter Abbeel · Mariano Phielipp