Timezone: »
We present a deep imitation learning framework for robotic bimanual manipulation in a continuous state-action space. A core challenge is to generalize the manipulation skills to objects in different locations. We hypothesize that modeling the relational information in the environment can significantly improve generalization. To achieve this, we propose to (i) decompose the multi-modal dynamics into elemental movement primitives, (ii) parameterize each primitive using a recurrent graph neural network to capture interactions, and (iii) integrate a high-level planner that composes primitives sequentially and a low-level controller to combine primitive dynamics and inverse kinematics control. Our model is a deep, hierarchical, modular architecture. Compared to baselines, our model generalizes better and achieves higher success rates on several simulated bimanual robotic manipulation tasks. We open source the code for simulation, data, and models at: https://github.com/Rose-STL-Lab/HDR-IL.
Author Information
Fan Xie (Northeastern University)
Alexander Chowdhury (Northeastern University)
M. Clara De Paolis Kaluza (Northeastern University)
Linfeng Zhao (Northeastern University)
Lawson Wong (Northeastern University)
Rose Yu (University of California, San Diego)
More from the Same Authors
-
2020 : Paper 60: Traffic Forecasting using Vehicle-to-Vehicle Communication and Recurrent Neural Networks »
Rose Yu -
2022 : A Noether's theorem for gradient flow: Continuous symmetries of the architecture and conserved quantities of gradient flow »
Bo Zhao · Iordan Ganev · Robin Walters · Rose Yu · Nima Dehmamy -
2022 : Charting Flat Minima Using the Conserved Quantities of Gradient Flow »
Bo Zhao · Iordan Ganev · Robin Walters · Rose Yu · Nima Dehmamy -
2022 : Rethinking Neural Relational Inference for Granger Causal Discovery »
Stefanos Bennett · Rose Yu -
2022 : Scaling up and Stabilizing Differentiable Planning with Implicit Differentiation »
Linfeng Zhao · Huazhe Xu · Lawson Wong -
2022 : Rose Yu: "Physics-Guided Deep Learning for Climate Science" »
Rose Yu -
2022 : Keynote Talk 2 »
Rose Yu -
2022 : Panel Discussion I: Geometric and topological principles for representation learning in ML »
Irina Higgins · Taco Cohen · Erik Bekkers · Nina Miolane · Rose Yu -
2022 Poster: Robust Imitation of a Few Demonstrations with a Backwards Model »
Jung Yeon Park · Lawson Wong -
2022 Poster: Meta-Learning Dynamics Forecasting Using Task Inference »
Rui Wang · Robin Walters · Rose Yu -
2022 Poster: Symmetry Teleportation for Accelerated Optimization »
Bo Zhao · Nima Dehmamy · Robin Walters · Rose Yu -
2021 : Physics-Guided AI for Modeling Autonomous Vehicle Dynamics »
Rose Yu · Rose Yu -
2020 : Q/A and Discussion for ML Theory Session »
Karthik Kashinath · Mayur Mudigonda · Stephan Mandt · Rose Yu -
2020 : Rose Yu »
Rose Yu -
2020 : Rose Yu - Physics-Guided AI for Learning Spatiotemporal Dynamics »
Rose Yu -
2020 Workshop: Machine Learning for Engineering Modeling, Simulation and Design »
Alex Beatson · Priya Donti · Amira Abdel-Rahman · Stephan Hoyer · Rose Yu · J. Zico Kolter · Ryan Adams -
2020 : Invited Talk 11 Q&A by Rose »
Rose Yu -
2020 : Invited Talk 11: Tensor Methods for Efficient and Interpretable Spatiotemporal Learning »
Rose Yu -
2020 : Quantifying Uncertainty in Deep Spatiotemporal Forecasting for COVID-19 »
Yian Ma · Rose Yu -
2020 Poster: Learning Disentangled Representations of Videos with Missing Data »
Armand Comas · Chi Zhang · Zlatan Feric · Octavia Camps · Rose Yu -
2020 Session: Orals & Spotlights Track 06: Dynamical Sys/Density/Sparsity »
Animesh Garg · Rose Yu -
2018 : Long Range Sequence Generation via Multiresolution Adversarial Training »
Rose Yu -
2017 Poster: Subset Selection and Summarization in Sequential Data »
Ehsan Elhamifar · M. Clara De Paolis Kaluza