Timezone: »

 
Poster
MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies
Xue Bin Peng · Michael Chang · Grace Zhang · Pieter Abbeel · Sergey Levine

Wed Dec 11 05:00 PM -- 07:00 PM (PST) @ East Exhibition Hall B + C #44
in Algorithms -- Multitask and Transfer Learning »

Humans are able to perform a myriad of sophisticated tasks by drawing upon skills acquired through prior experience. For autonomous agents to have this capability, they must be able to extract reusable skills from past experience that can be recombined in new ways for subsequent tasks. Furthermore, when controlling complex high-dimensional morphologies, such as humanoid bodies, tasks often require coordination of multiple skills simultaneously. Learning discrete primitives for every combination of skills quickly becomes prohibitive. Composable primitives that can be recombined to create a large variety of behaviors can be more suitable for modeling this combinatorial explosion. In this work, we propose multiplicative compositional policies (MCP), a method for learning reusable motor skills that can be composed to produce a range of complex behaviors. Our method factorizes an agent's skills into a collection of primitives, where multiple primitives can be activated simultaneously via multiplicative composition. This flexibility allows the primitives to be transferred and recombined to elicit new behaviors as necessary for novel tasks. We demonstrate that MCP is able to extract composable skills for highly complex simulated characters from pre-training tasks, such as motion imitation, and then reuse these skills to solve challenging continuous control tasks, such as dribbling a soccer ball to a goal, and picking up an object and transporting it to a target location.

Keywords: Multitask and Transfer Learning · Reinforcement Learning · Algorithms · Reinforcement Learning and Planning -> Hierarchical RL; Reinforcement Learning and Planning

Author Information

Xue Bin Peng (UC Berkeley)
Michael Chang (University of California, Berkeley)

Ph.D. student at Berkeley AI Research, U.C. Berkeley B.S. in Computer Science from MIT Former research intern under Juergen Schmidhuber, Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA) Former undergraduate researcher under Joshua Tenenbaum and Antonio Torralba, MIT

Grace Zhang (UC Berkeley)
Pieter Abbeel (UC Berkeley & covariant.ai)

Pieter Abbeel is Professor and Director of the Robot Learning Lab at UC Berkeley [2008- ], Co-Director of the Berkeley AI Research (BAIR) Lab, Co-Founder of covariant.ai [2017- ], Co-Founder of Gradescope [2014- ], Advisor to OpenAI, Founding Faculty Partner AI@TheHouse venture fund, Advisor to many AI/Robotics start-ups. He works in machine learning and robotics. In particular his research focuses on making robots learn from people (apprenticeship learning), how to make robots learn through their own trial and error (reinforcement learning), and how to speed up skill acquisition through learning-to-learn (meta-learning). His robots have learned advanced helicopter aerobatics, knot-tying, basic assembly, organizing laundry, locomotion, and vision-based robotic manipulation. He has won numerous awards, including best paper awards at ICML, NIPS and ICRA, early career awards from NSF, Darpa, ONR, AFOSR, Sloan, TR35, IEEE, and the Presidential Early Career Award for Scientists and Engineers (PECASE). Pieter's work is frequently featured in the popular press, including New York Times, BBC, Bloomberg, Wall Street Journal, Wired, Forbes, Tech Review, NPR.

Sergey Levine (UC Berkeley)

More from the Same Authors