Timezone: »

Hierarchical Skills for Efficient Exploration
Jonas Gehring · Gabriel Synnaeve · Andreas Krause · Nicolas Usunier

Fri Dec 10 08:30 AM -- 10:00 AM (PST) @ None #None

In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. However, prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. In previous work on continuous control, the sensitivity of methods to this trade-off has not been addressed explicitly, as locomotion provides a suitable prior for navigation tasks, which have been of foremost interest. In this work, we analyze this trade-off for low-level policy pre-training with a new benchmark suite of diverse, sparse-reward tasks for bipedal robots. We alleviate the need for prior knowledge by proposing a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner. For utilization on downstream tasks, we present a three-layered hierarchical learning algorithm to automatically trade off between general and specific skills as required by the respective task. In our experiments, we show that our approach performs this trade-off effectively and achieves better results than current state-of-the-art methods for end-to-end hierarchical reinforcement learning and unsupervised skill discovery.

Author Information

Jonas Gehring (Facebook AI Research)
Gabriel Synnaeve (Facebook)
Andreas Krause (ETH Zurich)
Nicolas Usunier (Facebook AI Research)

More from the Same Authors