Timezone: »

 
Poster
Curriculum-guided Hindsight Experience Replay
Meng Fang · Tianyi Zhou · Yali Du · Lei Han · Zhengyou Zhang

Thu Dec 12 10:45 AM -- 12:45 PM (PST) @ East Exhibition Hall B + C #225
in Reinforcement Learning and Planning -- Reinforcement Learning »

In off-policy deep reinforcement learning, it is usually hard to collect sufficient successful experiences with sparse rewards to learn from. Hindsight experience replay (HER) enables an agent to learn from failures by treating the achieved state of a failed experience as a pseudo goal. However, not all the failed experiences are equally useful to different learning stages, so it is not efficient to replay all of them or uniform samples of them. In this paper, we propose to 1) adaptively select the failed experiences for replay according to the proximity to the true goals and the curiosity of exploration over diverse pseudo goals, and 2) gradually change the proportion of the goal-proximity and the diversity-based curiosity in the selection criteria: we adopt a human-like learning strategy that enforces more curiosity in earlier stages and changes to larger goal-proximity later. This ''Goal-and-Curiosity-driven Curriculum Learning'' leads to ''Curriculum-guided HER (CHER)'', which adaptively and dynamically controls the exploration-exploitation trade-off during the learning process via hindsight experience selection. We show that CHER improves the state of the art in challenging robotics environments.

Keywords: Reinforcement Learning · Reinforcement Learning and Planning · Applications -> Robotics; Reinforcement Learning and Planning -> Decision and Control; Reinforcement Learning and Planning · E
[ Paper]  [ Poster] 

Author Information

Meng Fang (Tencent)
Tianyi Zhou (University of Washington, Seattle)
Tianyi Zhou

Tianyi Zhou (https://tianyizhou.github.io) is a tenure-track assistant professor of computer science at the University of Maryland, College Park. He received his Ph.D. from the school of computer science & engineering at the University of Washington, Seattle. His research interests are in machine learning, optimization, and natural language processing (NLP). His recent works study curriculum learning that can combine high-level human learning strategies with model training dynamics to create a hybrid intelligence. The applications include semi/self-supervised learning, robust learning, reinforcement learning, meta-learning, ensemble learning, etc. He published >80 papers and is a recipient of the Best Student Paper Award at ICDM 2013 and the 2020 IEEE Computer Society TCSC Most Influential Paper Award.

Yali Du (University College London)

I am currently a research fellow at UCL. I am interested in multi-agent reinforcement learning, adversarial machine learning and recommendation systems.

Lei Han (Tencent AI Lab)
Zhengyou Zhang (Tencent)

More from the Same Authors