Timezone: »

 
Poster
Curriculum Offline Imitating Learning
Minghuan Liu · Hanye Zhao · Zhengyu Yang · Jian Shen · Weinan Zhang · Li Zhao · Tie-Yan Liu

Thu Dec 09 08:30 AM -- 10:00 AM (PST) @ Virtual
in Poster Session 6 »

Offline reinforcement learning (RL) tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. Despite the potential to surpass the behavioral policies, RL-based methods are generally impractical due to the training instability and bootstrapping the extrapolation errors, which always require careful hyperparameter tuning via online evaluation. In contrast, offline imitation learning (IL) has no such issues since it learns the policy directly without estimating the value function by bootstrapping. However, IL is usually limited in the capability of the behavioral policy and tends to learn a mediocre behavior from the dataset collected by the mixture of policies. In this paper, we aim to take advantage of IL but mitigate such a drawback. Observing that behavior cloning is able to imitate neighboring policies with less data, we propose \textit{Curriculum Offline Imitation Learning (COIL)}, which utilizes an experience picking strategy to make the agent imitate from adaptive neighboring policies with a higher return, and improves the current policy along curriculum stages. On continuous control benchmarks, we compare COIL against both imitation-based methods and RL-based methods, showing that COIL not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.

Keywords: Reinforcement Learning and Planning

Author Information

Minghuan Liu (Shanghai Jiao Tong University)
Hanye Zhao (Shanghai Jiao Tong University, Tsinghua University)
Zhengyu Yang (Shanghai Jiao Tong University)
Jian Shen (Shanghai Jiao Tong University)
Weinan Zhang (Shanghai Jiao Tong University)
Li Zhao (Microsoft Research)
Tie-Yan Liu (Microsoft Research Asia)

More from the Same Authors