Timezone: »

 
Poster
Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation
Peide Huang · Mengdi Xu · Jiacheng Zhu · Laixi Shi · Fei Fang · DING ZHAO

Wed Nov 30 09:00 AM -- 11:00 AM (PST) @ Hall J #223
in Poster Session 3 »

Curriculum Reinforcement Learning (CRL) aims to create a sequence of tasks, starting from easy ones and gradually learning towards difficult tasks. In this work, we focus on the idea of framing CRL as interpolations between a source (auxiliary) and a target task distribution. Although existing studies have shown the great potential of this idea, it remains unclear how to formally quantify and generate the movement between task distributions. Inspired by the insights from gradual domain adaptation in semi-supervised learning, we create a natural curriculum by breaking down the potentially large task distributional shift in CRL into smaller shifts. We propose GRADIENT which formulates CRL as an optimal transport problem with a tailored distance metric between tasks. Specifically, we generate a sequence of task distributions as a geodesic interpolation between the source and target distributions, which are actually the Wasserstein barycenter. Different from many existing methods, our algorithm considers a task-dependent contextual distance metric and is capable of handling nonparametric distributions in both continuous and discrete context settings. In addition, we theoretically show that GRADIENT enables smooth transfer between subsequent stages in the curriculum under certain conditions. We conduct extensive experiments in locomotion and manipulation tasks and show that our proposed GRADIENT achieves higher performance than baselines in terms of learning efficiency and asymptotic performance.

Keywords: Reinforcement Learning · curriculum learning · Domain Adaptation
[ Paper]  [ Poster]  [ OpenReview] 

Author Information

Peide Huang (Carnegie Mellon University)
Mengdi Xu (Carnegie Mellon University)
Jiacheng Zhu (Carnegie Mellon University)
Laixi Shi (Carnegie Mellon University)

I'm actively looking for a research internship in Summer 2022. My research interests include signal processing, nonconvex optimization, high-dimensional statistical estimation, and reinforcement learning, ranging from theory to application.

Fei Fang (Carnegie Mellon University)
DING ZHAO (Carnegie Mellon University)

More from the Same Authors