Timezone: »

 
Poster
Model-based Policy Optimization with Unsupervised Model Adaptation
Jian Shen · Han Zhao · Weinan Zhang · Yong Yu

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #547
in Poster Session 2 »

Model-based reinforcement learning methods learn a dynamics model with real data sampled from the environment and leverage it to generate simulated data to derive an agent. However, due to the potential distribution mismatch between simulated data and real data, this could lead to degraded performance. Despite much effort being devoted to reducing this distribution mismatch, existing methods fail to solve it explicitly. In this paper, we investigate how to bridge the gap between real and simulated data due to inaccurate model estimation for better policy optimization. To begin with, we first derive a lower bound of the expected return, which naturally inspires a bound maximization algorithm by aligning the simulated and real data distributions. To this end, we propose a novel model-based reinforcement learning framework AMPO, which introduces unsupervised model adaptation to minimize the integral probability metric (IPM) between feature distributions from real and simulated data. Instantiating our framework with Wasserstein-1 distance gives a practical model-based approach. Empirically, our approach achieves state-of-the-art performance in terms of sample efficiency on a range of continuous control benchmark tasks.

Author Information

Jian Shen (Shanghai Jiao Tong University)
Han Zhao (University of Illinois at Urbana-Champaign)
Weinan Zhang (Shanghai Jiao Tong University)
Yong Yu (Shanghai Jiao Tong Unviersity)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors