Timezone: »

 
Imitation Learning from Observations under Transition Model Disparity
Tanmay Gangwani · Yuan Zhou · Jian Peng

@
in Deep Reinforcement Learning »
Event URL: https://openreview.net/forum?id=7sVWxL1s3Ii »

Learning to perform tasks by leveraging a dataset of expert observations, also known as imitation learning from observations (ILO), is an important paradigm for learning skills without access to the expert reward function or the expert actions. We consider ILO in the setting where the expert and the learner agents operate in different environments, with the source of the discrepancy being the transition dynamics model. Recent methods for scalable ILO utilize adversarial learning to match the state-transition distributions of the expert and the learner, an approach that becomes challenging when the dynamics are dissimilar. In this work, we propose an algorithm that trains an intermediary policy in the learner environment and uses it as a surrogate expert for the learner. The intermediary policy is learned such that the state transitions generated by it are close to the state transitions in the expert dataset. To derive a practical and scalable algorithm, we employ concepts from prior work on estimating the support of a probability distribution. Experiments using MuJoCo locomotion tasks highlight that our method compares favorably to the baselines for ILO with transition dynamics mismatch.

Author Information

Tanmay Gangwani (University of Illinois, Urbana-Champaign)

I am a Ph.D. student in Computer Science at the University of Illinois, Urbana Champaign, supervised by Jian Peng. I'm interested in machine learning, especially Reinforcement Learning. My research is mainly focused on designing algorithms which efficiently leverage expert demonstrations for RL (imitation learning), address the exploration challenge in complex environment, and use generative modeling methods for model-based RL. For details, please visit https://tgangwani.github.io

Yuan Zhou (UIUC)
Jian Peng (University of Illinois at Urbana-Champaign)

More from the Same Authors