Timezone: »

 
Poster
Continuous MDP Homomorphisms and Homomorphic Policy Gradient
Sahand Rezaei-Shoshtari · Rosie Zhao · Prakash Panangaden · David Meger · Doina Precup

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #518
in Poster Session 1 »

Abstraction has been widely studied as a way to improve the efficiency and generalization of reinforcement learning algorithms. In this paper, we study abstraction in the continuous-control setting. We extend the definition of MDP homomorphisms to encompass continuous actions in continuous state spaces. We derive a policy gradient theorem on the abstract MDP, which allows us to leverage approximate symmetries of the environment for policy optimization. Based on this theorem, we propose an actor-critic algorithm that is able to learn the policy and the MDP homomorphism map simultaneously, using the lax bisimulation metric. We demonstrate the effectiveness of our method on benchmark tasks in the DeepMind Control Suite. Our method's ability to utilize MDP homomorphisms for representation learning leads to improved performance when learning from pixel observations.

Keywords: Reinforcement Learning · continuous control · policy optimization · representation learning in reinforcement learning · state abstraction

Author Information

Sahand Rezaei-Shoshtari (McGill University / Mila)

I am a PhD student at McGill University and Mila co-supervised by Prof. David Meger and Prof. Doina Precup. I'm interested in temporal and state abstraction in reinforcement learning, particularly in the context of robotics for learning skills across a wide range of tasks.

Rosie Zhao (McGill University)
Prakash Panangaden (McGill University, Montreal)
David Meger (McGill University)
Doina Precup (McGill University / Mila / DeepMind Montreal)

More from the Same Authors