Timezone: »

 
Poster
An operator view of policy gradient methods
Dibya Ghosh · Marlos C. Machado · Nicolas Le Roux

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #196
in Poster Session 2 »
We cast policy gradient methods as the repeated application of two operators: a policy improvement operator $\mathcal{I}$, which maps any policy $\pi$ to a better one $\mathcal{I}\pi$, and a projection operator $\mathcal{P}$, which finds the best approximation of $\mathcal{I}\pi$ in the set of realizable policies. We use this framework to introduce operator-based versions of well-known policy gradient methods such as REINFORCE and PPO, which leads to a better understanding of their original counterparts. We also use the understanding we develop of the role of $\mathcal{I}$ and $\mathcal{P}$ to propose a new global lower bound of the expected return. This new perspective allows us to further bridge the gap between policy-based and value-based methods, showing how REINFORCE and the Bellman optimality operator, for example, can be seen as two sides of the same coin.

Author Information

Dibya Ghosh (UC Berkeley)
Marlos C. Machado (Google Brain)
Nicolas Le Roux (Google AI)

More from the Same Authors