Timezone: »

 
Poster
A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Bo Liu · Xidong Feng · Jie Ren · Luo Mai · Rui Zhu · Haifeng Zhang · Jun Wang · Yaodong Yang

Tue Nov 29 09:00 AM -- 11:00 AM (PST) @ Hall J #1034
in Poster Session 1 »
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually \textbf{biased}. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of $\mathcal{O}\big(K\alpha^{K}\hat{\sigma}_{\text{In}}|\tau|^{-0.5}\big)$ \emph{w.r.t.} inner-loop update step $K$, learning rate $\alpha$, estimate variance $\hat{\sigma}^{2}_{\text{In}}$ and sample size $|\tau|$, and 2) the multi-step Hessian estimation bias $\hat{\Delta}_{H}$ due to the use of autodiff, which has a polynomial impact $\mathcal{O}\big((K-1)(\hat{\Delta}_{H})^{K-1}\big)$ on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.
Keywords: Meta Reinforcement Learning · Gradient Bias
[ Paper]  [ Slides]  [ Poster]  [ OpenReview] 

Author Information

Bo Liu (Peking University)
Xidong Feng (University College London)
Jie Ren (University of Edinburgh, University of Edinburgh)
Luo Mai (University of Edinburgh, University of Edinburgh)
Rui Zhu (DeepMind)
Haifeng Zhang (Institute of automation, Chinese academy of science, Chinese Academy of Sciences)
Jun Wang (UCL)
Yaodong Yang (AIG)

More from the Same Authors