Poster
A Theoretical Understanding of Gradient Bias in Meta-Reinforcement Learning
Bo Liu · Xidong Feng · Jie Ren · Luo Mai · Rui Zhu · Haifeng Zhang · Jun Wang · Yaodong Yang
Hall J (level 1) #1034
Keywords: [ Gradient Bias ] [ Meta Reinforcement Learning ]
Abstract:
Gradient-based Meta-RL (GMRL) refers to methods that maintain two-level optimisation procedures wherein the outer-loop meta-learner guides the inner-loop gradient-based reinforcement learner to achieve fast adaptations. In this paper, we develop a unified framework that describes variations of GMRL algorithms and points out that existing stochastic meta-gradient estimators adopted by GMRL are actually \textbf{biased}. Such meta-gradient bias comes from two sources: 1) the compositional bias incurred by the two-level problem structure, which has an upper bound of O(KαKˆσIn|τ|−0.5)O(KαK^σIn|τ|−0.5) \emph{w.r.t.} inner-loop update step KK, learning rate αα, estimate variance ˆσ2In^σ2In and sample size |τ||τ|, and 2) the multi-step Hessian estimation bias ˆΔH^ΔH due to the use of autodiff, which has a polynomial impact O((K−1)(ˆΔH)K−1)O((K−1)(^ΔH)K−1) on the meta-gradient bias. We study tabular MDPs empirically and offer quantitative evidence that testifies our theoretical findings on existing stochastic meta-gradient estimators. Furthermore, we conduct experiments on Iterated Prisoner's Dilemma and Atari games to show how other methods such as off-policy learning and low-bias estimator can help fix the gradient bias for GMRL algorithms in general.
Chat is not available.