Timezone: »

 
Spotlight
When to Update Your Model: Constrained Model-based Reinforcement Learning
Tianying Ji · Yu Luo · Fuchun Sun · Mingxuan Jing · Fengxiang He · Wenbing Huang

Wed Dec 07 09:00 AM -- 11:00 AM (PST) @
in Lightning Talks 3B-1 »

Designing and analyzing model-based RL (MBRL) algorithms with guaranteed monotonic improvement has been challenging, mainly due to the interdependence between policy optimization and model learning. Existing discrepancy bounds generally ignore the impacts of model shifts, and their corresponding algorithms are prone to degrade performance by drastic model updating. In this work, we first propose a novel and general theoretical scheme for a non-decreasing performance guarantee of MBRL. Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. These discoveries encourage us to formulate a constrained lower-bound optimization problem to permit the monotonicity of MBRL. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns. Motivated by these analyses, we design a simple but effective algorithm CMLO (Constrained Model-shift Lower-bound Optimization), by introducing an event-triggered mechanism that flexibly determines when to update the model. Experiments show that CMLO surpasses other state-of-the-art methods and produces a boost when various policy optimization methods are employed.

Keywords: Model-based Reinforcement Learning · event-triggered mechanism · monotonic improvements

Author Information

Tianying Ji (Tsinghua University)
Yu Luo (Tsinghua University)
Fuchun Sun (Tsinghua University)
Mingxuan Jing (Tsinghua University)
Fengxiang He (JD.com Inc)

Fengxiang He received his BSc in statistics from University of Science and Technology of China, MPhil and PhD in computer science from the University of Sydney. He is currently an algorithm scientist at JD Explore Academy, JD.com Inc., leading its trustworthy AI team. His research interest is in the theory and practice of trustworthy AI, including deep learning theory, privacy-preserving ML, decentralized learning, and their applications. He has published in prominent journals and conferences, including TNNLS, TMM, TCSVT, ICML, NeurIPS, ICLR, CVPR, and ICCV. He is the area chair of AISTATS, BMVC, and ACML. He is the leading author of several standards on trustworthy AI.

Wenbing Huang (Tsinghua University)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors