Timezone: »
It has long been recognized that multi-agent reinforcement learning (MARL) faces significant scalability issues due to the fact that the size of the state and action spaces are exponentially large in the number of agents. In this paper, we identify a rich class of networked MARL problems where the model exhibits a local dependence structure that allows it to be solved in a scalable manner. Specifically, we propose a Scalable Actor-Critic (SAC) method that can learn a near optimal localized policy for optimizing the average reward with complexity scaling with the state-action space size of local neighborhoods, as opposed to the entire network. Our result centers around identifying and exploiting an exponential decay property that ensures the effect of agents on each other decays exponentially fast in their graph distance.
Author Information
Guannan Qu (California Institute of Technology)
Yiheng Lin (California Institute of Technology)
Adam Wierman (California Institute of Technology)
Na Li (Harvard University)
More from the Same Authors
-
2021 Spotlight: Perturbation-based Regret Analysis of Predictive Control in Linear Time Varying Systems »
Yiheng Lin · Yang Hu · Guanya Shi · Haoyuan Sun · Guannan Qu · Adam Wierman -
2022 : Robustifying machine-learned algorithms for efficient grid operation »
Nicolas Christianson · Christopher Yeh · Tongxin Li · Mahdi Torabi Rad · Azarang Golmohammadi · Adam Wierman -
2022 : Stability Constrained Reinforcement Learning for Real-Time Voltage Control »
Jie Feng · Yuanyuan Shi · Guannan Qu · Steven Low · Anima Anandkumar · Adam Wierman -
2022 : SustainGym: A Benchmark Suite of Reinforcement Learning for Sustainability Applications »
Christopher Yeh · Victor Li · Rajeev Datta · Yisong Yue · Adam Wierman -
2022 Poster: On the Sample Complexity of Stabilizing LTI Systems on a Single Trajectory »
Yang Hu · Adam Wierman · Guannan Qu -
2022 Poster: Policy Optimization for Markov Games: Unified Framework and Faster Convergence »
Runyu Zhang · Qinghua Liu · Huan Wang · Caiming Xiong · Na Li · Yu Bai -
2022 Poster: Bounded-Regret MPC via Perturbation Analysis: Prediction Error, Constraints, and Nonlinearity »
Yiheng Lin · Yang Hu · Guannan Qu · Tongxin Li · Adam Wierman -
2022 Poster: On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games »
Runyu Zhang · Jincheng Mei · Bo Dai · Dale Schuurmans · Na Li -
2021 Poster: Multi-Agent Reinforcement Learning in Stochastic Networked Systems »
Yiheng Lin · Guannan Qu · Longbo Huang · Adam Wierman -
2021 Poster: Pareto-Optimal Learning-Augmented Algorithms for Online Conversion Problems »
Bo Sun · Russell Lee · Mohammad Hajiesmaili · Adam Wierman · Danny Tsang -
2021 Poster: Perturbation-based Regret Analysis of Predictive Control in Linear Time Varying Systems »
Yiheng Lin · Yang Hu · Guanya Shi · Haoyuan Sun · Guannan Qu · Adam Wierman -
2020 Poster: Online Optimization with Memory and Competitive Control »
Guanya Shi · Yiheng Lin · Soon-Jo Chung · Yisong Yue · Adam Wierman -
2020 Poster: Leveraging Predictions in Smoothed Online Convex Optimization via Gradient-based Algorithms »
Yingying Li · Na Li -
2020 Poster: The Power of Predictions in Online Control »
Chenkai Yu · Guanya Shi · Soon-Jo Chung · Yisong Yue · Adam Wierman -
2019 Poster: Beyond Online Balanced Descent: An Optimal Algorithm for Smoothed Online Optimization »
Gautam Goel · Yiheng Lin · Haoyuan Sun · Adam Wierman -
2019 Spotlight: Beyond Online Balanced Descent: An Optimal Algorithm for Smoothed Online Optimization »
Gautam Goel · Yiheng Lin · Haoyuan Sun · Adam Wierman -
2019 Poster: Online Optimal Control with Linear Dynamics and Predictions: Algorithms and Regret Analysis »
Yingying Li · Xin Chen · Na Li