Learning in Stackelberg Markov Games

This paper studies a general framework for learning Stackelberg equilibria in dynamic and uncertain environments, where a single leader interacts with a population of adaptive followers. Motivated by equitable electricity rate design for customers with distributed energy resources, we formalize a class of Stackelberg Markov games and establish the learning framework for stationary equilibrium. We extend the framework to incorporate a continuum of agents via mean-field (MF) approximation. We validate the framework on an energy market, where a utility company sets electricity rates for a large population of households. Our results show that learned policies can achieve economic efficiency, equity across income groups, and stability in energy systems, while also encouraging renewable adoption and reducing reliance on fossil-fuel generation to mitigate climate change.