Wasserstein Proximal Policy Gradient

Wasserstein policy optimization incorporates the geometry of continuous actions, but its finite-time theory remains limited. We study Wasserstein Proximal Policy Gradient (WPPG), which maximizes an entropy-regularized action-value objective subject to a squared Wasserstein proximal penalty. Because this penalty is not a Bregman divergence, standard three-point arguments do not apply. We develop a new contraction analysis based on Kantorovich duality, proximal optimality, a uniform $T_2$ inequality, and soft performance-difference identities. The result establishes global linear convergence and extends to inexact critic evaluation and approximate actor updates. A Lie--Trotter transport--diffusion splitting then motivates a practical implementation for implicit policies. Experiments on continuous control and inventory control demonstrate competitive performance.