Synthesis of Shields for Safe Reinforcement Learning in Both Discrete and Continuous State and Action Spaces with Linear Dynamics

Reinforcement Learning agents are increasingly deployed in domains where unsafe actions can have serious real-world consequences. These domains impose rules or constraints that the agent must observe, of the forms “always/never do X”, “do X within k timesteps”, “repeatedly do X next”, etc. Currently, agents must learn those constraints through trial and error, with no guarantee of compliance and significant wasted training time, and the possible experience of unacceptably dangerous states if training in the real world. Shielding addresses this by intercepting agent actions and rejecting any that violate constraints or foreclose future safe choices, providing provable safety guarantees for actions accepted by the shield. Existing shield construction methods are largely limited to finite, discrete state and action spaces. We present a novel shield synthesis approach that handles infinite, continuous or discrete state and action spaces. The shield is constructed offline, is minimally interfering whenever synthesis reaches a greatest fixpoint, and is agnostic to the choice of RL algorithm.