Support-Constrained Hierarchical Offline Reinforcement Learning via Empirical Transition Graphs

Hierarchical offline reinforcement learning (RL) decomposes long-horizon decision making into high-level subgoal selection and low-level control, which is attractive for complex continuous-control problems [1]. However, when the high-level policy operates in a continuous latent space, it can generate subgoals outside the support of the offline dataset. This creates a distinct source of distributional shift at the high-level planning layer. Even if the low-level policy stays close to the behavior data, the high-level policy may request state transitions that were never observed, leading to unreliable and potentially unsafe deployment behavior. Existing hierarchical offline methods typically remain vulnerable to this problem because they do not explicitly constrain the set of allowable subgoals. Unlike hierarchical offline RL methods that allow the manager to propose arbitrary latent goals, our framework restricts high-level decisions to subgoals supported by transitions observed in the offline trajectories. A variational autoencoder first maps high-dimensional observations into a compact latent representation. K-means clustering is then applied to the latent codes to produce a discrete set of nodes. From the offline trajectories, we construct a sparse directed graph whose edges represent latent-node transitions exceeding a minimum empirical frequency. A tabular Implicit Q-Learning (IQL) manager is restricted to select subgoals exclusively from the graph-supported neighbors of the current node, while a behavior-cloning-regularized actor–critic executes continuous low-level actions. This construction restricts high-level planning to the empirical transition support represented by the graph. We evaluate the approach on commercial building Heating, Ventilation, and Air Conditioning (HVAC) control, a safety-sensitive continuous-control domain with long horizons and competing energy and comfort objectives. The resulting graph provides a data-driven feasibility prior, implicitly encoding thermal dynamics and operational constraints observed in the offline trajectories. Across full-year EnergyPlus simulations in three ASHRAE climate zones, our method achieves 9.16% energy savings and a 9.21% reduction in comfort violations relative to the rule-based controller (RBC), substantially outperforming the evaluated flat offline RL baselines and Hierarchical IQL (HIQL). An off-support analysis provides direct evidence for the proposed mechanism. Under this graph-based definition of empirical support, HIQL selects unsupported high-level transitions on 53.7% of timesteps, while our constrained manager selects none by construction. The method is further validated through a 21-workday interleaved deployment comparing controllers in an operational university office building interfaced with a production Building Management System. Relative to the existing rule-based controller, it achieves 10.1% energy savings and a 7.3% reduction in comfort violations, outperforming Conservative Q-Learning (CQL) [2] by 4.3 and 2.4 percentage points, respectively. Here, Δ Reward denotes the improvement in the combined energy–comfort-smoothness control objective relative to RBC. Method | Energy Savings↑(%) | Comfort Violations↓(%) | Δ Reward↑ RBC | 0.0 | 0.0 | 0.0 CQL | 5.8 | 4.9 | 41 Ours | 10.1 | 7.3 | 58 These results show that hierarchical decomposition alone is insufficient to resolve offline distributional shift: constraining high-level planning to empirically supported transitions provides an effective mechanism for improving both reliability and control performance. [1] S. Park, D. Ghosh, B. Eysenbach, and S. Levine. HIQL: Offline Goal-Conditioned RL with Latent States as Actions. NeurIPS, 2023. [2] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative Q-Learning for Offline Reinforcement Learning. NeurIPS, 2020.