Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

Many real-world control problems, from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. As time gaps shrink, the standard $Q$-function collapses toward the value function $V$, motivating continuous-time formulations based on an instantaneous advantage rate $q$. Existing methods typically characterize $V$ and $q$ through joint continuous-time objectives. Rather than optimizing both quantities simultaneously, we separate their roles into simpler sequential updates and take a decoupled iterative approach. Specifically, $q$ is learned from diffusion generators on a fixed $V$, while $V$ is then updated through a Hamiltonian-based value flow that remains informative at infinitesimal time scales, where standard max/softmax backups degenerate. A single-critic reformulation then turns this continuous-time iteration into a practical off-policy actor-critic update. We establish rigorous convergence guarantees for the resulting method. Experiments show strong performance against both continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, where our method achieves 21\% profit over a single quarter$-$nearly doubling the second-best method.