Rethinking GFlowNet Training: Negative Reinforce Dynamics

Generative Flow Networks (GFlowNets) learn stochastic policies that sample structured objects proportionally to a reward. Standard training treats samples as fixed data, without differentiating through the distribution that generated them. When the sampling distribution depends on the model, full differentiation introduces a score-function term that admits a natural two-player interpretation between a GFlowNet and its sampler. Reversing the sampler direction turns this interaction into an adversarial one, motivating a training dynamics that actively seeks unresolved balance errors. We introduce Negative Reinforce Dynamics (NRD), which retains the standard semi-gradient correction while applying this reversed score direction only to negative residuals. In a simple parametric model, NRD converges globally to the balanced solution and contracts its natural Lyapunov potential no slower than semi-gradient training, with strict acceleration in tractable regimes. Experiments show improved convergence to multimodal target distributions and exploration in peptide-generation tasks.