Is Forgetting Always Bad? Revisiting Replay in Continual Offline RL

Offline reinforcement learning typically assumes a fixed dataset, but in practice new logged data may arrive over time. We study how the quality of replayed experience affects continual offline RL when incoming data improves or degrades. Using IQL on HalfCheetah, Hopper, and Walker2d, we compare no replay, uniform replay, high-return replay, and low-return replay under the same memory and optimization budgets. We find that replay quality interacts with data order: low-return replay performs better than high-return replay in degrading streams, while the opposite pattern appears in improving streams. Replay is also not always beneficial, with no replay achieving the highest mean final return in Hopper and Walker2d. These results show that high-return experience is not always the best to retain, and that forgetting can sometimes help adaptation.