Exploring and Addressing Reward Confusion in Offline Preference Learning

Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. In this work, we study the reward confusion problem in offline RLHF where spurious correlations exist in data. We create a lightweight benchmark to study this problem and propose a method that can reduce reward confusion by leveraging model uncertainty and the transitivity of preferences with active learning.