SPUR: Scaling Reward Learning from Human Demonstrations

Learning reward functions from human demonstrations is critical for scalable robot learning, yet most approaches either require impractical ground‑truth state access, costly online retraining, or yield domain‑specific models with poor transferability. We propose SPUR, a unified reward modeling framework that combines a large pre‑trained vision‑language model (VLM) backbone fine‑tuned to encode robot image sequences and language instructions, a progress‑based reward objective trained on successful demonstrations augmented with video rewind to simulate failures, and a preference‑learning objective over mismatched and rewound trajectories to enable training on failed executions without explicit progress labels. This design leverages the generalization of VLMs while integrating complementary progress and preference signals for improved robustness and generalization. Experiments on out‑of‑distribution tasks in LIBERO and Meta‑World show that each component contributes to performance gains across a set of reward metrics, and their combination achieves state-of-the-art results compared to recent baselines, demonstrating scalable training of reward models.