Curriculum Learning for Safety Alignment

Direct Preference Optimization (DPO) is a widely used approach for safety alignment that aims to reduce harmful behaviors in large language models. However, prior work shows that it can be brittle and exhibits poor out-of-distribution (OOD) generalization \citep{qi2025safety}. In this paper we investigate whether Curriculum Learning can improve the robustness of DPO-based safety alignment. We propose \textbf{Staged-Competence}, a curriculum-based framework that organizes preference data by difficulty, employs competence-based sampling, and progressively updates the reference model during training. Averaged across three model families, Staged-Competence reduces OOD harmful response rates by 16\% and jailbreak attack success rates by 20\%, while preserving general capabilities and maintaining near-zero over-refusal. We further show that Staged-Competence: (1) matches baseline safety performance with only 75\% of the training data, demonstrating improved data efficiency and (2) yields better separation between safe and unsafe responses. Staged-Competence is agnostic to the underlying policy optimization loss and can extend to other DPO variants and alignment domains beyond safety. Our code and data can be found at: \url{link/post/acceptance.}