The Impact of Quantization on Large Reasoning Model Reinforcement Learning

Strong reasoning capabilities can now be achieved by large-scale reinforcement learning (RL) without any supervised fine-tuning. Although post-training quantization (PTQ) and quantization-aware training (QAT) are well studied in the context of fine-tuning, how quantization interacts with RL in large reasoning models (LRMs) remains an open question. To answer this question, we conducted systematic experiments and discovered a significant gap in reasoning performance between post-RL quantized models and their quantization-aware RL optimized counterparts. Our study found that quantizing models during RL training negatively impacted the learning process, wheras using post training quantization and QLoRA led to greater performance.