Understanding Reasoning from Pretraining to Post-Training: Chess as a Controlled Testbed

Reinforcement learning (RL) has become central to improving language models on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: how does pretraining shape RL scaling, and what does RL actually do to the inherited policy? These questions are difficult to study in the standard language-model setting: web-scale pretraining corpora make it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across pretraining and RL are prohibitively expensive. To address this challenge, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline, with verifiable outcomes and a controllable pretraining corpus. We pretrain language models from 5M to 1B parameters on human games, fine-tune them on synthetic reasoning traces, and run RL on chess puzzles with verifiable rewards. Using this framework, we establish a scaling law connecting pretraining and RL: pretraining loss predicts post-RL performance at fixed RL compute, while pretraining token count predicts the rate of improvement with additional RL. Beyond scaling, we find that RL does not simply sharpen the SFT policy: its effect varies qualitatively with puzzle difficulty, primarily amplifying correct moves the SFT policy already preferred on easy puzzles, while on hard puzzles RL surfaces moves that were nearly absent under SFT for a substantial fraction of states. Together, these results provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying reasoning across the full training pipeline.