Skip to yearly menu bar Skip to main content


Workshop Sat, Dec 12, 2026 • 8:00 AM – 5:00 PM AEDT Cockle Bay Room 2

Transitioning from Pre-training to Post-training

Rachit Bansal ⋅ Clara Mohri ⋅ Tian Qin ⋅ Harman Singh ⋅ Samy Jelassi ⋅ Sham Kakade
[ OpenReview] 

Abstract

Chat is not available.

Schedule

Time zone: Conference local time zone
Exploration-Preserving Policy Optimization
Hangzhan Jin ⋅ Mohammad Hamdaqa ⋅ Doina Precup
Reward Fusion is a Pairwise Contrast Problem: Why Ceiling Contact Can Harm Group Relative RL
Zhenyu Zhang ⋅ Dingyang Chen ⋅ Yuan Ling ⋅ Xinyang Shen ⋅ Li Dong
Asynchronous Is Nearly Free for Evolution Strategies on Long-Horizon Agentic Tasks
William Hoy ⋅ Jingxuan Fan ⋅ Nurcin Celik ⋅ Xu Pan
Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation
Sarthak Harne ⋅ Chinmay Karkar ⋅ Yash Pandya ⋅ Akshay Nambi
Context-Grounded Data Curation and Gate-First RL Reward Design for SQL Autocomplete
Yueqin Yin ⋅ Paul Dong ⋅ Anni Li ⋅ Yite Wang ⋅ Arun Ponnusamy ⋅ Jaeseong Lee ⋅ Chiu W So ⋅ Mingyuan Zhou ⋅ Yuxiong He ⋅ Khalid Bijon ⋅ Zhewei Yao
Train for Many, Update with One: Posterior Learning for Multi-Sample Reasoning
Sungjun Lim ⋅ Robert Adrian Minut ⋅ Kyungwoo Song ⋅ Thomas Möllenhoff
On-Policy Delta Distillation
Byeongho Heo ⋅ Jaehui Hwang ⋅ Sangdoo Yun ⋅ Dongyoon Han
Adaptive Decoding Methods for Token-Efficient Reinforcement Learning
Zhaonan Li ⋅ Zelei Cheng ⋅ Amritansh Mishra ⋅ Michael Lee ⋅ Mann Patel ⋅ Sambit Sahu ⋅ Ben Zhou ⋅ William Campbell
Training LLMs to Follow Reasoning Instructions\\with Minimal Edits
Alexandra Yakovleva ⋅ Luat Gia Khoi Nguyen ⋅ Juho Kannala ⋅ Harri Valpola ⋅ Alexander Ilin
Curriculum Learning for Safety Alignment
Sandeep Kumar ⋅ Virginia Smith ⋅ Chhavi Yadav
Less Abstention, More Bias? Demographic Bias Drift After Supervised Fine-Tuning
Seema Guruvadoo ⋅ Pavithra Nair ⋅ Gilad Gressel ⋅ Krishnashree Achuthan
PINT: Effective On-Policy Self-Distillation via Privileged Information Internalization
Zian Su ⋅ Yilan Chen ⋅ Manish Prajapat ⋅ Rui Song ⋅ Rein Houthooft ⋅ Vijay Lingam
Training with Delta Attention for Long-Context Adaptation
Harsh Singh ⋅ Jeffrey Willette ⋅ Krishna Puvvada
What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after RL Post-Training
Subham Rath ⋅ Rajat Dandekar ⋅ Sreedath Panat ⋅ Raj Dandekar
Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It
Jonathan Williams ⋅ Esin Tureci ⋅ Karthik Narasimhan
Stress-Testing RL-based Capability Elicitation with Randomized Underperformance
Nina Żukowska ⋅ Eduard Hübner ⋅ Alexander Panfilov ⋅ Joschka Braun
AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL
Xiaolong Jin ⋅ Dingmin Wang ⋅ Vijay Lingam ⋅ Varun Kumar
Entropy Regularization: A Free Correction to Cross-Entropy for Verifiable Tasks
Mihir Dhanakshirur ⋅ Adam Ousherovitch ⋅ Ambuj Tewari
Understanding Reasoning from Pretraining to Post-Training
Jingyan Shen ⋅ Ang Li ⋅ Salman Rahman ⋅ Yifan Sun ⋅ Micah Goldblum ⋅ Matus Telgarsky ⋅ Pavel Izmailov
Scaling Near-Optimal SFT–RL Annotation Budget Allocation from Small to Large LLMs
Jingtan Wang ⋅ Arun Verma ⋅ Xiaoqiang Lin ⋅ Zhengyuan Liu ⋅ Nancy F Chen ⋅ Daniela Rus ⋅ Bryan Kian Hsiang Low
Learning to Think from Multiple Thinkers: How Heterogeneity in Chain of Thoughts Affects Efficient Learnability
Nirmit Joshi ⋅ Roey Magen ⋅ Nati Srebro ⋅ Nikolaos Tsilivis ⋅ Gal Vardi
From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
Yuqiao Tan ⋅ Minzheng Wang ⋅ Bo Liu ⋅ Zichen Liu ⋅ Shizhu He ⋅ Jun Zhao ⋅ Kang Liu
Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learning
Anthony GX-Chen ⋅ Ankit Anand ⋅ Gheorghe Comanici ⋅ Zaheer Abbas ⋅ Eser Aygün ⋅ David Smalling ⋅ Shibl Mourad ⋅ Doina Precup ⋅ Andre Barreto ⋅ Mark Rowland
SVSR: A Self-Verification and Self-Rectification Paradigm for Multimodal Reasoning
Zhe Qian ⋅ Nianbing Su ⋅ Zhonghua Wang ⋅ Zhongxing Xu ⋅ Hebei Li ⋅ Yueying Li ⋅ Fei Luo ⋅ Zhuohan Ouyang ⋅ Yanbiao Ma
Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads
Shachi Deshpande ⋅ Aidan Jay ⋅ Chenghui Zhou ⋅ Dayquan Julienne ⋅ Yuan Gao ⋅ Tianwei Chen ⋅ George Zerveas ⋅ ishmam zabir ⋅ Xiren Zhou ⋅ Chris Quirk ⋅ XIA SONG
Many Biases, One Circuit: Post-Training Tunes a Pretrained Cue-Routing Mechanism
Prakhar Gupta ⋅ Terry J Zhang ⋅ Florent Draye ⋅ Bernhard Schölkopf ⋅ Zhijing Jin
Forecasting Downstream Performance of LLMs With Proxy Metrics
Arkil Patel ⋅ Siva Reddy ⋅ Marius Mosbach ⋅ Dzmitry Bahdanau
A critical look at self-distillation for reasoning
Jason K Adhinarta ⋅ Raina Wu ⋅ Ashutosh Sharma ⋅ Rameswar Panda ⋅ Polina Golland ⋅ Yoon Kim ⋅ Zhang-Wei Hong
The Representation-Rationalizability Tradeoff in Reward Learning
Jing Dong ⋅ Pascal Poupart ⋅ Yaoliang Yu
The Ephemeral Cost of Switching Optimizers Across LLM Training Stages
Xinyu Zhao ⋅ Davis Wertheimer ⋅ Ashutosh Sharma ⋅ Rameswar Panda ⋅ Paul Liang
Reinforcing Internal Computations for Language Reasoning
Yibin Wang ⋅ Kaili Zhao ⋅ Jianyu Wang ⋅ Xiangjun Fan ⋅ Chengzhi Mao
Rotate the Loop: Restricted Injections for Content Retention
Alexey Zemtsov ⋅ Alexander Derevyagin ⋅ Albina Klepach ⋅ Vadim Lomshakov ⋅ Andrei Polubarov ⋅ Nikita Lyubaykin ⋅ Vladislav Kurenkov ⋅ Alexander Nikulin
Probe Guided Curriculum Learning: Models Secretly Know Good Prompts from Bad
William Bankes ⋅ Yi Y Ng ⋅ William Gitta Lugoloobi ⋅ Thomas Foster ⋅ Sangwoong Yoon ⋅ Ilija Bogunovic
Follow the Failure: Tracing Safety Degradation Back to Training Data
Saket Reddy ⋅ Amro Abbas ⋅ Anshuman Suri ⋅ Ari Morcos ⋅ David Schwab ⋅ Pratyush Maini
HyperGraphPro: Progress-Aware Reinforcement Learning for Structure-Guided Hypergraph RAG
Jinyoung Park ⋅ Sanghyeok Lee ⋅ Omar Khan ⋅ Hyunwoo J. Kim ⋅ Joo-Kyung Kim
Synthetic Persona Pretraining: Alignment from Token Zero
Julian Minder ⋅ Viktor Moskvoretskii ⋅ Raghav Singhal ⋅ Difan Jiao ⋅ Andy Arditi ⋅ Shaobo Cui ⋅ Yiderigun Borjigin ⋅ Kartik Bali ⋅ Stefan Krsteski ⋅ Harsh Raj ⋅ Huu Nguyen ⋅ Jannik Brinkmann ⋅ Ashton Anderson ⋅ Roland Aydin ⋅ Robert West
SURGE: Sparse Update Routing via Gated Entropy for Efficient Active Distillation
Tanvir Bhathal ⋅ Allen Shen ⋅ Gabriel Hübner ⋅ Dylan Zhou ⋅ Karthik Lakshmanan
Token Utility for On-Policy Distillation via Local Trust-Region Optimization
Yilan Chen ⋅ Vijay Lingam ⋅ Narayanan Sadagopan
InfoSFT: Learn More and Forget Less with Information-Aware Token Weighting
Mahdi Sabbaghi ⋅ George J. Pappas ⋅ Adel Javanmard ⋅ Hamed Hassani
Training Large Language Models for Effective Small-Molecule Design with Synthetic Task Scaling
Frank Hu ⋅ Shriram Chennakesavalu ⋅ Zichen Wang ⋅ Patricia Suriana ⋅ Bodhi Vani ⋅ Kirill Shmilovich ⋅ Kangway Chuang ⋅ Colin Grambow
ProbePO: Sparse Probes, Redistributed Credit for Critic-Free LLM Reinforcement Learning
Diyi Hu ⋅ Ruizhong Qiu ⋅ Lin Qiu ⋅ Hanghang Tong ⋅ Hanqing Zeng
Diverse Alternative Reasoning Traces for Hard Math Problems
Lukas Aichberger ⋅ Adam Golinski ⋅ Arno Blaas ⋅ Arwen Bradley ⋅ Jacob Springer ⋅ Mehrdad Farajtabar ⋅ Michael Kirchhof ⋅ Sinead Williamson
RealNLR: Do VLMs Learn Nonlocal Visual Algorithms from Synthetic Fine-Tuning?
Raagav Bala ⋅ Ishaan Kothari ⋅ Eric Li ⋅ Rik Halder ⋅ Ajay Agarwal
On the Off-Policy Teacher in On-Policy Distillation
Langlin Huang ⋅ Hao Liu ⋅ Mononito Goswami ⋅ Xinyu Li ⋅ Prithwish Jana ⋅ Nikos Kanakaris ⋅ Patrick Blöbaum ⋅ Purak Jain
When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play
Junyi Sha ⋅ Renfei Tan ⋅ David Simchi-Levi
SFT or RL for Tool-Calling LLM Agents? A Controlled Study Across Data, Method, and Scale
Md Tahmid Rahman Laskar ⋅ Xue-Yong Fu ⋅ Shashi Bhushan TN
On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics
Julianna Piskorz ⋅ Antonin Berthon ⋅ Mihaela van der Schaar
Robust Policy Optimization to Prevent Catastrophic Forgetting
Mahdi Sabbaghi ⋅ George J. Pappas ⋅ Adel Javanmard ⋅ Hamed Hassani
Reward-Aligned Distillation: Distill Your Model towards Higher Reward
Yilan Chen ⋅ Zian Su ⋅ Manish Prajapat ⋅ Dingmin Wang ⋅ Rein Houthooft ⋅ Rui Song ⋅ Vijay Lingam
Training Language Models via Neural Cellular Automata
Dan Lee ⋅ Seungwook Han ⋅ Akarsh Kumar ⋅ Pulkit Agrawal
Curated Learning for Self Improving Agents
Nicholas Lee ⋅ Sehoon Kim ⋅ Luca Manolache ⋅ Lakshya A Agrawal ⋅ Matei A Zaharia ⋅ Kurt Keutzer ⋅ Michael Mahoney ⋅ Amir Gholami ⋅ Zack Yu
Extremely Sparse Supervision Incentivizes Reasoning Ability
Zhishuai Liu ⋅ Xingzi Xu ⋅ Saygin Seyfioglu ⋅ Pan Xu ⋅ Karim Bouyarmane
Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor’s Internal States
Yunho Choi ⋅ Jongwon Lim ⋅ woojin Ahn ⋅ Minjae Oh ⋅ Jeonghoon Shim ⋅ Yohan Jo
Diagnosing and Stabilizing RLVR Training through Entropy Dynamics
Satya Sai Prasanth Yeduru ⋅ Zhichen Ren ⋅ Guangwei Yu ⋅ Satya Krishna Gorti
Post-training Trade-offs Under Limited Compute Budgets
Shougan Gu ⋅ Michael Ogezi ⋅ Freda Shi
Gap-Driven Post-Training: How SFT, Distillation, and RL Close Different Gaps in Web UI Generation
Zhunxuan Wang ⋅ Wei Liu ⋅ Abhishek Tripathi ⋅ Murat Sensoy
Misalignment Emerges During Benign Iterative Self-Improvement
Karim Abdel Sadek ⋅ Federico Di Gennaro ⋅ Fanny Yang
When Learning Signals Persist: Hidden Curricula in Vision–Language Reinforcement Learning
Yujin Jo ⋅ Hyunjik Jo ⋅ Yireun Kim ⋅ Jinsik Lee ⋅ Taesup Kim
Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVR
Chandak Chakma ⋅ Syed Nazmus Sakib ⋅ Nafiul Haque ⋅ Shifat E. Arman
Permissive Data Is Not the Bottleneck: Front-Loading Reasoning Data Enables Strong Post-Training
Ali Elganzory ⋅ Harsh Raj ⋅ Marianna Nezhurina ⋅ Victor May ⋅ Van Khue Nguyen ⋅ David Salinas ⋅ Huu Nguyen ⋅ Jenia Jitsev
What Happens When the Harness Comes and Goes? A Controlled Study of On-Policy Harness Self-Distillation
Kaiyuan Liu ⋅ Zezhou Huang ⋅ Janvijay Singh ⋅ Asli Celikyilmaz
The RLVR Policy Shift Is Sparse and Predictable from the Base Model
Jose Luis Cantarero ⋅ Jezabel R Garcia ⋅ Marcelo Marín ⋅ Antonio Tiene ⋅ Román Orús
Learning from Negative Policy: Revisiting Rollouts in On-Policy Distillation
Jaehui Hwang ⋅ Byeongho Heo ⋅ Sangdoo Yun ⋅ Dongyoon Han
Post-Training Language Models for Gold-Medal Performance in Coding Competitions
Aleksander Ficek ⋅ Sean Narenthiran ⋅ Mehrzad Samadi ⋅ Somshubra Majumdar ⋅ Boris Ginsburg
Catching RLHF Failures before Training
Matteo Lippi ⋅ Santiago Tomas Aranguri Diaz
RLPE: Teaching LLMs to Prompt with Reinforcement Learning
Rahul Singhal ⋅ Karime Maamari ⋅ Pradyumna Tambwekar