Value-Filtered Expert Query for Efficient Language Agent Finetuning: an Empirical Study

We study how to allocate a limited expert-labeling budget when supervised fine-tuning interactive language-model agents on a fixed task pool. Once behavioral cloning (BC) plateaus, a value model trained only on outcome-labeled student rollouts queries bounded teacher continuations from the student's on-policy states with the lowest predicted success. On ALFWorld, value filtering at 5\% of the full on-policy query budget raises valid-unseen success from 82.3±0.7\% to 85.8±0.9\%, is 2.4 points above budget-matched random selection, and matches the full unfiltered pool and scaled BC with 40\% and 49\% fewer teacher-output tokens. Gains concentrate at small budgets on this out-of-distribution split, making value filtering a prioritization device for tight expert budgets.