When Clean Speech Hides Fluency: Evaluating Disfluency-Preserved and Delivery-Enriched Representations for LLM-Based L2 Speaking Assessment

Text-only LLMs are increasingly deployed to assess L2 spoken proficiency from transcripts, yet ASR pipelines can silently discard hesitations, repe- titions, repairs, and timing cues. We present a controlled, representation- level study isolating the effect of transcript design on LLM scoring quality. Using the Speak & Improve Corpus 2025 with 40 L2 English speakers, we evaluate Qwen2.5-14B under three matched conditions—clean tran- script, disfluency-preserved transcript, and disfluency plus explicit de- livery metadata—while holding the model, rubric, and speakers fixed. Explicit metadata reduces MAE by approximately 34% (1.556 to 1.029) and RMSE by approximately 30% (1.649 to 1.148) relative to the clean baseline; disfluency-preserved text improves rank agreement (Spearman +0.054) with little change in absolute error. These findings suggest that transcript representation is a meaningful and controllable source of error in text-only LLM assessment, with disfluency information primarily af- fecting ranking and explicit delivery metadata improving absolute score agreement.