Analyzing the Effect of Noise in LLM Fine-Tuning

Fine-tuning is the dominant paradigm for adapting pretrained large language models (LLMs) to downstream NLP tasks. In practice, fine-tuning datasets may contain various forms of noise that arise from annotation errors or automated data collection. Although prior work has concentrated on designing robust learning algorithms to mitigate performance degradation under noisy conditions, comparatively little is known about how different types of noise affect the internal learning dynamics of LLMs during fine-tuning. In this work, we systematically study the impact of noise on model behaviour across five pretrained models (GPT-2, Qwen2, Qwen2.5-Instruct, Llama-2, and Gemma3) and four diverse NLP tasks. We introduce controlled perturbations that correspond to three common real-world noise types: label noise, grammatical noise, and typographical noise. Beyond task-level performance, we analyse layer-wise representation changes and attention patterns in order to understand how noise propagates through the network. Our results show that the corruption of labels (i.e., label noise) generally causes the largest performance degradation, whereas grammatical noise and typographical noise occasionally yield mild regularisation benefits. We further find that the effects of noise are localised primarily to task-specific layers, while attention structures remain comparatively stable.