Lost-in-the-Middle Effects in LLMs Can Emerge from Joint Short-Range and Long-Range Retrieval Demands

The performance of Large Language Models (LLMs) often degrades when crucial information is in the middle of a long context, a “lost-in-the-middle” phenomenon that mirrors the primacy and recency effects in human memory. We propose that this behavior is not simply a flaw indicative of information loss but an adaptation to different information retrieval demands during pre-training: some tasks require uniform recall across the entire input (a long-range retrieval demand), while others prioritize the most recent information (a short-range retrieval demand). Consistent with this view, we show that this U-shaped performance curve emerges when LLMs (GPT-2 and Llama variants) are trained from random initialization on two simple human memory paradigms simulating long-term and short-term memory demands. Our analysis reveals that while the recency effect directly aligns with short-range retrieval demand in the training data, the primacy effect is induced by the uniform long-range retrieval demand, and is additionally influenced by the model's autoregressive properties and the formation of attention sinks. Our main findings from simple human memory paradigms also generalize to a masked sequence completion task on the WikiText dataset, which more closely resembles the next-token prediction objective and natural language data of LLM pre-training. Together, our findings reveal how information retrieval demands, model architecture, and structural attention dynamics during model training can jointly produce positional bias observed in LLMs.