Understanding the Interplay between Memorization and Learning in Large Language Models

We investigate foundational questions about the interplay between memorization and learning in large language models (LLMs). When an LLM generates a string, can we disentangle the roles of rote memorization and contextual learning? Rote memorization results in an LLM regurgitating specific details of a training string, while contextual learning leads to the generation of new strings that follow generalizable patterns of the underlying language. Related questions of interest include: can an LLM avoid memorization when optimally learning a language?, and if not, can we design training schemes to reduce memorization and improve learning? To address these questions, we propose contextual memorization, a new measure that precisely disentangles memorization from learning. Using this measure, we establish that memorization of some training strings is unavoidable when optimally learning a language. Finally, we propose memorization-aware training, a scheme that equalizes learning across all training strings and in doing so, reduces memorization and improves learning. Our conclusions are supported by extensive experiments on multiple LLMs using formal languages as a controlled testbed – enabling precise language specification, exact string sampling, and elimination of data contamination – and further validated on natural language datasets.