Deeper Can Be Better: Join Level Governs Hierarchical LLM Memory

Language model memory systems often summarize a history in stages before seeing the question they will answer. Summarization order can determine which facts summaries retain. Suppose one chunk lists account restrictions and another names the account being used. Summarizing them together identifies the relevant restriction; summarizing the table first risks losing it. We call the number of summarization steps each piece of evidence undergoes before they meet their join level. At the same requested budget for each node, a deeper tree with seven compression nodes outperforms a shallower tree with three by 42.5 percentage points when evidence joins earlier. In a separate fixed tree experiment, moving one evidence chunk so that it joins later reduces accuracy from 1.00 to 0.05. Textual probes locate the loss: summaries retain account identifiers but omit their restrictions. The gap grows with the number of candidate pairs of accounts and restrictions. Controls that state the answer in one chunk show little join level dependence where the gap is largest. An information bound explains why more capacity is needed before related evidence meets and identifies relation structures that require large summaries under balanced grouping. A broader fan-in and depth study tests three dependency structures and selected LongMemEval and LoCoMo histories, showing how useful grouping choices depend on the task.