Every Token Counts: Online Memory Worth for Resource-Constrained Lifelong Agents

Lifelong agents continually accumulate and reuse experience to improve after deployment, making them attractive for long-running applications where adaptation must occur without repeatedly updating the underlying model. Yet, such agents may operate under tight generation, interaction, and memory budgets, while existing lifelong-memory methods are largely developed under resource-abundant settings. We find that jointly restricting output length, interaction rounds, memory capacity, and memory injection causes existing methods to degrade sharply, with several sophisticated approaches providing little advantage over simple retrieval or even no-memory agents. We show that resource scarcity amplifies the cost of poor memory decisions: with fewer stored experiences, limited memory access, and fewer opportunities for reasoning and recovery, an irrelevant or unreliable memory can substantially impair downstream execution. This suggests that resource-constrained lifelong agents should explicitly learn which past experiences are worth retaining and reusing, rather than treating retrieval relevance as a sufficient proxy for memory value. We propose TokenWorth, a lightweight online memory framework that learns query-dependent memory worth directly from streaming execution feedback. TokenWorth uses an AGOP-adapted Recursive Feature Machine to continually reshape the query--memory value space, and applies the learned worth throughout memory retrieval, admission, replacement, and eviction under a bounded memory budget. Experiments across resource-constrained lifelong-agent settings demonstrate improved robustness over existing memory methods, highlighting that effective lifelong adaptation under scarcity depends not on remembering more, but on learning what is worth remembering.