Learning You: A Thousand Possible Yous — Self-Evolving Personalized Agents through Synthetic Experience Reinforcement Learning

Foundation-model agents are rapidly evolving from short-lived assistants into persistent systems that plan, act, use tools, and interact with the same users over extended periods of time. For such agents, effective personalization requires more than remembering what a user has already said: the agent must infer preferences, constraints, and behavioral tendencies that may never have appeared in the limited interaction history available to it. This creates a fundamental bottleneck. Real-user interactions are sparse and costly, while passive memory and retrieval can only reuse experiences that have already occurred. We ask whether a personalized agent can instead actively construct the experiences from which it most needs to learn about its user. We introduce Personalized Synthetic Experience Reinforcement Learning (P-SERL), a closed-loop framework in which an agent maintains uncertainty over a user’s latent preference state and a distribution of plausible counterfactual user worlds. Rather than generating synthetic experiences uniformly, P-SERL selects interactions using a utility that jointly captures uncertainty, diversity, expected personalization regret, and coverage of the preference space. Selected scenarios are labeled through the corresponding user world and incorporated into the personalized policy update. A central idea is that limited observations may remain consistent with many plausible users. Instead of collapsing prematurely to a single user model, P-SERL identifies where these plausible user worlds disagree and generates targeted counterfactual experiences that are maximally informative. As new real interactions arrive, the belief over plausible user worlds is updated and the synthetic curriculum adapts accordingly, turning synthetic experience into a moving instrument for continual personalization rather than a fixed augmented dataset. We evaluate P-SERL in a controlled synthetic personalization benchmark spanning preference-conditioned decision making, personalized planning, and user-specific tool selection. Across the three tasks, P-SERL achieves 70.4% mean personalized decision accuracy, compared with 68.6% for the strongest baseline, random synthetic experience generation, 65.9% without counterfactual selection, 46.4% using real interactions alone, and 44.6% with retrieval-based personalization. The improvement over the strongest baseline is statistically significant under a paired sign-flip permutation test (p=0.0288). P-SERL also reaches 70% mean accuracy using 12 real-user interactions, compared with 16 for random synthetic generation and 24 for the counterfactual-selection ablation, corresponding to a 25% reduction in real-user interactions relative to the strongest baseline. These results suggest that the key challenge is not simply generating more synthetic data, but deciding what synthetic experience should be generated next. P-SERL moves personalization from passive inference toward personalized self-improvement, where an agent actively constructs the experiences required to reduce uncertainty about an individual user.