Self-Evolving Agents Should Build Internal and External Models of the World

Developing self-evolving agents that learn continuously from streams of information represents an emerging frontier in artificial intelligence. This capability is critical for efficiently updating computationally expensive systems such as foundation models. When agents learn by exploring the environment and interacting with other agents to achieve multiple tasks over time, two sources of non-stationarity emerge: shifting task distributions and co-evolving peers. However, developing frameworks for continual multi-agent reinforcement learning (CMARL) and effective policy improvement remains challenging. In this paper, we argue that training self-evolving agents should incorporate two classes of model-based principles: internal models that allow an agent to compare policy checkpoints, monitor its learning capacity, and represent its current objective; and external models that learn environment dynamics and the behavior of other agents. We anticipate that discussion on these principles will facilitate the design of algorithms, architectures, and benchmarks that explicitly evaluate model-based components in CMARL.