AgenticOS: Co-designing Systems and ML Foundations of an OS Layer for Agentic AI

Today’s agentic stacks are characterized by framework proliferation without common foundations. Each orchestration harness independently reimplements cross-cutting services (state management, memory, context engineering, resource budgeting, tool orchestration, and safety enforcement) making agents non-portable, difficult to audit, and brittle under composition. Simultaneously, self-evolving agents—systems that learn, adapt, and update their own behavior from operational experience—are proliferating in research and production. This introduces an entirely new class of challenge: how do we enable useful, constrained self-evolution while maintaining safety, reproducibility, and resource bounds? Current approaches leave this largely to individual frameworks, with no shared abstraction or principled mechanism for governing what changes, when, and under what conditions. The security implications of self-modifying systems operating autonomously at scale are underexplored and constitute a pressing open problem. Just as pre-OS computing was rescued not by better programs but by a shared execution substrate, the resolution to agentic fragmentation is not a better framework but an OS layer. Designing this OS layer requires genuine co-design between the ML and systems communities. The ML community must ask how models should be trained, structured, and exposed as system components—not merely as API endpoints. The systems community must ask what abstractions, memory hierarchies, scheduling policies, and execution substrates are required to make agentic behavior reliable and governable. These are not separable questions: the right abstraction boundary depends on what models can learn to do, and what models need to learn depends on what the system layer can support.