Orbit: A Framework for Multi-Agent Security Evaluations

Multi-agent LLM systems are increasingly deployed for complex, long-horizon tasks or emerge as a natural consequence of agents interacting in the wild. Yet, they often have significant security vulnerabilities: the flexible protocols that enable task generalization also expose novel threats, from cascading prompt injection to inter-agent collusion. Progress in defending against these threats has been slowed by a lack of shared empirical infrastructure, which forces bespoke environment development for every new defense and makes standardized comparison impossible. Existing evaluations address isolated threat models or single-agent settings, but none jointly vary attack, defense, and architecture across realistic multi-agent environments. To address this gap, we introduce Orbit, a configurable evaluation framework for empirical multi-agent security research, built on UK AISI's Inspect. Orbit lets researchers configure communication topologies, memory, scheduling, and agent roles. It supports four threat types (indirect prompt injection, misuse, compromised agents, collusion) and four defense strategies (security prompting, guardian agents, monitors, dual-LLM patterns). The benchmark suite spans 5 scenario families covering browser use, computer use, agentic coding, customer service, and cooperative allocation. Using Orbit, we conduct controlled experiments varying defenses, attacks, topologies, and models. Our central finding is a gap in defense transferability across threats: per-action defenses (an LLM monitor and a guardian agent) that cut a compromised agent's attack success by 60 points on multi-issue coding give no measurable protection against colluding agents, and none of the defenses we tested generalized over all of the attacks we tested. We further use Orbit to demonstrate security--performance tradeoffs and interactions between architectural choices and defense effectiveness. To accelerate empirical multi-agent security work, we make Orbit available open-source at: https://anonymous.4open.science/r/orbit-D1A9.