MetaPI: Constructing Prompt Injection Benchmarks from Any Agent Benchmarks

Agents powered by large language models (LLMs) are vulnerable to indirect prompt injection (IPI), in which malicious instructions embedded in external content can influence agent behavior at execution time. Progress in defending against IPI has largely been assessed using existing benchmarks. However, current IPI benchmarks are often limited by (1) oversimplified tasks and environments, (2) static attack objectives and strategies, and (3) non-adaptive attack payloads. To address these limitations, we present MetaPI, an extensible toolkit that turns any existing non-security agent benchmark into an agentic IPI benchmark, with built-in support for dynamic attack objective selection and adaptive attack payload optimization. We apply MetaPI to diverse agentic settings and evaluate a broad range of defenses across multiple backbone LLMs. Our results suggest that widely used benchmarks such as AgentDojo can substantially overstate both robustness and utility, especially when defenses are deployed in more realistic, complex environments. This further underscores the significance of MetaPI, the first toolkit of its kind to provide scalable, realistic, dynamic, and adaptive benchmarks for evaluating IPI defenses.