As AI systems become increasingly capable and autonomous, evaluating their failure modes requires more than static benchmarks and manually authored test cases. In this talk, we explore two complementary perspectives on how models behave when they are given objectives and allowed to optimize against them.

First, we present our automated red-teaming (ART) pipeline which we use to find vulnerabilities in our systems. An attacker model operates in a controlled environment and attempts to induce a target model to perform a specified undesirable behavior, including techniques like jailbreaks and prompt injections. We will walk through how we generate and execute attacks, verify whether they actually succeeded, and turn successful attack traces into training data for improving models.

Second, we explore reward hacking through examples from our training runs. We will talk about the ways AI agents cheat to hit their goals, reveal insights into RL training and maybe make some agents more relatable.