ATLAS: Adaptive Attackers Learn Latent Adversarial Level Sets with Active Learning

This work-in-progress paper investigates how techniques in active learning for level set estimation can be leverage to generate sets of attacks for learning-based system. First, we consider the case of attack MNIST image classification systems and show how this attack generation problem can be formulated into an active learning problem to be leveraged by existing solutions. However, while existing solutions can be leveraged to search our high dimensional input space and find some adversarial examples, we find that these techniques will leave large areas of the adversarial region unexplored - potentially unable to recover large amounts of true adversarial inputs. Therefore, we design an active learning method, ATLAS, that balances between adversarial region discovery and saturation to sample widely but effectively inside of those spaces to capture and learn as many adversarial inputs as possible. We present preliminary results using toy problems to show how our framework finds representative adversarial inputs more efficiently than its predecessors. These results suggest that using active learning mechanisms that incorporate this balance is better suited to solving adversarial security problems.