Poster
Saliency-based Sequential Image Attention with Multiset Prediction
Sean Welleck · Jialin Mao · Kyunghyun Cho · Zheng Zhang

Mon Dec 4th 06:30 -- 10:30 PM @ Pacific Ballroom #129 #None
in Posters Mon »

Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.

Keywords: Recurrent Networks · Deep Learning · Attention Models · Computer Vision · Perception · Hierarchical RL

Author Information

Sean Welleck (NYU)
Jialin Mao (New York University)
Kyunghyun Cho (NYU)

Kyunghyun Cho is an associate professor of computer science and data science at New York University and a research scientist at Facebook AI Research. He was a postdoctoral fellow at the Université de Montréal until summer 2015 under the supervision of Prof. Yoshua Bengio, and received PhD and MSc degrees from Aalto University early 2014 under the supervision of Prof. Juha Karhunen, Dr. Tapani Raiko and Dr. Alexander Ilin. He tries his best to find a balance among machine learning, natural language processing, and life, but almost always fails to do so.

Zheng Zhang (Shanghai New York Univeristy)

More from the Same Authors