Sequence-to-Segment Networks for Segment Detection
Zijun Wei · Boyu Wang · Minh Hoai Nguyen · Jianming Zhang · Zhe Lin · Xiaohui Shen · Radomir Mech · Dimitris Samaras

Tue Dec 4th 05:00 -- 07:00 PM @ Room 210 #66
Detecting segments of interest from an input sequence is a challenging problem which often requires not only good knowledge of individual target segments, but also contextual understanding of the entire input sequence and the relationships between the target segments. To address this problem, we propose the Sequence-to-Segment Network (S$^2$N), a novel end-to-end sequential encoder-decoder architecture. S$^2$N first encodes the input into a sequence of hidden states that progressively capture both local and holistic information. It then employs a novel decoding architecture, called Segment Detection Unit (SDU), that integrates the decoder state and encoder hidden states to detect segments sequentially. During training, we formulate the assignment of predicted segments to ground truth as bipartite matching and use the Earth Mover's Distance to calculate the localization errors. We experiment with S$^2$N on temporal action proposal generation and video summarization and show that S$^2$N achieves state-of-the-art performance on both tasks.

Author Information

Zijun Wei (Stony Brook University)

I am currently a graduate student at Department of Computer Science in Stony Brook Univeristy. From fall 2014 I'm working in the Computer Vision Lab under the supervision of Prof. Dimitris Samaras, Prof. Minh Hoai and Prof. Gregory Zelinsky Prior to this, I received my master degree from the Robotics Institute, Carnege Mellon University in 2013 advised by Prof. Mel Siegel. I work on research problems in Computer Vision and Machine Learning. I am especially interested in plugging human visual perception experience into computer vision to either boost performance or enable human-like results. I am also interested in the other way around -- using computer vision algorithms to model human visual perception systems. I'm a recipient of the Renaissance Technologies Fellowship from 2014 to 2017. I worked as research intern at Adobe Research twice: 2017 spring and 2018 summer.

Boyu Wang (Stony Brook University)
Minh Hoai Nguyen (Stony Brook University)
Jianming Zhang (Adobe Research)
Zhe Lin (Adobe Research)
Xiaohui Shen (ByteDance AI Lab)
Radomir Mech (Adobe Systems Incorporated)
Dimitris Samaras (Stony Brook University)

More from the Same Authors