In this work, we propose a new graph for video salient object ranking. This graph simultaneously explores multi-scale spatial contrasts (such as local contrast and global contrast) and intra-/inter-instance temporal correlations across frames to extract diverse spatio-temporal saliency cues. It has two advantages: 1. Unlike previous methods that only perform global inter-frame comparisons or compare all proposals from different frames, we explicitly model the motion trajectories of each instance by comparing its features with those in the same spatial region in adjacent frames, thus obtaining more accurate motion saliency cues. 2. We synchronize the spatio-temporal saliency cues in a single graph for joint optimization, which exhibits better dynamics compared to the previous stage-wise methods that prioritize spatial cues followed by temporal cues. Additionally, we propose a simple yet effective video retargeting method based on video saliency ranking. Extensive experiments demonstrate the superiority of our graph in video salient object ranking and the effectiveness of the video retargeting method.