Poster
Dancing to Music
Hsin-Ying Lee · Xiaodong Yang · Ming-Yu Liu · Ting-Chun Wang · Yu-Ding Lu · Ming-Hsuan Yang · Jan Kautz

Tue Dec 10th 10:45 AM -- 12:45 PM @ East Exhibition Hall B + C #122
in Deep Learning -- Generative Models »

Dancing to music is an instinctive move by humans. Learning to model the music-to-dance generation process is, however, a challenging problem. It requires significant efforts to measure the correlation between music and dance as one needs to simultaneously consider multiple aspects, such as style and beat of both music and dance. Additionally, dance is inherently multimodal and various following movements of a pose at any moment are equally likely. In this paper, we propose a synthesis-by-analysis learning framework to generate dance from music. In the top-down analysis phase, we decompose a dance into a series of basic dance units, through which the model learns how to move. In the bottom-up synthesis phase, the model learns how to compose a dance by combining multiple basic dancing movements seamlessly according to input music. Experimental qualitative and quantitative results demonstrate that the proposed method can synthesize realistic, diverse, style-consistent, and beat-matching dances from music.

Keywords: Deep Learning -> Generative Models · Applications -> Computer Vision; Deep Learning -> Adversarial Networks

Author Information

Hsin-Ying Lee (University of California, Merced)
Xiaodong Yang (QCraft)
Ming-Yu Liu (Nvidia Research)
Ting-Chun Wang (NVIDIA)
Yu-Ding Lu (UC Merced)
Ming-Hsuan Yang (Google / UC Merced)
Jan Kautz (NVIDIA)

More from the Same Authors