Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

In short videos and live livestreams, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation-based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the separation module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. A benchmark dataset is constructed and released to evaluate the proposed methods. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.