Timezone: »
Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2.0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results. However, directly applying widely adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is suboptimal in the computational cost needed. Moreover, we show that the discovered subnetworks yield minimal performance gain compared to the original dense network.We present Prune-Adjust-Re-Prune (PARP), which discovers and finetunes subnetworks for much better performance, while only requiring a single downstream ASR finetuning run. PARP is inspired by our surprising observation that subnetworks pruned for pre-training tasks need merely a slight adjustment to achieve a sizeable performance boost in downstream ASR tasks. Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational advantage and performance gain of PARP over baseline pruning methods.In particular, on the 10min Librispeech split without LM decoding, PARP discovers subnetworks from wav2vec 2.0 with an absolute 10.9%/12.6% WER decrease compared to the full model. We further demonstrate the effectiveness of PARP via: cross-lingual pruning without any phone recognition degradation, the discovery of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and its applicability to pre-trained BERT/XLNet for natural language tasks1.
Author Information
Cheng-I Jeff Lai (MIT)
Yang Zhang (MIT-IBM Watson AI Lab)
Alexander Liu (Massachusetts Institute of Technology)
Shiyu Chang (UC Santa Barbara)
Yi-Lun Liao (Massachusetts Institute of Technology)
Yung-Sung Chuang (Massachusetts Institute of Technology)
Kaizhi Qian (MIT-IBM Watson AI Lab)
Sameer Khurana (Massachusetts Institute of Technology)
David Cox (MIT-IBM Watson AI Lab, IBM Research)
Jim Glass (Massachusetts Institute of Technology)
Related Events (a corresponding poster, oral, or spotlight)
-
2021 Poster: PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition »
Tue. Dec 7th 04:30 -- 06:00 PM Room
More from the Same Authors
-
2021 : ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation »
Chuang Gan · Jeremy Schwartz · Seth Alter · Damian Mrowca · Martin Schrimpf · James Traer · Julian De Freitas · Jonas Kubilius · Abhishek Bhandwaldar · Nick Haber · Megumi Sano · Kuno Kim · Elias Wang · Michael Lingelbach · Aidan Curtis · Kevin Feigelis · Daniel Bear · Dan Gutfreund · David Cox · Antonio Torralba · James J DiCarlo · Josh Tenenbaum · Josh McDermott · Dan Yamins -
2022 : PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation »
Jingyu Zhang · Jim Glass · Tianxing He -
2023 Poster: DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learning »
Alexander Liu · Heng-Jui Chang · Michael Auli · Wei-Ning Hsu · Jim Glass -
2022 : PCFG-based Natural Language Interface Improves Generalization for Controlled Text Generation »
Jingyu Zhang · Jim Glass · Tianxing He -
2022 Poster: Fairness Reprogramming »
Guanhua Zhang · Yihua Zhang · Yang Zhang · Wenqi Fan · Qing Li · Sijia Liu · Shiyu Chang -
2022 Poster: Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing »
Yonggan Fu · Yang Zhang · Kaizhi Qian · Zhifan Ye · Zhongzhi Yu · Cheng-I Jeff Lai · Celine Lin -
2021 Poster: Understanding Interlocking Dynamics of Cooperative Rationalization »
Mo Yu · Yang Zhang · Shiyu Chang · Tommi Jaakkola -
2021 Poster: TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up »
Yifan Jiang · Shiyu Chang · Zhangyang Wang -
2021 Poster: Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks »
Yonggan Fu · Qixuan Yu · Yang Zhang · Shang Wu · Xu Ouyang · David Cox · Yingyan Lin -
2021 Poster: Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception »
Joel Dapello · Jenelle Feather · Hang Le · Tiago Marques · David Cox · Josh McDermott · James J DiCarlo · Sueyeon Chung -
2021 : ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation »
Chuang Gan · Jeremy Schwartz · Seth Alter · Damian Mrowca · Martin Schrimpf · James Traer · Julian De Freitas · Jonas Kubilius · Abhishek Bhandwaldar · Nick Haber · Megumi Sano · Kuno Kim · Elias Wang · Michael Lingelbach · Aidan Curtis · Kevin Feigelis · Daniel Bear · Dan Gutfreund · David Cox · Antonio Torralba · James J DiCarlo · Josh Tenenbaum · Josh McDermott · Dan Yamins -
2020 : Towards Semi-Supervised Semantics Understanding from Speech »
Cheng-I Jeff Lai -
2020 Poster: Training Stronger Baselines for Learning to Optimize »
Tianlong Chen · Weiyi Zhang · Zhou Jingyang · Shiyu Chang · Sijia Liu · Lisa Amini · Zhangyang Wang -
2020 Spotlight: Training Stronger Baselines for Learning to Optimize »
Tianlong Chen · Weiyi Zhang · Zhou Jingyang · Shiyu Chang · Sijia Liu · Lisa Amini · Zhangyang Wang -
2020 Poster: The Lottery Ticket Hypothesis for Pre-trained BERT Networks »
Tianlong Chen · Jonathan Frankle · Shiyu Chang · Sijia Liu · Yang Zhang · Zhangyang Wang · Michael Carbin -
2019 Poster: Tight Certificates of Adversarial Robustness for Randomly Smoothed Classifiers »
Guang-He Lee · Yang Yuan · Shiyu Chang · Tommi Jaakkola -
2019 Poster: A Game Theoretic Approach to Class-wise Selective Rationalization »
Shiyu Chang · Yang Zhang · Mo Yu · Tommi Jaakkola -
2018 Workshop: The second Conversational AI workshop – today's practice and tomorrow's potential »
Alborz Geramifard · Jason Williams · Larry Heck · Jim Glass · Milica Gasic · Dilek Hakkani-Tur · Steve Young · Lazaros Polymenakos · Y-Lan Boureau · Maxine Eskenazi -
2018 Poster: Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization »
Sijia Liu · Bhavya Kailkhura · Pin-Yu Chen · Paishun Ting · Shiyu Chang · Lisa Amini -
2018 Poster: Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces »
Yu-An Chung · Wei-Hung Weng · Schrasing Tong · Jim Glass -
2018 Spotlight: Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces »
Yu-An Chung · Wei-Hung Weng · Schrasing Tong · Jim Glass -
2017 : Learning Word Embeddings from Speech »
Jim Glass · Yu-An Chung -
2017 Workshop: Conversational AI - today's practice and tomorrow's potential »
Alborz Geramifard · Jason Williams · Larry Heck · Jim Glass · Antoine Bordes · Steve Young · Gerald Tesauro -
2017 Poster: Dilated Recurrent Neural Networks »
Shiyu Chang · Yang Zhang · Wei Han · Mo Yu · Xiaoxiao Guo · Wei Tan · Xiaodong Cui · Michael Witbrock · Mark Hasegawa-Johnson · Thomas Huang -
2017 Poster: Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems »
Yonatan Belinkov · Jim Glass -
2016 : David Cox : Predictive Coding for Unsupervised Feature Learning »
David Cox -
2016 Poster: Tensor Switching Networks »
Chuan-Yung Tsai · Andrew M Saxe · David Cox