Timezone: »

 
Poster
Towards Efficient Post-training Quantization of Pre-trained Language Models
Haoli Bai · Lu Hou · Lifeng Shang · Xin Jiang · Irwin King · Michael R Lyu

Thu Dec 01 09:00 AM -- 11:00 AM (PST) @ Hall J #139
in Poster Session 5 »
Network quantization has gained increasing attention with the rapid growth of large pre-trained language models~(PLMs). However, most existing quantization methods for PLMs follow quantization-aware training~(QAT) that requires end-to-end training with full access to the entire dataset. Therefore, they suffer from slow training, large memory overhead, and data accessibility issues. In this paper, we study post-training quantization~(PTQ) of PLMs, and propose module-wise quantization error minimization~(MREM), an efficient solution to mitigate these issues. By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module. In addition, we design a new model parallel training strategy such that each module can be trained locally on separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., $4\times$ on $4$ GPUs). Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
Keywords: post-training quantization · BERT · natural langauge processing · training efficiency

Author Information

Haoli Bai (Huawei Noah's Ark Lab)
Lu Hou (Huawei Technologies Co., Ltd)
Lifeng Shang (Huawei Noah's Ark Lab)
Xin Jiang (Noah’s Ark Lab, Huawei Technologies)
Irwin King (Chinese University of Hong Kong)
Michael R Lyu (CUHK)

More from the Same Authors