Skip to yearly menu bar Skip to main content


Poster

SVD-Softmax: Fast Softmax Approximation on Large Vocabulary Neural Networks

Kyuhong Shim · Minjae Lee · Iksoo Choi · Yoonho Boo · Wonyong Sung

Pacific Ballroom #99

Keywords: [ Recurrent Networks ] [ Efficient Inference Methods ]


Abstract:

We propose a fast approximation method of a softmax function with a very large vocabulary using singular value decomposition (SVD). SVD-softmax targets fast and accurate probability estimation of the topmost probable words during inference of neural network language models. The proposed method transforms the weight matrix used in the calculation of the output vector by using SVD. The approximate probability of each word can be estimated with only a small part of the weight matrix by using a few large singular values and the corresponding elements for most of the words. We applied the technique to language modeling and neural machine translation and present a guideline for good approximation. The algorithm requires only approximately 20\% of arithmetic operations for an 800K vocabulary case and shows more than a three-fold speedup on a GPU.

Live content is unavailable. Log in and register to view live content