Timezone: »

 
Poster
Searching for Low-Bit Weights in Quantized Neural Networks
Zhaohui Yang · Yunhe Wang · Kai Han · Chunjing XU · Chao Xu · Dacheng Tao · Chang Xu

Wed Dec 09 09:00 AM -- 11:00 AM (PST) @ Poster Session 3 #844
in Poster Session 4 »

Quantized neural networks with low-bit weights and activations are attractive for developing AI accelerators. However, the quantization functions used in most conventional quantization methods are non-differentiable, which increases the optimization difficulty of quantized networks. Compared with full-precision parameters (\emph{i.e.}, 32-bit floating numbers), low-bit values are selected from a much smaller set. For example, there are only 16 possibilities in 4-bit space. Thus, we present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately. In particular, each weight is represented as a probability distribution over the discrete value set. The probabilities are optimized during training and the values with the highest probability are selected to establish the desired quantized network. Experimental results on benchmarks demonstrate that the proposed method is able to produce quantized neural networks with higher performance over the state-of-the-arts on both image classification and super-resolution tasks.

Keywords: Algorithms -> Density Estimation; Deep Learning -> Deep Autoencoders; Deep Learning -> Generative Models; Probabilistic Methods · Deep Learning

Author Information

Zhaohui Yang (peking university)
Yunhe Wang (Huawei Noah's Ark Lab)
Kai Han (Huawei Noah's Ark Lab)
Chunjing XU (Huawei Technologies)
Chao Xu (Peking University)
Dacheng Tao (University of Sydney)
Chang Xu (University of Sydney)

More from the Same Authors