Timezone: »

 
Poster
ShiftAddNet: A Hardware-Inspired Deep Network
Haoran You · Xiaohan Chen · Yongan Zhang · Chaojian Li · Sicheng Li · Zihao Liu · Zhangyang Wang · Yingyan Lin

Mon Dec 07 09:00 PM -- 11:00 PM (PST) @ Poster Session 0 #142
in Poster Session 1 »

Multiplication (e.g., convolution) is arguably a cornerstone of modern deep neural networks (DNNs). However, intensive multiplications cause expensive resource costs that challenge DNNs' deployment on resource-constrained edge devices, driving several attempts for multiplication-less deep networks. This paper presented ShiftAddNet, whose main inspiration is drawn from a common practice in energy-efficient hardware implementation, that is, multiplication can be instead performed with additions and logical bit-shifts. We leverage this idea to explicitly parameterize deep networks in this way, yielding a new type of deep network that involves only bit-shift and additive weight layers. This hardware-inspired ShiftAddNet immediately leads to both energy-efficient inference and training, without compromising the expressive capacity compared to standard DNNs. The two complementary operation types (bit-shift and add) additionally enable finer-grained control of the model's learning capacity, leading to more flexible trade-off between accuracy and (training) efficiency, as well as improved robustness to quantization and pruning. We conduct extensive experiments and ablation studies, all backed up by our FPGA-based ShiftAddNet implementation and energy measurements. Compared to existing DNNs or other multiplication-less models, ShiftAddNet aggressively reduces over 80% hardware-quantified energy cost of DNNs training and inference, while offering comparable or better accuracies. Codes and pre-trained models are available at https://github.com/RICE-EIC/ShiftAddNet.

Author Information

Haoran You (Rice University)
Xiaohan Chen (University of Texas at Austin)
Yongan Zhang (Rice University)
Chaojian Li (Rice University)
Sicheng Li (Alibaba group)
Zihao Liu (Alibaba Group)
Zhangyang Wang (University of Texas at Austin)
Yingyan Lin (Rice University)

The assistant professor working on energy-efficient machine learning systems

More from the Same Authors