Timezone: »

 
Poster
Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating Point
Bita Darvish Rouhani · Daniel Lo · Ritchie Zhao · Ming Liu · Jeremy Fowers · Kalin Ovtcharov · Anna Vinogradsky · Sarah Massengill · Lita Yang · Ray Bittner · Alessandro Forin · Haishan Zhu · Taesik Na · Prerak Patel · Shuai Che · Lok Chand Koppaka · XIA SONG · Subhojit Som · Kaustav Das · Saurabh K T · Steve Reinhardt · Sitaram Lanka · Eric Chung · Doug Burger

Tue Dec 08 09:00 AM -- 11:00 AM (PST) @ Poster Session 1 #532
in Poster Session 2 »

In this paper, we explore the limits of Microsoft Floating Point (MSFP), a new class of datatypes developed for production cloud-scale inferencing on custom hardware. Through the co-evolution of hardware design and algorithms, MSFP16 incurs 3x lower cost compared to Bfloat16 and MSFP12 has 4x lower cost compared to INT8 while delivering a comparable or better accuracy. MSFP incurs negligible impact to accuracy (<1%), requires no changes to the model topology, and is integrated with a mature cloud production pipeline. MSFP supports various classes of deep learning models including CNNs, RNNs, and Transformers without modification. Finally, we characterize the accuracy and implementation of MSFP and demonstrate its efficacy on a number of production scenarios, including models that power major online scenarios such as web search, question-answering, and image classification.

Author Information

Bita Darvish Rouhani (Microsoft)
Daniel Lo (Microsoft)
Ritchie Zhao (Microsoft)
Ming Liu (Microsoft)
Jeremy Fowers (Microsoft)
Kalin Ovtcharov (Microsoft)
Anna Vinogradsky (Caltech)
Sarah Massengill (Microsoft)
Lita Yang (Microsoft)
Ray Bittner (Microsoft Research)
Alessandro Forin (Microsoft)
Haishan Zhu (Microsoft)
Taesik Na (Microsoft)
Prerak Patel (Microsoft)
Shuai Che (Microsoft)
Lok Chand Koppaka (Microsoft)
XIA SONG (Microsoft)
Subhojit Som (Microsoft)
Kaustav Das (Microsoft)
Saurabh K T (Microsoft Corporation)
Steve Reinhardt (Microsoft)
Sitaram Lanka (Microsoft)
Eric Chung (Microsoft)
Doug Burger (Microsoft Research)

More from the Same Authors