August 24, 2020
Deep Learning Recommendation Model (DLRM) captures our representative model architectures developed for click-through-rate (CTR) prediction based on high-dimensional sparse categorical data. Collective communications can account for a significant fraction of time in synchronous training of DLRM at scale. In this work, we explore using fine-grain integer quantization to reduce the communication volume of alltoall and allreduce collectives. We emulate quantized alltoall and allreduce, the latter using ring or recursive-doubling and each with optional carried-forward error compensation. We benchmark accuracy loss of quantized alltoall and allreduce with a representative DLRM model and Kaggle 7D dataset. We show that alltoall forward and backward passes, and dense allreduce can be quantized to 4 bits without accuracy loss compared to full-precision training.
Written by
Jie (Amy) Yang
Jongsoo Park
Srinivas Sridharan
Ping Tak Peter Tang
Publisher
Knowledge Discovery and Data Mining (KDD) 2020
Research Topics
November 30, 2020
Koustuv Sinha, Christopher Pal, Nicolas Gontier, Siva Reddy
November 30, 2020
December 03, 2018
Gabriel Synnaeve, Daniel Gant, Jonas Gehring, Nicolas Carion, Nicolas Usunier, Vasil Khalidov, Vegard Mella, Zeming Lin
December 03, 2018
December 03, 2018
Gabriel Synnaeve, Zeming Lin, Jonas Gehring, Dan Gant, Vegard Mella, Vasil Khalidov, Nicolas Carion, Nicolas Usunier
December 03, 2018
April 24, 2017
Nicolas Usunier, Gabriel Synnaeve, Zeming Lin, Soumith Chintala
April 24, 2017
May 06, 2019
Kenneth Marino, Abhinav Gupta, Rob Fergus, Arthur Szlam
May 06, 2019
July 03, 2019
July 03, 2019
Foundational models
Our approach
Latest news
Foundational models