A Unified Architecture for Accelerating Distributed DNN ...

1 This paper is included in the Proceedings of the 14th USENIX Symposium on Operating Systems Design and ImplementationNovember 4 6, 2020978-1-939133-19-9 Open access to the Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation is sponsored by USENIXA Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU ClustersYimin Jiang, Tsinghua University and ByteDance; Yibo Zhu, ByteDance; Chang Lan, Google; Bairen Yi, ByteDance; Yong Cui, Tsinghua University; Chuanxiong Guo, Unified Architecture for Accelerating Distributed DNN Training inHeterogeneous GPU/CPU ClustersYimin Jiang , Yibo Zhu , Chang Lan , Bairen Yi , Yong Cui , Chuanxiong Guo Tsinghua University, ByteDance, GoogleAbstractData center clusters that run DNN training jobs are inher-ently heterogeneous.

2 They have GPUs and CPUs for computa-tion and network bandwidth for Distributed training. However,existing Distributed DNN training architectures, all-reduceand Parameter Server (PS), cannot fully utilize such heteroge-neous resources. In this paper, we present a new distributedDNN training Architecture called BytePS. BytePS can lever-age spare CPU and bandwidth resources in the cluster toaccelerate Distributed DNN training tasks running on provides a communication framework that is bothprovedoptimaland Unified existing all-reduce and PS become twospecial cases of BytePS.

3 To achieve the proved optimality inpractice, BytePS further splits the functionalities of a parame-ter optimizer. It introduces aSummation Serviceabstractionfor aggregating gradients, which is common for all the op-timizers. Summation Service can be accelerated by AVXinstructions and can be efficiently run on CPUs, while DNNmodel-related optimizer algorithms are run on GPUs for com-putation acceleration. BytePS can accelerate DNN trainingfor major frameworks including TensorFlow, PyTorch andMXNet. For representative DNN training jobs with up to 256 GPUs, BytePS outperforms the state-of-the-art open sourceall-reduce and PS by up to 84% and 245%, IntroductionIn recent years, research on Deep Neural Networks (DNNs)has experienced a renaissance.

4 DNNs have brought break-throughs to computer vision [32,43], speech recognition andsynthesis [33,69], natural language processing (NLP) [26],and many other areas. Training these DNN models usuallyrequires a huge amount of arithmetic computation , GPUs are preferred. To run many such tasksand achieve high resource utilization, large GPU clusters withthousands or more GPUs are introduced [29,35,52,71].Such GPU clusters have not only GPUs, but also CPUs andhigh speed networks. GPU machines typically also have high-end CPUs [2,11].

5 There may also be CPU-only machinesused for training data pre-processing and generation, ,in reinforcement learning. These GPU/CPU machines areconnected by high-speed Ethernet or Infiniband network tofacilitate Distributed training. Based on our experience inoperating production GPU clusters ( ) and recent literaturefrom others [35], GPUs are usually better utilized while thereare often spare CPU and bandwidth are two major families of Distributed training archi-tectures, all-reduce [54] and Parameter Server (PS) [44].

6 Theyare both based on data parallelism ( 2). In a task that usesall-reduce, only GPU machines are involved. In an iteration,GPUs compute the gradients of the model parameters inde-pendently, and then aggregate gradients using the all-reduceprimitive. In PS tasks, both GPU machines and CPU machinescan be used. Different from all-reduce, the gradients are sentto PS, which typically runs on CPU machines and aggregatesthe received gradients. PS then runs certain DNN trainingoptimizer, ,SGD [76] or Adam [42] and sends back theupdated model.

7 For both all-reduce and PS, the above happensin every iteration, until the training and PS are quite different, in both theory andpractice. Given a set of GPU machineswithoutadditionalCPU machines, all-reduce is proved to be bandwidth opti-mal [54]. However,withadditional CPU and bandwidth re-sources, the optimality of all-reduce no longer holds wefind that,in theory, PS can offer even better performance byutilizing additional CPU machines to aid the GPU machines( 2). It seems to be a good opportunity to accelerate DNNtraining because GPU clusters indeed have spare CPU andbandwidth resources ( ).

8 Unfortunately,in practice,allthe existing PS have inferior performance for multiple designreasons, as we shall see soon in this paper. It is therefore nota surprise to see that Distributed DNN training speed recordsare dominated by all-reduce [27,49,73].We are thus motivated to designBytePS1, an architecturethat is communication-optimal, both in theory and in , both all-reduce and PS are theoretically op-timal only in very specific GPU/CPU setups, while are not1 The name BytePS was chosen in the early stage of this project [4].

9 However,it is conceptually different from the conventional PS Association14th USENIX Symposium on Operating Systems Design and Implementation 463the optimal for more generic settings, ,there are some fi-nite additional CPU resources. By carefully allocating trafficloads, BytePS unifies the cases where PS or all-reduce is the-oretically optimal, and generalizes the optimality to any givennumber of GPU/CPU machines with different PCIe/NVLinkconfigurations, with analytical top of that, BytePS pushes its real-world performanceclose to the theoretical limit, by removing bottlenecks in exist-ing PS designs.

10 With fast high-speed networks, we found thatCPUs are not fast enough for the full fledged DNN optimiz-ers. We introduce a new abstraction,Summation Service, toaddress this issue. We split an optimizer into gradient aggre-gation and parameter update. We keep gradient aggregationin Summation Service running on CPUs and move param-eter update, which is more computation intensive, to addition, in implementation, we incorporated the idea ofpipelining and priority-scheduling from prior work [34,55]and resolved multiple RDMA-related performance a drop-in replacement for all-reduce and PS, BytePSaims to accelerate Distributed training without changing theDNN algorithm or its accuracy at all.