Zhang Qing zhangqingbj@inspur.com HPC …

1 Zhang Qing Application R&D Manager,Inspur Inspur-Nvidia GPU Joint Lab Introduction Caffe-MPI: Parallel CAFFE framework based on GPU clusterInspur-Nvidia GPU Joint Lab Introduction Inspur-Nvidia GPU Joint Lab App Research Directions Traditional HPC Deep Learning FieldApplicationClientsSpeed-up ratioPlatformLife ScienceBLASTNB eijing Institute of Genomics35X(kernel)1 GPU /1 CPU coreETInstitute of Biophysics, CSA48X1 GPU /1 CPU coreCFDLBM_LES 100X1 GPU /1 CPU coreOil&gasRNABGP8X24 GPU nodes /24 CPU nodesPSTM5X6 GPU nodes / 6 CPU nodesScandip9X4 GPU+2 CPU 16 GPU/1 GPUDNNIF lick13X16 GPU/1 GPUK-meansQihoo35X1 GPU/1 CPU coreNeural NetworkQihoo270X4 GPU/1 CPU core Application DNN Client IFLYTEK Performance 16 GPU/1 GPU = 13X Deep learning For speech recognitionMobile PhoneIntelligent customer serviceBusiness travel queryCar Application: neural network Client.

2 Qihoo Performance 4 GPU/1 CPU core =270 XTime(s)4%16%80%Net updateWeight computingForwardBackward computingData parallelSome part can be paralleledSome part can be paralleled Caffe has many users, it is very popular in China. Caffe need a long training time for big data based one GPU node. Caffe s ForwardBackward computing,weight computing and net update all can be paralleled with GPU cluster. What is Caffe-MPI? Developed by Inspur Open-source Based on the Berkeley Vision and Learning Center (BVLC) Single GPU Caffe version A GPU Cluster Caffe version Support 16+ GPUs to Train based on HPC Technology Hardware arch IB+GPU cluster+Lustre Software arch MPI+Pthread+CUDA Data parallel on GPU ClusterGPU Cluster Configuration GPU master nodeMulti GPUsGPU Salve NodeMulti GPUsStorageLustrenetwork56Gb/s MPI Mast-Slave model Master Process Multi Pthread Threads+CUDA Threads Slave Process CUDA ThreadsReference Q Ho J Cipar H Cui JK Kim S Lee.

3 More Effective Distributed ML via a Stale Synchronous Parallel Parameter Server. Master Process (0 process) Three Pthread Groups Parallel read data and send data Weight Computing and The parameter update The parameter communication Slave process CPU To receive training data from the master process To send weight data(GPU-to-GPU) To receive new net data(GPU-to-GPU) GPU ForwardBackward computing Slave Node The number of Slave process = the number of GPU GPU parallel computing Computing & Communication asynchronous parallel Communication Optimization GPU RDMA Weight Data and Net data between GPUsTotal Time=max(TRead Data+Send Data ,TForwardBackWord Computing+ Weight Computing and Net Update+ Net Send) Speed-up Ratio 16 GPU/1 GPU= Scalability efficiency 65% Speed-up Ratio 16 GPU/1 GPU= Scalability efficiency 67% Peformance speed by cuDNN =21% Speed-up Ratio 16 GPU/1 GPU= Scalability efficiency 79%GoogleNet(Iterations=4000,batchsize=6 4)Training Time(s)03507001,0501,400 The Number of GPU116(Caffe-MPI)16(Caffe-MPI+cuDNN)

4 1091321,380 Parallel read training data from Lustre Storage and send data to different GPUs GPU Cluster be divided into many groups Every group have a master node Every master node parallel read and send data with Multi Processes +Multi Threads Can support large-scale GPU computing for a big training platform Speed-up Ratio 16 GPU/1 GPU=13X Scalability efficiency 81% The Next work Support cuDNN MPI Framework tuning Symmetric model Caffe-MPI version open source roadmap Q2:Computing-Intensive Model support 32+ GPU parallel Q3:IO-Intensive Model:support 16+ GPU parallel Q4:Support Half Precision for Pascal GPUC onclusions Caffe-MPI is based on HPC technology architecture Performance:16 GPU/1 GPU=13X Caffe-MPI can support 16+ GPU to train big data Inspur will continue to open source new versions 32 GPU parallel version for Computing-Intensive Model 16+ GPU parallel version for IO Support Half Precision for Pascal GPU