Glow: Graph Lowering Compiler Techniques for Neural Networks

1 glow : Graph Lowering Compiler Techniques forNeural NetworksNadav Rotem, Jordan Fix, Saleem Abdulrasool, Garret Catron, Summer Deng,Roman Dzhabarov, Nick Gibson, James Hegeman, Meghan Lele, Roman Levenstein,Jack Montgomery, Bert Maher, Satish Nadathur, Jakob Olesen, Jongsoo Park,Artem Rakhov, Misha Smelyanskiy, Man WangFacebookAbstractThis paper presents the design of glow , a machinelearning Compiler for heterogeneous hardware. It isa pragmatic approach to compilation that enablesthe generation of highly optimized code for multi-ple targets.

2 glow lowers the traditional Neural net-work dataflow Graph into a two-phase strongly-typedintermediate representation. The high-level inter-mediate representation allows the optimizer to per-form domain-specific optimizations. The lower-levelinstruction-based address-only intermediate represen-tation allows the Compiler to perform memory-relatedoptimizations, such as instruction scheduling, staticmemory allocation and copy elimination. At the low-est level, the optimizer performs machine-specific codegeneration to take advantage of specialized hardwarefeatures.

3 glow features a Lowering phase which en-ables the Compiler to support a high number of inputoperators as well as a large number of hardware targetsby eliminating the need to implement all operators onall targets. The Lowering phase is designed to reducethe input space and allow new hardware backends tofocus on a small number of linear algebra IntroductionThe end of power saving due to Moore s Law, com-bined with the increased demand for compute powerdriven by machine learning, has led to a wave of in-novation in computer architecture.

4 Hennessy andPatterson [1] present five principles that guide the de-sign of machine-learning domain specific architectures(DSA): dedicated local memories, large numbers ofarithmetic units, simple forms of parallelism, reducedbit-widths, and domain-specific programming mod-els. Compilers need to perform advance whole-graphoptimizations in order to execute Neural Networks effi-ciently on DSAs. This paper describes some of thesetechniques as implemented in glow , an open-sourcemachine learning Compiler framework for heteroge-neous machine learning frameworks iterateover the nodes in the Graph and execute them oneby one.

5 Unfortunately this node-visitor method ofexecution is inefficient, even on traditional a result, machine learning frameworks have startedto hand over the Graph to compilers [2, 3] that exe-cute code more efficiently. Based on the increasingimportance of Neural Networks , the need for energyefficiency in data centers and mobile devices, and thedesign principles of domain-specific architectures, webelieve that the machine learning frameworks of thefuture will focus on providing attractive programmingmodels on top of a layer that integrates compilers formany different glow , we focus on the lower parts of the soft-ware stack.

6 We work to provide PyTorch [4] andother frameworks with a low-level Graph and a codegenerator for Neural Networks . The name glow is anabbreviation for Graph - Lowering , which is the maintechnique that the Compiler uses for generating effi-cient code. The glow low-level Graph will not replacethe machine learning high-level Graph , in the sameway that the low-level intermediate representation incompilers does not replace the abstract syntax aim to provide a useful Compiler toolkit that willallow hardware developers to focus on implementingefficient acceleration hardware, each of which likelydiffer in capabilities, and use glow for automatingcompilation tasks such as instruction selection, mem-ory allocation and Graph scheduling.

7 The full compilertoolkit is open-source and publicly [ ] 3 Apr 2019 CadenceR , EsperantoR , HabanaR , IntelR , andQualcomm Technologies , a subsidiary of Qual-comm IncorporatedR , have committed to supportingGlow in future silicon products. Each of their acceler-ators will likely differ in capabilities, and can use Glowfor automating compilation tasks such as instructionselection, memory allocation, and Graph Related Relationship to Neural NetworkFrameworksFrameworks such as PyTorch [4], Caffe [5], and Ten-sorFlow [6] have found success by providing a usefulway for developers to create Neural network models,and executing them on specific architectures.

8 How-ever, supporting new architectures and operators isnot scalable, because adding a new operator requiresit to be implemented on each supported architecture,and adding a new architecture requires all operatorsbe implemented for it. glow is designed to consume aneural network compute Graph , optimize it, and codegenerate for it for a diverse set of backends in a morescalable way. This includes target-independent opti-mizations and analysis prior to efficiently targeting aspecific [7] is an open-source format for represent-ing and serializing AI models.

9 It allows for interop-erability between different AI frameworks, allowingcompute graphs from one framework such as PyTorchto be converted to or from another framework suchas Cognitive Toolkit (CNTK) [8]. Compiler -Related ProjectsSeveral prior systems use a Compiler -oriented ap-proach to optimizing Neural Networks . TensorFlow sXLA [2] compiles Neural Networks for CPUs, GPUsand accelerators. It is a practical Compiler that solvesactual problems. XLA is used in production to drivea massive fleet of accelerators at Google.

10 It lowersnodes into primitive linear algebra operations, andthen calls into a backend-specific library for differentbackend (such as Eigen [9] for CPUs, or cuDNN [10]for GPUs) to perform the bulk of computation. Wepoint out that XLA emits vectorized LLVM interme-diate representation (IR) [11] for some nodes (suchas dot), and relies on the LLVM vectorizer [12] forother nodes. It aims to provide a backend flexibilityfor (..) A[i] = 3;for (..) A[i] = 4;return A[0];Figure 1:Compilers struggle to analyze and optimizethis code when the two loops come from two differentnodes in the dataflow TensorFlow, in a similar way that glow is work-ing toward providing for PyTorch and other neuralnetwork [3,13] lowers nodes into a low-levelHalide-based IR wherein loop-based optimizations canbe performed.