FINN: A Framework for Fast, Scalable Binarized …

1 FINN: A Framework for fast , Scalable Binarized NeuralNetwork InferenceYaman Umuroglu* , Nicholas J. Fraser* , Giulio Gambardella*, Michaela Blott*,Philip Leong , Magnus Jahre and Kees Vissers**Xilinx Research Labs; Norwegian University of Science and Technology; University of has shown that convolutional neural networks con-tain significant redundancy, and high classification accuracycan be obtained even when weights and activations are re-duced from floating point to binary values. In this paper,we presentFinn, a Framework for building fast and flexibleFPGA accelerators using a flexible heterogeneous stream-ing architecture. By utilizing a novel set of optimizationsthat enable efficient mapping of Binarized neural networksto hardware, we implement fully connected, convolutionaland pooling layers, with per-layer compute resources beingtailored to user-provided throughput requirements.

2 On aZC706 embedded FPGA platform drawing less than 25 Wtotal system power, we demonstrate up to million imageclassifications per second with s latency on the MNIST dataset with accuracy, and 21906 image classificationsper second with 283 s latency on the CIFAR-10 and SVHN datasets with respectively and accuracy. Tothe best of our knowledge, ours are the fastest classificationrates reported to date on these INTRODUCTIONC onvolutional neural Networks (CNNs) have dramaticallyimproved in recent years, their performance now exceedingthat of other visual recognition algorithms [14], and even sur-passing human accuracy on certain problems [23, 28]. Theyare likely to play an important role in enabling ubiquitousmachine vision and intelligence on all kinds of devices, but asignificant computational challenge remains. Modern CNNsmay contain millions of floating-point parameters and requirebillions of floating-point operations to recognize a single im-age.

3 Furthermore, these requirements tend to increase as re-searchers explore deeper networks. For instance, AlexNet [14](the winning entry for ImageNet Large Scale Visual Recogni-tion Competition (ILSVRC) [22] in 2012) required 244 MB ofparameters and billon floating point operations (GFLOP)per image, while VGG-16 [24] from ILSVRC 2014 required552MB of parameters and GFLOP per appear in the 25th International Symposium on Field-Programmable Gate Arrays, February the vast majority of CNNs implementations usefloating point parameters, a growing body of research demon-strates this approach incorporates significant , it has been shown [5, 26, 21, 12, 31] that neu-ral networks can classify accurately using one- or two-bitquantization for weights and activations. Such a combina-tion of low-precision arithmetic and small memory footprintpresents a unique opportunity for fast and energy-efficientimage classification using Field Programmable Grid Arrays(FPGAs).

4 FPGAs havemuchhigher theoretical peak per-formance for binary operations compared to floating point,while the small memory footprintremovesthe off-chip mem-ory bottleneck by keeping parameters on-chip, even for largenetworks. Binarized neural Networks (BNNs), proposed byCourbariaux et al. [5], are particularly appealing since theycan be implemented almost entirely with binary operations,with the potential to attain performance in the teraoperationsper second (TOPS) range on this work, we proposeFinn, a Framework for build-ing Scalable and fast BNN inference accelerators on accelerators can perform millions of classi-fications per second with sub-microsecond latency, therebymaking them ideal for supporting real-time embedded appli-cations such as augmented reality, autonomous driving androbotics. Compute resources can be scaled to meet a givenclassification rate requirement.

5 We demonstrateFinn s capa-bilities with a series of prototypes for classifying the MNIST,SVHN and CIFAR-10 benchmark datasets. Our classificationrate results surpass the best previously published results byover48 for MNIST, for CIFAR-10 and8 for the best of our knowledge, this is the fastest reportedneural network inference implementation on these novel contributions are: Quantification of peak performance for BNNs onFPGAs using a roofline model. A set of novel optimizations for mapping BNNs ontoFPGA more efficiently. A BNN architecture and accelerator construction tool,permitting customization of throughput. A range of prototypes that demonstrate the potentialof BNNs on an off-the-shelf FPGAs rest of this paper is organized as follows: Section 2provides background on CNNs, BNNs, and their hardwareimplementations. Section 3 discusses BNNs accuracy andpeak performance on FPGAs.

6 Section 4 describesFinn [ ] 1 Dec 2016architecture and optimizations. Section 5 presents the exper-imental evaluation, and Section 6 concludes the Convolutional neural NetworksThis work is focused onsupervisedlearning, in whichthe goal is to find a function,g(xi), which approximatesa mappingxi yi i, where{xi,yi}is an input/outputpair known as a training example. Amultilayer perceptronis a type of artificial neural network which has its neuronsarranged in multiple layers, with neurons taking the outputof all neurons of the previous layer as inputs. Mathematically,the output,al,n, for thenthneuron in thelthlayer of a fullyconnected network is calculated as follows:al,n=fact(Sl s=0wl,n,sal 1,s+bl,n),(1)wherewl,n,sis weight of thesthsynapse connected to the in-put of thenthneuron in thelthlayer,bl,nis a bias term,factis the activation function, andSlis the number of synapsesconnected to each neuron in thelthlayer.

7 Popular acti-vation functions include: the hyperbolic tangent function,fact(a) =tanh(a); and the rectified linear unit (ReLU),fact(a) =max(0,a). Furthermore, only theinferenceprob-lem is studied, the parameters,w, being assumed to havebeen learned neural networks [15] (CNNs) are a variantof multilayer perceptrons, in which a layer only receivesinputs from a smallreceptive fieldof the previous layer. Thisapproach greatly reduces the number of parameters involvedand allows local features ( , edges, corners) to be found [15].A basic 2D convolutional layer in a neural network is similarto a fully connected layer except that: a) each neuron receivesan image as inputs and produces an image as its output(instead of a scalar); b) each synapse learns a small arrayof weights which is the size of the convolutional window;and c) each pixel in the output image is created by thesum of the convolutions between all synapse weights and thecorresponding output of thelthconvolutional layer, which takes asinputSlimages of dimensionRl Cl, the pixel,pl,n,r,c, atlocation(r,c) of thenthoutput image is calculated as follows:pl,n,r,c=fact(Sl s=0Jl j=0Kl k=0wl,n,s,j,kpl 1,n,r+j,c+k),(2)whereJl Klare the dimensions of the convolution discussed in Section 4, a 2D convolutional layer can bereduced to a matrix multiply followed by an elementwiseactivation function.

8 CNN topologies are composed from afew common primitives: convolutional layers,poolinglayersand fully connected layers can be considered as simple downsamplersof 2D images. A basic max pooling layer divides an imageinto small sub-tiles of a given window size and then replaceseach sub-tile with its largest element. An average poolinglayer is similar but uses the average function instead of Binary neural NetworksAlthough floating point numbers are a natural choice forhandling the small updates that occur during neural networktraining, the resulting parameters can contain a lot of re-dundant information [8]. One of several possible dimensionspossessing redundancy is precision [26]. An extreme caseare BNNs in which some or all the arithmetic involved incomputing the outputs are constrained to single-bit consider three aspects of binarization for neural networklayers: binary input activations, binary synapse weights andbinary output activations.

9 If all three components are binary,we refer to this asfull binarization, and the cases with oneor two components aspartial and Smaragdis [12] consider full binarization with apredetermined portion of the synapses having zero weight,and all other synapses with a weight of one. They accuracy with fully-connected networks on the MNIST dataset, and observe that only XNOR and bitcount opera-tions are necessary for computing with such neural by Rastegari et al. [21] applies convolutionalBNNs on the ImageNet dataset with topologies inspired byAlexNet, ResNet and GoogLeNet, reporting top-1 accuraciesof up to for full binarization and for partial bi-narization. DoReFa-Net by Zhou et al. [31] explores reducedprecision during the forward pass as well as the backwardpass, and note that this opens interesting possibilities fortraining neural networks on FPGAs.

10 Their results includesconfigurations with partial and full binarization on the SVHNand ImageNet datasets, including best-case ImageNet top-1accuracies of 43% for full and 53% for partial , the work by Courbariaux et al. [5] describes howto train fully-connected and convolutional networks withfull binarization and batch normalization layers, reportingcompetitive accuracy on the MNIST, SVHN and CIFAR-10datasets. Training for this work was performed using theiropen source implementation. We use the acronym CNN torefer to conventional or non- Binarized neural networks forbrevity throughout the rest of this neural Networks in HardwareA great deal of prior work on mapping neural networks tohardware exist both for FPGAs and as ASICs. We refer thereader to the work by Misra and Saha [16] for a comprehen-sive survey. We cover a recent and representative set of workshere, roughly dividing them into four categories based ontheir basic architecture: 1) a single processing engine [19, 30,4, 2], usually in the form of a systolic array, which processeseach layer sequentially; 2) a streaming architecture [27, 1],consisting of one processing engine per network layer; 3) avector processor [7] with instructions specific to acceleratingthe primitives operations of convolutions; and 4) a neurosy-naptic processor [6], which implements many digital neuronsand their interconnecting arrays:Zhang et al.