Neural Architecture Search: A Survey

1 Journal of Machine learning Research 20 (2019) 1-21 Submitted 9/18; Revised 3/19; Published 3/19 Neural Architecture Search: A SurveyThomas Center for Artificial Intelligence71272 Renningen, Germanyand University of FreiburgJan Hendrik Center for Artificial Intelligence71272 Renningen, GermanyFrank of Freiburg79110 Freiburg, GermanyEditor:Sebastian NowozinAbstractDeep learning has enabled remarkable progress over the last years on a variety of tasks,such as image recognition, speech recognition, and machine translation. One crucial aspectfor this progress are novel Neural architectures.

2 Currently employed architectures havemostly been developed manually by human experts, which is a time-consuming and error-prone process. Because of this, there is growing interest in automatedneural architecturesearchmethods. We provide an overview of existing work in this field of research and cate-gorize them according to three dimensions: search space, search strategy, and performanceestimation : Neural Architecture Search, AutoML, AutoDL, Search Space Design, SearchStrategy, Performance Estimation Strategy1. IntroductionThe success of deep learning in perceptual tasks is largely due to its automation of thefeature engineering process: hierarchical feature extractors are learned in an end-to-endfashion from data rather than manually designed.

3 This success has been accompanied,however, by a rising demand forarchitecture engineering, where increasingly more complexneural architectures are designed Architecture Search(NAS), the processof automating Architecture engineering, is thus a logical next step in automating machinelearning. Already by now, NAS methods have outperformed manually designed architec-tures on some tasks such as image classification (Zoph et al., 2018; Real et al., 2019), objectdetection (Zoph et al., 2018) or semantic segmentation (Chen et al., 2018). NAS can beseen as subfield of AutoML (Hutter et al.)

4 , 2019) and has significant overlap with hyperpa-rameter optimization (Feurer and Hutter, 2019) and meta- learning (Vanschoren, 2019). Wecategorize methods for NAS according to three dimensions: search space, search strategy,and performance estimation strategy:c 2019 Thomas Elsken, Jan Hendrik Metzen and Frank : CC-BY , see Attribution requirements are provided SpaceASearch StrategyarchitectureA Aperformanceestimate ofAFigure 1: Abstract illustration of Neural Architecture Search methods. A search strategyselects an architectureAfrom a predefined search spaceA.

5 The Architecture ispassed to a performance estimation strategy, which returns the estimated perfor-mance ofAto the search strategy. Search search space defines which architectures can be representedin principle. Incorporating prior knowledge about typical properties of architectureswell-suited for a task can reduce the size of the search space and simplify the , this also introduces a human bias, which may prevent finding novel archi-tectural building blocks that go beyond the current human knowledge. Search search strategy details how to explore the search space(which is often exponentially large or even unbounded).

6 It encompasses the clas-sical exploration-exploitation trade-off since, on the one hand, it is desirable to findwell-performing architectures quickly, while on the other hand, premature convergenceto a region of suboptimal architectures should be avoided. Performance Estimation objective of NAS is typically to findarchitectures that achieve high predictive performance on unseen to the process of estimating this performance: the simplest option isto perform a standard training and validation of the Architecture on data, but this isunfortunately computationally expensive and limits the number of architectures thatcan be explored.

7 Much recent research therefore focuses on developing methods thatreduce the cost of these performance refer to Figure 1 for an illustration. The article is also structured according to thesethree dimensions: we start with discussing search spaces in Section 2, cover search strategiesin Section 3, and outline performance estimation methods in Section 4. We conclude withan outlook on future directions in Section Search SpaceThe search space defines which Neural architectures a NAS approach might discover inprinciple. We now discuss common search spaces from recent relatively simple search space is the space ofchain-structured Neural networks, as illus-trated in Figure 2 (left).

8 A chain-structured Neural network architectureAcan be writtenas a sequence ofnlayers, where the i th layerLireceives its input from layeri 1 and2inputL0L1 LnoutputinputL0L2L4L6L8L10L1L3L7L9L5outp utLn 1 Figure 2: An illustration of different Architecture spaces. Each node in the graphs cor-responds to a layer in a Neural network, , a convolutional or pooling layer types are visualized by different colors. An edge from layerLitolayerLjdenotes thatLjreceives the output ofLias input. Left: an element of achain-structured space.

9 Right: an element of a more complex search space withadditional layer types and multiple branches and skip output serves as the input for layeri+ 1, ,A=Ln ..L1 L0. The search spaceis then parametrized by: (i) the (maximum) number of layersn(possibly unbounded);(ii) the type of operation every layer executes, , pooling, convolution, or more advancedoperations like depthwise separable convolutions (Chollet, 2016) or dilated convolutions (Yuand Koltun, 2016); and (iii) hyperparameters associated with the operation, , numberof filters, kernel size and strides for a convolutional layer (Baker et al.)

10 , 2017a; Suganumaet al., 2017; Cai et al., 2018a), or simply number of units for fully-connected networks (Men-doza et al., 2016). Note that the parameters from (iii) are conditioned on (ii), hence theparametrization of the search space is not fixed-length but rather a conditional work on NAS (Brock et al., 2017; Elsken et al., 2017; Zoph et al., 2018; Elskenet al., 2019; Real et al., 2019; Cai et al., 2018b) incorporates modern design elementsknown from hand-crafted architectures, such as skip connections, which allow to buildcomplex,multi-branch networks, as illustrated in Figure 2 (right).