TPOT: A Tree-based Pipeline Optimization Tool for ...

1 JMLR: Workshop and Conference Proceedings 64:66 74, 2016 ICML 2016 AutoML WorkshopTPOT: A Tree-based Pipeline Optimization Toolfor automating machine LearningRandal S. H. for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USAA bstractAs data science becomes more mainstream, there will be an ever-growing demand for datascience tools that are more accessible, flexible, and scalable. In response to this demand,automated machine learning (AutoML) researchers have begun building systems that auto-mate the process of designing and optimizing machine learning pipelines.

2 In this paper wepresent TPOT , an open source genetic programming-based AutoML system that op-timizes a series of feature preprocessors and machine learning models with the goal of max-imizing classification accuracy on a supervised classification task. We benchmark TPOTon a series of 150 supervised classification tasks and find that it significantly outperformsa basic machine learning analysis in 21 of them, while experiencing minimal degradationin accuracy on 4 of the benchmarks all without any domain knowledge nor human such, GP-based AutoML systems show considerable promise in the AutoML :automated machine learning, hyperparameter Optimization , Pipeline optimiza-tion, genetic programming, Pareto Optimization , data science, Python1.

3 IntroductionMachine learning is commonly described as a field of study that gives computers the abilityto learn without being explicitly programmed (Simon, 2013). Despite this common claim, machine learning practitioners know that designing effective machine learning pipelinesis often a tedious endeavor, and typically requires considerable experience with machinelearning algorithms, expert knowledge of the problem domain, and brute force search toaccomplish (Olson et al., 2016a). Thus, contrary to what machine learning enthusiastswould have us believe, machine learning still requires considerable explicit response to this challenge, several automated machine learning methods have beendeveloped over the years (Hutter et al.)

4 , 2015). Over the past year, we have been developinga Tree-based Pipeline Optimization tool (TPOT) that automatically designs and optimizesmachine learning pipelines for a given problem domain (Olson et al., 2016b), without anyneed for human intervention. In short, TPOT optimizes machine learning pipelines usinga version of genetic programming (GP), a well-known evolutionary computation techniquefor automatically constructing computer programs (Banzhaf et al., 1998). Previously, wedemonstrated that combining GP with Pareto Optimization enables TPOT to automaticallyconstruct high-accuracyandcompact pipelines that consistently outperform basic machinelearning analyses (Olson et al.

5 , 2016a). In this paper, we extend that benchmark to include150 supervised classification tasks and evaluate TPOT in a wide variety of applicationdomains ranging from genetic analyses to image classification and 2016 Olson & : A tool for automating machine Learning2. MethodsIn the following sections, we provide an overview of the Tree-based Pipeline OptimizationTool (TPOT) , including the machine learning operators used as genetic programming(GP) primitives, the Tree-based pipelines used to combine the primitives into working ma-chine learning pipelines, and the GP algorithm used to evolve said Tree-based pipelines.

6 Wefollow with a description of the data sets used to evaluate the latest version of TPOT inthis paper. TPOT is an open source project on GitHub, and the underlying Python codecan be found machine Learning Pipeline OperatorsAt its core, TPOT is a wrapper for the Python machine learning package, scikit-learn (Pe-dregosa et al., 2011). Thus, each machine learning Pipeline operator ( , GP primitive)in TPOT corresponds to a machine learning algorithm, such as a supervised classificationmodel or standard feature scaler. All implementations of the machine learning algorithmslisted below are from scikit-learn (except XGBoost), and we refer to the scikit-learn docu-mentation (Pedregosa et al.)

7 , 2011) and Hastie et al. (2009) for detailed explanations of themachine learning algorithms used in Classification , RandomForest, eXtreme Gra-dient Boosting Classifier (from XGBoost, Chen and Guestrin (2016)), LogisticRegression,and KNearestNeighborClassifier. Classification operators store the classifier s predictionsas a new feature as well as the classification for the Preprocessing , RobustScaler, MinMaxScaler,MaxAbsScaler, RandomizedPCA (Martinsson et al., 2011), Binarizer, and PolynomialFea-tures. Preprocessing operators modify the data set in some way and return the modifieddata Selection , SelectKBest, SelectPercentile, Se-lectFwe, and Recursive Feature Elimination (RFE).

8 Feature selection operators reduce thenumber of features in the data set using some criteria and return the modified data also include an operator that combines disparate data sets, as demonstrated inFigure 1, which allows multiple modified copies of the data set to be combined into asingle data set. Lastly, we provide integer and float terminals to parameterize the variousoperators, such as the number of neighborskin thek-Nearest Neighbors Constructing Tree-based PipelinesTo combine these operators into a machine learning Pipeline , we treat them as GP primitivesand construct GP trees from them.

9 Figure 1 shows an example Tree-based Pipeline , wheretwo copies of the data set are provided to the Pipeline , modified in a successive mannerby each operator, combined into a single data set, and finally used to make all operators receive a data set as input and return the modified data set as output,it is possible to construct arbitrarily shaped machine learning pipelines that can act onmultiple copies of the data set. Thus, GP trees provide an inherently flexible representationof machine learning MooreEntire Data SetPCAE ntire Data SetPolynomial FeaturesCombine FeaturesSelect k Best FeaturesModified data setflows through thepipeline operatorsThe final classificationis performed on thefinal feature setMultiple copies of thedata set can enter thepipeline for analysisPipeline operatorsmodify the featuresLogistic RegressionFigure 1: An example Tree-based Pipeline from TPOT.

10 Each circle corresponds to a machinelearning operator, and the arrows indicate the direction of the data order for these Tree-based pipelines to operate, we store three additional variables foreach record in the data set. The class variable indicates the true label for each record, andis used when evaluating the accuracy of each Pipeline . The guess variable indicates thepipeline s latest guess for each record, where the classifications from the last classificationoperator in the Pipeline are stored as the guess . Finally, the group variable indicateswhether the record is to be used as a part of the internal training or testing set, such thatthe Tree-based pipelines are only trained on the training data and evaluated on the testingdata.