Mastering Machine Learning with scikit-learn

1 Machine Learning with scikit -learnApply effective Learning algorithms to real-world problems using scikit -learnGavin HackelingBIRMINGHAM - Machine Learning with scikit -learnCopyright 2014 Packt PublishingAll rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals.

2 However, Packt Publishing cannot guarantee the accuracy of this published: October 2014 Production reference: 1221014 Published by Packt Publishing Place35 Livery StreetBirmingham B3 2PB, image by Amy-Lee Winfield HackelingReviewersFahad ArshadSarah GuidoMikhail KorobovAman MadaanAcquisition EditorMeeta RajaniContent Development EditorNeeshma RamakrishnanTechnical EditorFaisal SiddiquiCopy EditorsRoshni BanerjeeAdithi ShettyProject CoordinatorDanuta JonesProofreadersSimran BhogalTarsonia SangheraLindsey ThomasIndexerMonica Ajmera MehtaGraphicsSheetal AuteRonak DhruvDisha HariaProduction CoordinatorKyle AlbuquerqueCover WorkKyle the AuthorGavin Hackeling develops Machine Learning services for large-scale documents and image classification at an advertising network in New York. He received his Master's degree from New York University's Interactive Telecommunications Program, and his Bachelor's degree from the University of North Hallie, for her support, and Zipper, without whose contributions this book would have been completed in half the the ReviewersFahad Arshad completed his PhD at Purdue University in the Department of Electrical and Computer Engineering.

3 His research interests focus on developing algorithms for software testing, error detection, and failure diagnosis in distributed systems. He is particularly interested in data-driven analysis of computer systems. His work has appeared at top dependability conferences DSN, ISSRE, ICAC, Middleware, and SRDS and he has been awarded grants to attend DSN, ICAC, and ICNP. Fahad has also been an active contributor to security research while working as a cybersecurity engineer at NEES comm IT. He has recently taken on a position as a systems engineer in the Guido is a data scientist at Reonomy, where she's helping build disruptive technology in the commercial real estate industry. She loves Python, Machine Learning , and the startup world. She is an accomplished conference speaker and an O'Reilly Media author, and is very involved in the Python community. Prior to joining Reonomy, Sarah earned a Master's degree from the University of Michigan School of Korobov is a software developer at ScrapingHub Inc.

4 , where he works on web scraping, information extraction, natural language processing, Machine Learning , and web development tasks. He is an NLTK team member, Scrapy team member, and an author or contributor to many other open source 'd like to thank my wife, Aleksandra, for her support and patience and for the Madaan is currently pursuing his Master's in Computer Science and Engineering. His interests span across Machine Learning , information extraction, natural language processing, and distributed computing. More details about his skills, interests, and experience can be found at files, eBooks, discount offers, and moreYou might want to visit for support files and downloads related to your you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at and as a print book customer, you are entitled to a discount on the eBook copy.

5 Get in touch with us at for more , you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read, and search across Packt's entire library of subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via web browserFree access for Packt account holdersIf you have an account with Packt at , you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate of ContentsPreface 1 chapter 1: The Fundamentals of Machine Learning 7 Learning from experience 8 Machine Learning tasks 10 Training data and test data 11 Performance measures, bias, and variance 13An introduction to scikit - learn 16 Installing scikit - learn 16 Installing scikit - learn on Windows 17 Installing scikit - learn on Linux 17 Installing scikit - learn on OS X 18 Verifying the installation 18 Installing pandas and matplotlib 18 Summary 19 chapter 2.

6 linear Regression 21 Simple linear regression 21 Evaluating the fitness of a model with a cost function 25 Solving ordinary least squares for simple linear regression 27 Evaluating the model 29 Multiple linear regression 31 Polynomial regression 35 Regularization 40 Applying linear regression 41 exploring the data 41 Fitting and evaluating the model 44 Fitting models with gradient descent 46 Summary of Contents[ ii ] chapter 3: Feature Extraction and Preprocessing 51 Extracting features from categorical variables 51 Extracting features from text 52 The bag-of-words representation 52 Stop-word filtering 55 Stemming and lemmatization 56 Extending bag-of-words with TF-IDF weights 59 Space-efficient feature vectorizing with the hashing trick 62 Extracting features from images 63 Extracting features from pixel intensities 63 Extracting points of interest as features 65 SIFT and SURF 67 Data standardization 69 Summary 70 chapter 4.

7 From linear Regression to Logistic Regression 71 Binary classification with logistic regression 72 Spam filtering 73 Binary classification performance metrics 76 Accuracy 77 Precision and recall 79 Calculating the F1 measure 80 ROC AUC 81 Tuning models with grid search 84 Multi-class classification 86 Multi-class classification performance metrics 90 Multi-label classification and problem transformation 91 Multi-label classification performance metrics 94 Summary 95 chapter 5: Nonlinear Classification and Regression with Decision Trees 97 Decision trees 97 Training decision trees 99 Selecting the questions 100 Information gain 103 Gini impurity 108 Decision trees with scikit - learn 109 Tree ensembles 112 The advantages and disadvantages of decision trees 113 Summary of Contents[ iii ] chapter 6: Clustering with K-Means 115 Clustering with the K-Means algorithm 117 Local optima 123 The elbow method 124 Evaluating clusters 128 Image quantization 130 Clustering to learn features 132 Summary 135 chapter 7.

8 Dimensionality Reduction with PCA 137An overview of PCA 137 Performing Principal Component Analysis 142 Variance, Covariance, and Covariance Matrices 142 Eigenvectors and eigenvalues 143 Dimensionality reduction with Principal Component Analysis 146 Using PCA to visualize high-dimensional data 149 Face recognition with PCA 150 Summary 153 chapter 8: The Perceptron 155 Activation functions 157 The perceptron Learning algorithm 158 Binary classification with the perceptron 159 Document classification with the perceptron 166 Limitations of the perceptron 167 Summary 169 chapter 9: From the Perceptron to Support Vector Machines 171 Kernels and the kernel trick 172 Maximum margin classification and support vectors 176 Classifying characters in scikit - learn 179 Classifying handwritten digits 179 Classifying characters in natural images 182 Summary 185 chapter 10.

9 From the Perceptron to Artificial Neural Networks 187 Nonlinear decision boundaries 188 Feedforward and feedback artificial neural networks 189 Multilayer perceptrons 189 Minimizing the cost function 191 Forward propagation 192 Backpropagation of Contents[ iv ]Approximating XOR with Multilayer perceptrons 212 Classifying handwritten digits 213 Summary 214 Index years have seen the rise of Machine Learning , the study of software that learns from experience. While Machine Learning is a new discipline, it has found many applications. We rely on some of these applications daily; in some cases, their successes have already rendered them mundane. Many other applications have only recently been conceived, and hint at Machine Learning 's this book, we will examine several Machine Learning models and Learning algorithms.

10 We will discuss tasks that Machine Learning is commonly applied to, and learn to measure the performance of Machine Learning systems. We will work with a popular library for the Python programming language called scikit - learn , which has assembled excellent implementations of many Machine Learning models and algorithms under a simple yet versatile book is motivated by two goals: Its content should be accessible. The book only assumes familiarity with basic programming and math. Its content should be practical. This book offers hands-on examples that readers can adapt to problems in the real [ 2 ]What this book coversChapter 1, The Fundamentals of Machine Learning , defines Machine Learning as the study and design of programs that improve their performance of a task by Learning from experience. This definition guides the other chapters; in each chapter , we will examine a Machine Learning model, apply it to a task, and measure its 2, linear Regression, discusses linear regression, a model that relates explanatory variables and model parameters to a continuous response variable.