Data Mining with Python (Working draft) - DTU

1 Data Mining with Python (Working draft) Finn Arup NielsenNovember 29, 2017 ContentsContentsiList of FiguresviiList of Tablesix1 Other introductions to Python ? .. Why Python for data Mining ? .. Why not Python for data Mining ? .. Components of the Python language and software .. Developing and running Python .. , pypy, IPython .. Notebook .. 2 vs. Python 3 .. in the cloud .. Python in the browser ..72 Basics .. Datatypes .. (bool) .. (int,float,complexandDecimal) .. (str) .. (dict) .. and times .. containers classes.

2 Functions and arguments .. functions withlambdas.. function arguments .. Object-oriented programming .. as functions .. Modules and import .. import .. with Python 2/3 incompatibility .. Persistency .. and JSON .. Documentation .. Testing .. for type .. testing .. layout and test discovery .. coverage .. in different environments .. Profiling .. Coding style .. Where isprivateandpublic? .. Command-line interface scripting .. Distinguishing between module and script .. Argument parsing .. Exit status.

3 Debugging .. Logging .. Advices ..313 Python for data Numpy .. Plotting .. plotting .. plotting .. for the Web .. Pandas .. data types .. indexing .. joining, merging and concatenations .. statistics .. SciPy .. transform .. Statsmodels .. Sympy .. Machine learning .. Text Mining .. expressions .. from webpages .. and part-of-speech tagging .. detection .. analysis .. Network Mining .. Miscellaneous issues .. Lazy computation .. Testing data Mining code ..574 Case: Pure Python matrix Code listing.

4 59ii5 Case: Pima data Problem description and objectives .. Descriptive statistics and plotting .. Statistical tests .. Predicting diabetes type ..696 Case: Data Mining a Problem description and objectives .. Reading the data .. Graphical overview on the connections between the tables .. Statistics on the number of tracks sold ..747 Case: Twitter information Problem description and objectives .. Building a news classifier ..758 Case: Big Problem description and objectives .. Stream processing of JSON.

5 Processing of JSON Lines ..78 Bibliography81 Index85iiiivPrefacePython has grown to become one of the central languages in data Mining offering both a general programminglanguage and libraries specifically targeted numerical book is continuously being written and grew out of course given at the Technical University of The Python hierarchy.. Overview of methods and attributes in the common Python 2 built-in data types plotted as aformal concept analysis lattice graph. Only a small subset of methods and attributes is shown. Sklearn classes derivation.

6 Comorbidity for ICD-10 disease code (appendicitis).. Seaborn correlation plot on the Pima data set .. Database tables graph ..73viiviiiList of Basic built-in and Numpy and Pandas datatypes .. Class methods and attributes .. Testing concepts .. Function for generation of Numpy data structures.. Some of the subpackages of SciPy.. Python machine learning packages .. Scikit-learn methods .. sklearn classifiers .. Metacharacters and character classes .. NLT submodules.. Variables in the Pima data set.

7 65ixxChapter Other introductions to Python ?Although we cover a bit of introductory Python programming in chapter 2 you should not regard this book asa Python introduction: Several free introductory ressources exist. First and foremost the officialPython Tu-torialat Beginning programmers with no or little programming experiencemay want to look into the bookThink Pythonavailable from as a book [1], while more experienced programmers can start withDive Into Pythonavailable Sheppard s presently 381-pageIntroduction to Python for Econo-metrics, Statistics and Data Analysiscovers both Python basics and Python -based data analysis with Numpy,SciPy, Matplotlib and Pandas, and it is not just relevant for econometrics [2].

8 Developers already well-versed in standard Python development but lacking experience with Python for data Mining can begin withchapter 3. Readers in need of an introduction to machine learning may take a look in Marsland sMachinelearning: An algorithmic perspective[3], that uses Python for its Why Python for data Mining ?Researchers have noted a number of reasons for using Python in the data science area (data Mining , scientificcomputing) [4, 5, 6]:1. Programmers regard Python as a clear and simple language with a highreadability. Even non-programmers may not find it too difficult.

9 The simplicity exists both in the language itself as well asin the encouragement to write clear and simple code prevalent among Python programmers. See thisin contrast to, , Perl where short form variable names allow you to write condensed code but alsorequires you to remember nonintuitive variable names. A Python program may also be 2 5 shorterthan corresponding programs written in Java, C++ or C [7, 8]. Python will run on the three main desktop computing platforms Mac, Linuxand Windows, as well as on a number of other program. With Python you get an interactive prompt with REPL (read-eval-print loop)like in Matlab and R.

10 The prompt facilitates exploratory programming convenient for many datamining tasks, while you still can develop complete programs in an edit-run-debug cycle. The Python -derivatives IPython and Jupyter Notebook are particularly suited for interactive purpose language. Python is a general purpose language that can be used to a wide varietyof tasks beyond data Mining , , user applications, system administration, gaming, web developmentpsychological experiment presentations and recording. This is in contrast to Matlab and further free website for learning Python see see how well Python with its modern data Mining packages compares with R take a look at Carl s blog posts onWill it Python ?