An Introduction to Variable and Feature Selection

1 Journal of machine learning Research 3 (2003) 1157-1182 Submitted 11/02; Published 3/03An Introduction to Variable and Feature SelectionIsabelle Creston RoadBerkeley, CA 94708-1501, USAAndr e Inference for machine learning and Perception DepartmentMax Planck Institute for Biological CyberneticsSpemannstrasse 3872076 T ubingen, GermanyEditor:Leslie Pack KaelblingAbstractVariable and Feature Selection have become the focus of muchresearch in areas of application forwhich datasets with tens or hundreds of thousands of variables are available. These areas includetext processing of internet documents, gene expression array analysis, and combinatorial objective of Variable Selection is three-fold: improving the prediction performance of the pre-dictors, providing faster and more cost-effective predictors, and providing a better understanding ofthe underlying process that generated the data.

2 The contributions of this special issue cover a widerange of aspects of such problems: providing a better definition of the objective function, featureconstruction, Feature ranking, multivariate Feature Selection , efficient search methods, and featurevalidity assessment : Variable Selection , Feature Selection , space dimensionality reduction, pattern discov-ery, filters, wrappers, clustering, information theory, support vector machines, model Selection ,statistical testing, bioinformatics, computational biology, gene expression, microarray, genomics,proteomics, QSAR, text classification, information IntroductionAs of 1997, when a special issue on relevance including several papers on Variable and featureselection was published (Blum and Langley, 1997, Kohavi and John, 1997), few domains exploredused more than 40 features.

3 The situation has changed considerably in thepast few years and, inthis special issue, most papers explore domains with hundreds to tens of thousands of variables orfeatures:1 New techniques are proposed to address these challenging tasks involving many irrelevantand redundant variables and often comparably few training examples are typical of the new application domains and serve us as illustration throughoutthis Introduction . One is gene Selection from microarray data and the other is text the gene Selection problem, the variables are gene expression coefficients corresponding to the1. We call Variable the raw input variables and features variables constructed for the input variables.

4 We usewithout distinction the terms Variable and Feature when there is no impact on the Selection algorithms, , whenfeatures resulting from a pre-processing of input variables are explicitly computed. The distinction is necessary inthe case of kernel methods for which features are not explicitly computed (see section ).c 2003 Isabelle Guyon and Andr e ANDELISSEEFF abundance of mRNA in a sample ( tissue biopsy), for a number of patients. A typical clas-sification task is to separate healthy patients from cancer patients, based ontheir gene expression profile . Usually fewer than 100 examples (patients) are available altogether for training and test-ing.

5 But, the number of variables in the raw data ranges from 6000 to 60,000. Some initial filteringusually brings the number of variables to a few thousand. Because the abundance of mRNA variesby several orders of magnitude depending on the gene, the variables are usually standardized. In thetext classification problem, the documents are represented by a bag-of-words , that is a vector ofdimension the size of the vocabulary containing word frequency counts (proper normalization of thevariables also apply). Vocabularies of hundreds of thousands of words are common, but an initialpruning of the most and least frequent words may reduce the effectivenumber of words to 15, document collections of 5000 to 800,000 documents are available forresearch.

6 Typical tasksinclude the automatic sorting of URLs into a web directory and the detection of unsolicited email(spam). For a list of publicly available datasets used in this issue, see Table 1at the end of the are many potential benefits of Variable and Feature Selection : facilitating data visualizationand data understanding, reducing the measurement and storage requirements, reducing training andutilization times, defying the curse of dimensionality to improve prediction performance. Somemethods put more emphasis on one aspect than another, and this is another point of distinctionbetween this special issue and previous work. The papers in this issue focus mainly on constructingand selectingsubsets of featuresthat areusefulto build a good predictor.

7 This contrasts with theproblem of finding or ranking all potentially relevant variables. Selecting the most relevant variablesis usually suboptimal for building a predictor, particularly if the variables areredundant. Conversely,a subset of useful variables may exclude many redundant, but relevant, variables. For a discussionof and definitions of the various notions of relevance, see the review articlesof Kohavi and John (1997) and Blum and Langley (1997).This Introduction surveys the papers presented in this special issue. Thedepth of treatment ofvarious subjects reflects the proportion of papers covering them: the problem of supervised learningis treated more extensively than that of unsupervised learning ; classification problems serve moreoften as illustration than regression problems, and only vectorial input datais considered.

8 Complex-ity is progressively introduced throughout the sections: The first section starts by describingfiltersthat select variables by ranking them with correlation coefficients (Section2). Limitations of suchapproaches are illustrated by a set of constructed examples (Section 3).Subset Selection methodsare then introduced (Section 4). These includewrapper methodsthat assess subsets of variables ac-cording to their usefulness to a given predictor. We show how some embedded methods implementthe same idea, but proceed more efficiently by directly optimizing a two-part objective function witha goodness-of-fit term and a penalty for a large number of variables.

9 We then turn to the problem offeature construction, whose goals include increasing the predictor performance and building morecompact Feature subsets (Section 5). All of the previous steps benefit from reliably assessing thestatistical significance of the relevance of features. We briefly review model Selection methods andstatistical tests used to that effect (Section 6). Finally, we conclude the paper with a discussion sec-tion in which we go over more advanced issues (Section 7). Because the organization of our paperdoes not follow the work flow of building a machine learning application, we summarize the stepsthat may be taken to solve a Feature Selection problem in a check list2:2.

10 We caution the reader that this check list is heuristic. The only recommendation that is almost surely valid is to trythe simplest things TOVARIABLE you have domain knowledge?If yes, construct a better set of ad hoc your features commensurate?If no, consider normalizing you suspect interdependence of features?If yes, expand your Feature set by constructingconjunctive features or products of features, as much as your computer resources allow you(see example of use in Section ). you need to prune the input variables( for cost, speed or data understanding rea-sons)? If no, construct disjunctive features or weighted sums of features ( by clusteringor matrix factorization, see Section 5).