Knowledge Discovery and Data Mining: Towards a Unifying ...

1 Knowledge Discovery and data Mining: Towards a Unifying FrameworkUsama FayyadMicrosoft ResearchOne Microsoft WayRedmond, WA 98052, Piatetsky-ShapiroGTE Laboratories, MS 44 Waltham, MA 02154, SmythInformation and Computer ScienceUniversity of California, IrvineCA 92717-3425, paper presents a first step Towards a unifyingframework for Knowledge Discovery in Databases. Wedescribe finks between data milfing, Knowledge dis-covery, and other related fields. We then define theKDD process and basic data mining algorithms, dis-cuss application issues and conclude with an analysisof challenges facing practitioners in the IntroductionAcross a wide variety of fields, data are being collectedand accumulated at a dramatic pace.

2 There is an ur-gent need for a new generation of computational tech-niques and tools to assist humans in extracting use-ful information ( Knowledge ) from the rapidly growingvolumes of data , These techniques and tools are thesubject of the emerging field of Knowledge discoveryin databases (KDD). This paper is an initial step to-wards a common framework that we hope will allowus to understand the variety of activities in this multi-disciplinary field and how they fit together. We viewthe Knowledge Discovery process as a set of various ac-tivities for making sense of data .

3 At the core of thisprocess is the application of data mining methods forpatternt Discovery . We examine how data mining isused and outline some of its methods. Finally, we lookat practical application issues of KDD and enumeratechallenges for future research and KDD, data mining , and Relation toother FieldsHistorically the notion of finding useful patterns indata has been given a variety of names including datamining, Knowledge extraction, information Discovery ,information harvesting, data archaeology, and datapattern processing.

4 The term data mining has beenmostly used by statisticians, data analysts, and the~Throughout this paper we use the term "pattern" todesignate pattern or model extracted from the information systems (MIS) has also gained popularity in the database term KDD was coined at the first KDD work-shop in 1989 (Piatetsky-Shapiro 199t) to emphasizethat " Knowledge " is the end product of a data -drivendiscovery. It has been popularized in artificial intelli-gence and machine our view KDD refers to the overall process of dis-covering useful Knowledge from data while data miningrefers to a particular step in this process.

5 data miningis the application of specific algorithms for extractingpatterns from data . The distinction between the KDDprocess and the data mining step (within the process)is a central point of this paper. The additional steps inthe KDD process, such as data preparation, data se-lection, data cleaning, incorporating appropriate priorknowledge, and proper interpretation of the results ofmining, are essential to ensure that useful Knowledge isderived from the data . Blind application of data min-ing methods (rightly criticised as " data dredging" inthe statistical literature)

6 Can be a dangerous activityeasily leading to Discovery of meaningless has evolved, and continues to evolve, from theintersection of research fields such as machine learning,pattern recognition, databases, statistics, artificial in-telligence, Knowledge acquisition for expert systems, data visualization, and high performance Unifying goal is extracting high-level knowledgefrom low-level data in the context of large data overlaps with machine learning and patternrecognition in the study of particular data mining the-ories and algorithms: means for modeling data andextracting patterns.

7 KDD focuses on aspects of find-ing understandable patterns that can be interpreted asuseful or interesting Knowledge , and puts a strong em-phasis on working with large sets of real-world , scaling properties of algorithms to large datasets are of fundamental also has much in common with statistics, par-ticularly exploratory data analysis methods. The sta-82I<DD-96 From: KDD-96 Proceedings. Copyright 1996, AAAI ( ). All rights reserved. tistical approach offers precise methods for quantifyingthe inherent uncertainty which results when one triesto infer general patterns from a particular sample of anoverall population.

8 KDD software systems often em-bed particular statistical procedures for sampling andmodeling data , evaluating hypotheses, and handlingnoise within an overall Knowledge Discovery contrast to traditional approaches in statistics, KDDapproaches typically employ more search in model ex-traction and operate in the context of larger data setswith richer data addition to its strong relation to the databasefield (the 2nd D in KDD), another related area data warehousing, which refers to the popular businesstrend for collecting and cleaning transactional datato make them available for on-line analysis and de-cision support.

9 A popular approach for analysis ofdata warehouses has been called OLAP (on-line an-alytical processing), after a set of principles proposedby Codd (1993). OLAP tools focus on providing multi-dimensional data analysis, which is superior to SQL incomputing summaries and breakdowns along many di-mensions. OLAP tools are targeted Towards simplify-ing and supporting interactive data analysis, while theKDD tool s goal is to automate as much of the processas Basic DefinitionsWe define KDD (Fayyad, Piatetsky-Shapiro, & Smyth1996) asKnowledge Discovery in Databases is the non-trivial process oi" identifying valid, novel, potentiallyuseful, and ultimately understandable patterns data is a set of facts ( , cases in a database)

10 And pattern is an expression in some language describ-ing a subset of the data or a model applicable to thatsubset. Hence, in our usage here, extracting a patternalso designates fitting a model to data , finding struc-ture from data , or in general any high-level descrip-tion of a set of data . The term process implies thatKDD is comprised of many steps, which involve datapreparation, search for patterns, Knowledge evaluation,and refinement, all repeated in multiple iterations. Bynon-trivial we mean that some search or inference is in-volved, it is not a straightforward computation ofpredefined quantities like computing the average valueof a set of numbers.