Introduction to Data Mining - exinfm

1 Osmar R. Za ane, 1999 CMPUT690 Principles of Knowledge Discovery in Databases University of Alberta page 1 Department of Computing Science Chapter I: Introduction to data Mining We are in an age often referred to as the information age. In this information age, because we believe that information leads to power and success, and thanks to sophisticated technologies such as computers, satellites, etc., we have been collecting tremendous amounts of information. Initially, with the advent of computers and means for mass digital storage, we started collecting and storing all sorts of data , counting on the power of computers to help sort through this amalgam of information.

2 Unfortunately, these massive collections of data stored on disparate structures very rapidly became overwhelming. This initial chaos has led to the creation of structured databases and database management systems (DBMS). The efficient database management systems have been very important assets for management of a large corpus of data and especially for effective and efficient retrieval of particular information from a large collection whenever needed. The proliferation of database management systems has also contributed to recent massive gathering of all sorts of information.

3 Today, we have far more information than we can handle: from business transactions and scientific data , to satellite pictures, text reports and military intelligence. Information retrieval is simply not enough anymore for decision-making. Confronted with huge collections of data , we have now created new needs to help us make better managerial choices. These needs are automatic summarization of data , extraction of the essence of information stored, and the discovery of patterns in raw data . What kind of information are we collecting? We have been collecting a myriad of data , from simple numerical measurements and text documents, to more complex information such as spatial data , multimedia channels, and hypertext documents.

4 Here is a non-exclusive list of a variety of information collected in digital form in databases and in flat files. Business transactions: Every transaction in the business industry is (often) memorized for perpetuity. Such transactions are usually time related and can be inter-business deals such as purchases, exchanges, banking, stock, etc., or intra-business operations such as management of in-house wares and assets. Large department stores, for example, thanks to the widespread use of bar codes, store millions of transactions daily representing often terabytes of data .

5 Storage space is not the major problem, as the price of hard disks is continuously dropping, but the effective use of the data in a reasonable time frame for competitive decision-making is definitely the most important problem to solve for businesses that struggle to survive in a highly competitive world. Scientific data : Whether in a Swiss nuclear accelerator laboratory counting particles, in the Canadian forest studying readings from a grizzly bear radio collar, on a South Pole iceberg gathering data about oceanic activity, or in an American university investigating human psychology, our society is amassing colossal amounts of scientific data that need to be analyzed.

6 Unfortunately, we can capture and store more new data faster than we can analyze the old data already accumulated. Osmar R. Za ane, 1999 CMPUT690 Principles of Knowledge Discovery in Databases University of Alberta page 2 Department of Computing Science Medical and personal data : From government census to personnel and customer files, very large collections of information are continuously gathered about individuals and groups. Governments, companies and organizations such as hospitals, are stockpiling very important quantities of personal data to help them manage human resources, better understand a market, or simply assist clientele.

7 Regardless of the privacy issues this type of data often reveals, this information is collected, used and even shared. When correlated with other data this information can shed light on customer behaviour and the like. Surveillance video and pictures: With the amazing collapse of video camera prices, video cameras are becoming ubiquitous. Video tapes from surveillance cameras are usually recycled and thus the content is lost. However, there is a tendency today to store the tapes and even digitize them for future use and analysis.

8 Satellite sensing: There is a countless number of satellites around the globe: some are geo-stationary above a region, and some are orbiting around the Earth, but all are sending a non-stop stream of data to the surface. NASA, which controls a large number of satellites, receives more data every second than what all NASA researchers and engineers can cope with. Many satellite pictures and data are made public as soon as they are received in the hopes that other researchers can analyze them. Games: Our society is collecting a tremendous amount of data and statistics about games, players and athletes.

9 From hockey scores, basketball passes and car-racing lapses, to swimming times, boxer s pushes and chess positions, all the data are stored. Commentators and journalists are using this information for reporting, but trainers and athletes would want to exploit this data to improve performance and better understand opponents. Digital media: The proliferation of cheap scanners, desktop video cameras and digital cameras is one of the causes of the explosion in digital media repositories. In addition, many radio stations, television channels and film studios are digitizing their audio and video collections to improve the management of their multimedia assets.

10 Associations such as the NHL and the NBA have already started converting their huge game collection into digital forms. CAD and Software engineering data : There are a multitude of Computer Assisted Design (CAD) systems for architects to design buildings or engineers to conceive system components or circuits. These systems are generating a tremendous amount of data . Moreover, software engineering is a source of considerable similar data with code, function libraries, objects, etc., which need powerful tools for management and maintenance.