Data Warehousing 2 - devgear.co.kr

1 Corporate Headquarters EMEA Headquarters Asia-Pacific Headquarters 100 California Street, 12th Floor San Francisco, California 94111 York House 18 York Road Maidenhead, Berkshire SL6 1SF, United Kingdom L7. 313 La Trobe Street Melbourne VIC 3000 Australia White Paper Data Warehousing Modeling and Metadata Strategies for Next Generation Architectures By Bill H. inmon Forest Rim Technology, LLC April 2010 Data Warehousing Bill inmon Embarcadero Technologies, Inc. Page - 1 - INTRODUCTION Data Warehousing has undergone a constant state of evolution since the beginning. Just when you think that everything has been discovered and developed, data Warehousing evolves once again, mutating into a new form and structure. EVOLUTION OF THE DATA WAREHOUSE From the beginning the evolution of data Warehousing has been shaped by powerful forces.

2 First there was the need for access to data. Then there was the need for integration. Then came the need for a single version of the truth. Then different departmental needs for looking at the same foundation of data arose. The general evolution of the first stages of the data warehouse environment is shown by Fig 1. edwFirst there were applications, then there was a data warehouse, then there was an infrastructure surrounding the data warehouse Figure 1 In the beginning there were applications. Applications grew so quickly that there developed what was termed the spider web environment where the same data was scattered all over the landscape. The frustration with the spider web environment led to the creation of the first data warehouse. The simple and singular data warehouse addressed many of the problems of the spider web environment.

3 But soon it was discovered that other components of the data warehouse infrastructure were needed. There was a need for ETL, the process that allows data to be read in an unintegrated application format and written out in a corporate, integrated format. Then there ware data marts, where different departments had their own version of the base data found in the data warehouse environment. Soon the ODS appeared where there was a need for high performance, transaction processing on integrated data. An entire infrastructure grew up around the world of the simple data warehouse. An architecture called the cif , or the corporate information factory, grew up around the data warehouse. But the evolution of the data warehouse did not stop there. Soon the evolution of the data warehouse grew to include a broader set of requirements.

4 Soon the notion of a data warehouse expanded to include a full set of data fulfilling a newly discovered set of requirements that reached well beyond the original notion of what a data warehouse should be. Data Warehousing Bill inmon Embarcadero Technologies, Inc. Page - 2 - DW : ARCHITECTURE FOR THE NEXT GENERATION OF DATA Warehousing This new architecture was called DW . Fig 2 shows the basic description of DW Current++OlderLess thancurrentVe rycurrentInteractiveIntegratedNear lineArchivalSummarySubjSubjSubjSubjDetai ledSummarySubjSubjSubjSubjDetailedSummar ySubjSubjSubjSubjDetailedTransactiondata ApplApplApplContinuoussnapshotdataSubjSu bjSubjProfiledataContinuoussnapshotdataS ubjSubjSubjProfiledataContinuoussnapshot dataSubjSubjSubjProfiledataText to subjText id ..Internal, externalTex tualsu bjectsCapturedtextLinkageText to subjText id.

5 Internal, externalTex tualsubjectsCapturedtextLinkageText to subjText id ..Internal, externalTex tualsubjectsCapturedtextLin kageDW for the next generation of data warehousingSimplepointerSimplepointerSim plepointerThen there was DW ,architecture for the nextgeneration of datawarehousing Figure 2 Like the earlier renditions of data warehouse architecture that preceded DW , DW was shaped by powerful evolutionary forces. THE LIFE CYCLE OF DATA The first powerful evolutionary force that shaped DW was the recognition that the data within the data warehouse contained its own life cycle. It was not enough to merely place data Data Warehousing Bill inmon Embarcadero Technologies, Inc. Page - 3 - on disk storage and call it a data warehouse. As time passed that data began to exhibit its own characteristics.

6 The first manifestation of the life cycle of the data within the data warehouse was that over time, the probability of access to data in the data warehouse dropped. The older data became, the less that data was accessed. A second manifestation of the lifecycle of data with the data warehouse was that over time the volumes of data in the data warehouse grew rapidly. This led to a paradox: the larger a data warehouse became and the older the data in the data warehouse, the smaller percentage of data was being used. UNSTRUCTURED DATA The second manifestation of evolutionary forces in the data warehouse was the realization that unstructured, textual data belonged in the data warehouse. The original data that was placed in the data warehouse was transaction based, repetitive data. Many corporate decisions were based on this kind of data.

7 But there is much important data that is not transaction based that belongs in the data warehouse. METADATA A third realization was that metadata belonged in the data warehouse as a formal and integral component. In first generation data warehouses, metadata was an afterthought. But there are powerful reasons why metadata needs to become an integral and formal part of the data warehouse environment. Fig 3 shows the evolutionary forces and their effect on the DW environment. Current++OlderLess thancurrentVe rycu rrentInteractiveIntegratedNear lineArchivalSummarySubjSubjSubjSubjDetai ledSummarySubjSubjSubjSubjDetailedSummar ySubjSubjSubjSubjDetailedTransactiondata ApplApplApplContin uoussnapshotdataSubjSubjSubjProfiledataC ontin uoussnapshotdataSubjSubjSubjProfiledataC ontin uoussn apshotdataSubjSubjSubjProfiledataText to subjText id.

8 Internal, externalTex tualsubjectsCapturedtextLinkageText to subjText id ..Interna l, externalTex t ualsu bjec tsCapturedtextLin kageText to subjText id ..Internal, externalTex t ualsu bjec tsCapturedtextLinkageDW for the next generation of data warehousingSimplepointerSimplepointerSim plepointerA recognition of the life cycleof data within the data warehouseRecognition of the need to integrateboth structured and unstructured datain the data warehouseRecognitionof the needfor a formal metadatainfrastructure Figure 3 DW has become the architectural paradigm for modern data warehouses. For a detailed description of DW refer to the book DW ARCHITECTURE FOR THE NEXT GENERATION OF DATA Warehousing , Morgan Kaufman, 2008. Data Warehousing Bill inmon Embarcadero Technologies, Inc. Page - 4 - Like most large architectures, DW is not build all at once.

9 Instead an organization builds first one component of DW then another component. Indeed some components of the data warehouse are optional, such as the near line sector. COMPARTMENTS OF PROCESSING In any case, the different components of DW consist of sectors that perform one basic function. These sectors are in a sense their own small operating environments. Each sector has its own purpose and its own functionality. Fig 4 shows that each sector is neatly compartmentalized. Ve rycurrentInteractiveLess thancurrentNear lineText t o subjText id ..Internal, externalTe xtu alsubjectsCapturedtextLinkageSimplepoint erCurrent++IntegratedText to subjText id ..Internal, externalTe xtu alsubjectsCapturedtextLinkageSimplepoint erTransactiondataApplApplApplSummarySubj SubjSubjSubjDetailedContinuoussnapshotda taSubjSubjSubjProf iledataSummarySubjSubjSubjSubjDetailedCo ntinuoussnapshotdataSubjSubjSubjProfiled ataSummarySubjSubjSubjSubjDetailedContin uoussnapshotdataSubjSubjSubjProf iledataOlderArchivalText to subjText id.

10 Inte rna l, ex ter nalTe xtu alsubjectsCapturedtextLinkageSimplepoint erDW is made up ofdifferent components Figure 4 In light of the individual compartmentalization of the sectors of DW , the question then becomes how is work distributed and coordinated across the different components? The answer is that work is distributed and coordinated by means of the passage of metadata from one component to the next. METADATA AS THE GLUE Fig 5 shows that metadata is the glue that binds the different operating components of DW together. Data Warehousing Bill inmon Embarcadero Technologies, Inc. Page - 5 - Ve rycurrentInteractiveLess thancurrentNear lineText to subjText id ..Internal, externalTe xtu alsubjectsCapturedtextLinkageSimplepoint erCurrent++IntegratedText to subjText id ..Internal, externalTe xtu alsubjectsCapturedtextLinkageSimplepoint erTransactiondataApplApplApplSummarySubj SubjSubjSubjDetailedContinuoussnapshotda taSubjSubjSubjProfiledataSummarySubjSubj SubjSubjDetailedContinuoussnapshotdataSu bjSubjSubjProfiledataSummarySubjSubjSubj SubjDetailedContinuoussnapshotdataSubjSu bjSubjProfiledataOlderArchivalText to subjText id.