Methodology for data validation 1 - European Commission

1 Methodology for data validation Revised edition June 2016 Essnet Validat Foundation Marco Di Zio, Nade da Fursova, Tjalling Gelsema, Sarah Gie ing, Ugo Guarnera, J rat Petrauskien , Lucas Quensel-von Kalben, Mauro Scanu, ten Bosch, Mark van der Loo, Katrin Walsdorfer 1 Table of contents Foreword .. 3 1 What is data validation .. 5 2 Why data validation . Relationship between validation and quality.. 7 3 How to perform data validation : validation levels and validation rules .. 9 4 validation levels from a business perspective .. 10 validation rules .. 14 5 validation levels based on decomposition of metadata.

2 18 A formal typology of data validation functions .. 20 validation 22 6 Relation between validation levels from a business and a formal perspective .. 22 Applications and examples .. 24 7 Data validation as a process .. 26 Data validation in a statistical production process (GSBPM) .. 26 The informative objects of data validation (GSIM) .. 29 8 The data validation process life cycle .. 31 Design phase .. 33 Implementation phase .. 34 Execution phase .. 35 Review phase .. 36 9 Metrics for data validation .. 38 10 Properties of validation rules .. 39 Completeness .. 40 Peer review .. 40 Formal 41 Redundancy.

3 43 Methods for redundancy removal .. 43 Feasibility .. 45 Methods for finding inconsistencies .. 47 Complexity .. 48 Information needed to evaluate a rule .. 48 Computational complexity .. 49 Cohesion of a rule set .. 50 2 11 Metrics for a data validation procedure .. 52 Indicators on validation rule sets derived from observed data .. 53 Indicators on validation rule sets derived from observed and reference data .. 61 True values as reference data .. 61 Plausible data as reference data .. 65 Simulation approach .. 65 12 Assessment of validation rules .. 67 Appendix A: List of validation rules.

4 70 13 References .. 74 3 Foreword Data validation is a task that is usually performed in all National Statistical Institutes, in all the statistical domains. It is indeed not a new practice, and although it has been performed for many years, nevertheless, procedures and approaches have never been systematized, most of them are ad-hoc and also its conceptualisation within a statistical production process is not always clear. This is a cause of inefficiency both in terms of methodologies and of organization of the production system. There is an urge of producing a generic framework for data validation in order to have a reference context, and to provide tools for setting an efficient and effective data validation procedure.

5 The first part of the document is devoted to establish a generic reference framework for data validation . Firstly, the main elements needed to understand clearly what is data validation , why data validation is performed and how to process data validation are discussed. To this aim a definition for data validation is provided, the main purpose of data validation is discussed taking into account the European quality framework, and finally, for the how perspective, the key elements necessary for performing data validation , that are validation rules, are illustrated. Afterwards, data validation is analysed within a statistical production system by using the main current references in this context, , GSBPM for the business process and GSIM for defining the informative objects of data validation described as a process with an input and an output.

6 Connections with statistical data editing are clarified by considering the GSDEMs (Generic Statistical Data Editing Models). Finally, the data validation process life cycle is described to allow a clear management of such an important task. The second part of the document is concerned with the measurement of important characteristics of a data validation procedure (metrics for data validation ). The introduction of characteristics of a data validation procedure and the proposal of indicators providing quantitative information about them is useful to help the design, maintenance and monitoring of a data validation procedure.

7 In this second part, the reader can find a discussion of concepts concerning the properties of validation rules such as complexity, redundancy, completeness, and suggestions about how to analyse them. Moreover, once a validation process has been designed, there is the need to analyse its performance with respect to data. In the document, suggestions on indicators based only on observed data are proposed. These indicators are particularly useful for tuning the parameters of the data validation rules. Finally, in order to measure the quality of a data validation procedure, indicators based both on observed and reference data ( , simulated or cleaned data) are illustrated.

8 The document is intended for a broad category of readers: survey managers, methodologists, statistical production designers, and more in general for all the people involved in a data validation process. In fact, the first important objective of the document is to provide a common language about data validation that can be used in the design phase, and in the production phase as well. 4 The common language we are referring to is concerned with concepts. The introduction of a common technical language for expressing validation rules is certainly an essential part, however it is beyond of the scope of this paper.

9 Finally, it is worthwhile to remark that more research is still needed, especially in the field concerned with the development of metrics for measuring the quality of a data validation procedure. This document was developed between January and December 2015 as an output of the Essnet project Validat foundation primarily financed by Eurostat, which involved CBS, Destatis, Istat and Statistic Lithuania. 5 A generic framework for data validation 1 What is data validation (Marco Di Zio, Nade da Fursova, Tjalling Gelsema, Sarah Gie ing, Ugo Guarnera, J rat Petrauskien , Lucas Quensel-von Kalben, Mauro Scanu, ten Bosch, Mark van der Loo, Katrin Walsdorfer) The first important concept to clarify is concerned with the definition of data validation .

10 A definition for data validation is given in the Unece glossary on statistical data editing (UNECE 2013): An activity aimed at verifying whether the value of a data item comes from the given (finite or infinite) set of acceptable values. In this definition, the validation activity is referred to a single data item without any explicit mention to the verification of consistency among different data items. If the definition is interpreted as stating that validation is the verification that values of single variables belong to set of prefixed sets of values (domains) it is too strict since important activities generally considered part of data validation are left out.