GUIDE TO THE HEALTH FACILITY DATA QUALITY …

1 GUIDE TO THE HEALTH FACILITY data QUALITY REPORT CARD Introduction No HEALTH data is there is no one definition of data No HEALTH data from any source can be considered perfect: all data are subject to a number of limitations related to QUALITY , such as missing values, bias, measurement error, and human errors in data entry and computation. data QUALITY assessment is needed to understand how much confidence can be put in the HEALTH data presented. In particular, it is important to know the reliability of national coverage estimates and other estimates derived from HIS data that are generated for HEALTH sector reviews, as these often form the basis for annual monitoring. However, there is no one definition of data QUALITY that is used consistently across institutions.

2 data QUALITY is a multi-dimensional construct. Overall data QUALITY , then, becomes a function of each of its dimensions. If data QUALITY is a latent construct and is defined as a function of its different dimensions, by extension the assessment of data QUALITY would entail an assessment of each of its dimensions. There are different tools that assess different dimensions of data Some tools focus on assessing the status of the system that is producing the data . This is based on questions about presence of trained staff, forms and electronic media, procedures, adherence to definitions, and ways in which data QUALITY is assessed. Other tools focus on the assessment of the QUALITY of the actual data that are generated by the system. The first step is a desk review of data QUALITY .

3 A desk review of data will not necessarily identify the underlying causes of inaccurate data but it will identify problems of data completeness, accuracy and external HEALTH FACILITY data are a critical input into assessing national progress and performance on an annual basis and they provide the basis for subnational / district performance assessment. WHO proposes the HEALTH FACILITY data QUALITY Report Card (DQRC), which is a methodology that examines certain dimensions of data QUALITY through a desk review of available data and a data verification component. The aim of DQRC is to ensure systematic assessment of completeness and internal and external consistency of the reported data or computed statistics and determine whether there are any data QUALITY problems that need to be addressed.

4 The desk review component of the DQRC is conducted through the use of the WHO data QUALITY Assessment (DQA) Tool, an Excel-based tool, that reviews the QUALITY of data generated by a HEALTH FACILITY -based information system for four key indicators: antenatal care first visit (ANC1), HEALTH FACILITY deliveries, diphtheria-pertussis-tetanus third dose (DTP3) and outpatient department (OPD) visits. Through analysis of these four standard tracer indicators, the tool quantifies problems of data completeness, accuracy and external consistency and thus provides valuable information on fit-for-purpose of HEALTH FACILITY data to support planning and annual monitoring. data verification refers to the assessment of reporting correctness , that is, comparing HEALTH FACILITY source documents to HIS reported data to determine the proportion of the reported numbers that can be verified from the source documents.

5 It checks whether the information contained in the source documents has been transmitted correctly to the next higher level of reporting, for each level of reporting, from the HEALTH FACILITY level to the national level. It is recommended to implement data verification with the annual HEALTH FACILITY survey (Service Availability Readiness Assessment (SARA)) on a 1 Different frameworks and different dimensions of data QUALITY will be discussed in a data QUALITY assessment guideline document that is under development. In this document, we will only focus on two of the dimensions of data QUALITY that are assessed by the data QUALITY Report Card. representative sample of HEALTH facilities to obtain a national level estimate of the verification factor for the HEALTH information system.

6 The DQRC examines four dimensions of data QUALITY . These dimensions are: 1) completeness of reporting; 2) internal consistency of reported data ; 3) external consistency of population data ; 4) external consistency of coverage rates. There are two levels of assessment for the indicators in the DQRC: 1) an assessment of each indicator at the national level; and 2) performance of sub-national units, mostly districts or provinces/regions, on the selected indicator. The indicator definitions change when evaluated nationally or sub-nationally. The DQRC has been primarily designed to be examined at the national level. However, it is possible that certain provinces/states/regions in a country might want to look just at their own performance. For example, a province might want to examine data QUALITY in their districts.

7 This is possible to do in the DQRC. Any reference to the word national can be replaced with a sub-national For each of the four dimensions a small set of indicators is used. The indicators can, with adaptations, be used for most indicators that can be derived from HEALTH FACILITY data . data QUALITY problems are usually systemic and are not specific to any one program area. For instance, there may be a group of districts or facilities that do not report at all or poorly. Or the denominators of the coverage indicators, based on population projections, are systematically off. Even if it is not possible to do an exhaustive data QUALITY analysis of all the key indicators in the national HEALTH strategy, conducting a data QUALITY analysis as described below can indicate potential problems in multiple program areas.

8 The focus in this manual is on maternal and child HEALTH indicators. The DQRC should be generated on annual basis to evaluate the QUALITY of the data to be used for annual reviews. The following table gives a quick overview of the indicators of the DQRC. Detailed explanation for each of the indicator are given in the subsequent section. 2 This should, however, be done cautiously. For external comparison, survey aggregation levels are usually only at the state/province/regional levels. If a province wants to look at within province data QUALITY performance, it will not be able to make external comparisons of their FACILITY data with survey data if the survey aggregation level from the most recent population-based survey is only available at the province level.

9 National Profile Sub-national Profile Indicator Definition Application of indicator at the Sub-national unit level Completeness of reporting Completeness of sub-national unit reporting % of monthly sub-national unit (such as district) reports received for a specified period time (usually one year) Total # of sub-national unit reports received Total # of expected sub-national unit reports Number and % of sub-national units that had less than 80% completeness of monthly reporting for a specified period time (usually one year) # of sub-national units with less than 80% reporting completeness nationally_____ Total # of sub-national units in the country Completeness of FACILITY reporting % of expected monthly FACILITY reports received for a specified period time (usually one year) Total # of FACILITY reports received nationally Total # of expected FACILITY reports nationally Number and % of sub-national units with monthly FACILITY reporting rates below 80% for a specified period time (usually one year)

10 # of sub-national units with monthly reporting rates less than 80% nationally_____ Total # of sub-national units in the country Completeness of indicator data (zero/missing values) % of monthly Sub-national unit reports that are NOT zero/missing (Average of 4 indicators: ANC1, Deliveries, DTP3, OPD) Total # of zero/missing values for all sub-national units for the reporting year for ANC1 + Deliveries + DTP3 + OPD_____ Total # of sub-national units X 12 X 4 Number and % of sub-national units with at least 20% zero/missing values (Average of four indicators: ANC1, Deliveries, DTP3, OPD) ((# of sub-national units with more than 20% zero and missing values for all four indicators (ANC1 + Deliveries + DTP3 + OPD) combined* Total # sub-national units in the country *A sub-national unit has more than 20% missing/zero values for the four indicators combined when the equation below is greater than 20%.))