TUTORIAL No 2 How to determine repeatability and ...

1 3rdEuropean-American Workshop on NDE ReliabilityTUTORIAL No 2 How to determine repeatability and reproducibility (R&R)Dr Damir of Zagreb, Department of Quality, Faculty of Mechanical Engineering & Naval Architecture, CroatiaTutorial , terminology / experiment(how to perform,layout, DoE) analysis (statistical analysis, examples) remarks& discussionLiterature21. IntroductionConcept of R&R as a measure of precision in measurement laboratories, was introduced & developed during 1978-1983 [2]. It was widely accepted but also criticised, and finally resulted with ISO 5725 in any other field,first essential prerequisite is to exchange information, data and knowledge and to understand each other properly so, convention about definitions is "Each term must have the same meaning for all of its users; it must therefore at the same time express a well defined concept and not to be in conflict with everyday language" [1].

2 First, we ll discuss two terms; accuracyand precision; so we ll come to R& IntroductionIt is known from common everyday use that when we repeatsome measurement or want to reproducemeasurement result in some other occasion,we will almost never get exactly the same result. We could observe same phenomena if we want to measure some quantity with different systems!..51. Introductionjust now !.. now, we can carry out simple measurement:6 What time is ..1. IntroductionChristine s watch10:0211:0212:0213:0214:0215:02 Which one you more prefer ?True time10:0011:0012:0013:0014:0015:00My watch10:0011:0112:0213:0314:0215 your watch an accurateand/or preciseone ?72. Definitions / Accuracy, trueness, precision R&R conditions and limits Elements of test system Systematic and random errors Accepted reference value8 Accuracydesignates closeness of agreement between a test result(measurement result) and the accepted reference value (true value) [1,3].

3 NotesThe term precision should not be used for accuracy [1].9 Truenessdesignates closeness of agreement between the average valueobtained from a large series of test result and an accepted reference value [3].NotesThe measure of trueness is usually expressed in terms of bias[3].Trueness has been referred to as accuracy of the mean , what is not recommended [3].10 Precisiondesignates variability of results, or closeness of the agreement between test resultsof repetitive measurements [3].In total, the results of particular set of measurements will be (randomly) distributed around expected measure of dispersionof results gives us information about (trueness and/or precision) .. variability of resultsA)B)C)12 Both, trueness and precision are quantitative can estimatedispersion (precision) by calculation of standard , we can estimateso-called "true value" by calculation of a variability of results13variabilityexample A)14variabilityexample B)15variabilityexample C)16 More important is that we can compare and analyse differences between particular sets of test results.

4 This directs us to the subject of repeatability and reproducibility .. variability of results17 Differentvariabilitiesbetween differentsets of test resultsregarding todifferent sourcesof variabilities. CADFEB18 Regarding factors (elements)of the systemwhich contribute to the variability of the test results,two conditions of precision, termed repeatability and reproducibilityconditions, have been found necessary and useful for describing the variability of a measurement method [3].. definitions of R&R ..19 Conditions where independent test results are obtained with the same methodon identical test itemsin the same laboratoryby the same operatorusing the same equipmentwithin short intervals of time [3].Repeatabilityis precision under repeatability conditions 20 Conditions where test results are obtained with the same methodon identical test itemsin different laboratorieswith different operatorsusing different equipment [3].

5 Reproducibilityis precision under reproducibility conditions 21 Thus repeatability and reproducibility are the two extremes of minimumandreproducibilitythe maximum variabilityin extremes of variability22 The value less than or equal to which the absolute difference betweentwo test resultsobtained under repeatability conditionsmay be expected to be with a probability of 95 % [3]. repeatability limit,r232rsfr =The value less than or equal to which the absolute difference betweentwo test resultsobtained under reproducibility conditionsmay be expected to be with a probability of 95 % [3]. reproducibility limit, R242 RsfR =RepeatabilityIllustration ofthe meaninig ofrepeatabilitystandard deviationand ofthe meaninig ofreproducibilitystandard deviationand elements of test system [4] Operator Equipment Calibration Time27 Elements of test system regarding NDT/mine_det Personnel(operator/technician human factor) Equipment(instruments, tools, utensils, probes/sensors,manipulators, spare parts & materials, reference blocks.)

6 Procedure(measurement/testmethod, calibration) Test object / measured entity(any characteristic: microstructure, geometry, surface conditions, orientation, ) Environment (temperature, humidity, viewing conditions ..)28 Trueness and systematic errorspecified toleranceexpected value29 bias (B) for infinite groups ofmeasurementsBis randomlydistributedarroundexpected valueIs a meanthat would result from an infinite number of measurements of the same measurand carried out under repeatability conditionsminusa true valueof the measurand [1].Systematic error30specified toleranceexpected valuePrecision and random error31Is a resultof a measurement minusthe meanthat would result from an infinite number of measurements of the same measurand carried out under repeatability conditions[1].Because only a finite number of measurements can be made, it is possible to determine only an estimate of random error [1].

7 Random error32 Accepted reference valueA value that serves as an agreed-upon referencefor comparison [3].Could be derived as theoretical or established value, an assigned or certified value, a consensus. When not available, the expectationof the quantity, the mean of a specified population of measurements [3].333. R&R experiment Experiment layout How to organise and carry out the experiment DoE34 Experiment layout The objective of the experiment is to determine the precisionobtainable by the general population of operators performing standard measurement (test) method. Each combination of a laboratory and a level is called cell. Each cell contains nrepetitive test results. 35 Experiment layout36p 1level 2level 3125,337,543,9A225,237,744,4325,337,744, 0125,437,444,4B225,437,544,5325,337,744, 9125,137,744,3C225,337,344,9325,037,644, 0125,137,444,6D225,537,244,6325,537,544, 0125,337,544,3E225,237,444,5325,337,744, 7 Experiment layoutqlevelsplaboratoriesnreplicate test results CADFEB37 Performance of the experiment(1) Perform the measurements according to the specified standard method !

8 Group of nmeasurementson each level shall be carried out under repeatability conditions(short time, without intermediate recalibration of the apparatus unless this is an integral part of standard procedure).38 Performance of the experiment(2) It is essential that a group of nmeasurements on each level be performed independently (independent repetitive test results). Sometimes, despite instructions to the operator, during obtaining group of nmeasurements at specific level, previous results may influence subsequent test results, and thus the repeatability variance. Then the codification of samples should be considered in such way that the operator will not know which are the replicates for a given level. ( repeatability conditions?)39 Performance of the experiment(3) It is not essential that all the qgroups of nmeasurements each be performed strictly within short interval; different groups of measurements may be carried out on different days.

9 Measurements of all qlevels shall be performed by one and the same operator. In addition, the nmeasurements at a given level shall be performed using the same equipmentthroughout. 40 Performance of the experiment(4) If unexpectedly operator becomes unavailable, another operator may complete measurements but this can only occur between two of the q groups, and it has to be reported. A time limit shall be All samples shall be clearly and properly labeled. (identification)41 Performance -Tips (1) For some measurements, there may be in fact a teamof operators, each of whom performs some specific part of the procedure. In such case, the team shall be regarded as "the operator". Any change in team indicates "different operator". In precision experiment test results shall be reported to at least one more digitthan specified in the standard procedure. 42 Performance -Tips (2) For the purposes of this ISO, a laboratoryis considered to be a combination of the operator, the equipmentand the test site[3].

10 One test site (or laboratory) may thus provide several "laboratories"(several operators each with the independent sets of equipment and situations). 43 Performance -Tips (3) It should be pointed out to the operators that the purpose of the exercise is to discover the extent to which results can vary in practice,so that there will be less temptation for them to discard or rework results that they feel are inconsistent. Operators shall reportany anomaliesor difficulties, rather than adjusting test result in order to avoid missing data. (There is R&R analysis specifically for experiments with missing data, and also for redundant data and outliers.)444. R&R analysis Statistical analysis (step by step) Calculation of R&R values Examples45R&R step by step "procedure"(1) to determine R&R values for a standard measurement/test )Premises[3, 4]: repeatability conditions within labs (nreplicate test results @ qlevels) reproducibility conditions between plabs it is assumed that distribution of the test results is approximately normal the model for test result is y = m + B + e, mis a general mean (expectation), Bbias, eerror.