Threats to Internal & External Validity

1 1 Robert S MichaelInternal & External Validity -1 Threats to Internal & External ValidityY520 Strategies for Educational InquiryRobert S MichaelRobert S MichaelInternal & External Validity -2 Why Use Experimental Designs?Man prefers to believe what he prefersto be true. Francis Bacon2 Robert S MichaelInternal & External Validity -3 Threats to Internal & External Validity Is the investigator s conclusion correct? Are the changes in the independent variable indeedresponsible for the observed variation in thedependent variable? Might the variation in the dependent variable beattributable to other causes?Robert S MichaelInternal & External Validity -4 Why is Internal Validity Important? We often conduct research in order to determinecause-and-effect relationships.

2 Can we conclude that changes in the independentvariable caused the observed changes in thedependent variable? Is the evidence for such a conclusion good or poor? If a study shows a high degree of Internal Validity thenwe can conclude we have strong evidence ofcausality. If a study has low Internal Validity , then we mustconclude we have little or no evidence of S MichaelInternal & External Validity -5 Variables & Internal Validity Extraneousvariables are variables thatmaycompetewith the independent variable in explaining theoutcome of a study. Aconfoundingvariable is an extraneous variable thatdoesindeed influence the dependent variable. A confounding variable systematically varies orinfluences the independent variable and alsoinfluences the dependent variable.

3 Researchers must always worry about extraneousvariables when they make conclusions about causeand S MichaelInternal & External Validity -6 Necessary Conditions for Causality Three conditions that are necessary to claim thatvariable A causes changes in variable B: Relationship condition: Variable A and variable Bmust be related. Temporal Antecedence condition: Proper time ordermust be established. Lack of Alternative Explanation Condition:Relationship between variable A and variable Bmust not be attributable to a confounding,extraneous S MichaelInternal & External Validity -7 Example:Necessary Conditions for Causality A correlation (aka, relationship) exists between coffee drinkingand the likelihood of having a heart attack.

4 Are we justified in concluding that coffee drinkingcausesheartattacks? Cigarette smoking is related to both of these variables. Individualswho drink little coffee are less likely to smoke cigarettes than arepeople who drink a lot of coffee. The observed relationship between coffee drinking and heartattacks might be the result of the third variable of smoking. A researcher must control the effect of smoking in order todetermine if this rival explanation accounts for the S MichaelInternal & External Validity -8 Threats to Internal & External Validity The controlled or experimental design enables theinvestigator to control for Threats to Internal andexternal Validity . Threats tointernalvalidity compromise our confidencein saying that a relationship exists between theindependent and dependent variables.

5 Threats toexternalvalidity compromise ourconfidence in stating whether the study s results areapplicable to other S MichaelInternal & External Validity -9 Threats to Internal Validity (a) History:Did some unanticipated event occurwhile the experiment was in progress and didthese events affect the dependent variable? History is a threat for the one group designbut not for the two group design. In the one group pre-post test design, theeffect of the treatment is the difference inthe pre-test and post-test scores. Thisdifference may be due to the treatment orto S MichaelInternal & External Validity -10 Threats to Internal Validity (b) History: Is not a threat for the two group(treatment/experimental andcomparison/control) design because thecomparison is between the treatmentgroup and the comparison group.

6 If the history threat occurs for both groups,the difference between the two groups willnot be due to the history S MichaelInternal & External Validity -11 Threats to Internal Validity (c) Maturation:Were changes in the dependentvariable due to normal developmentalprocesses operating within the subject as afunction of time? Is a threat to for the one group design. Is not a threat to the two group design,assuming that participants in both groupschange ( mature )at same S MichaelInternal & External Validity -12 Examples: Threats to Internal Validity (d) History:In a short experiment designed toinvestigate the effect of computer-basedinstruction, Ss missed some instructionbecause of a power failure at the school.

7 Maturation:The performance of first gradersin a learning experiment begins decreasingafter 45 minutes because of S MichaelInternal & External Validity -13 Threats to Internal Validity (e) Statistical regression:An effect that is theresult of a tendency for subjects selected onthe bases of extreme scores to regresstowards the mean on subsequent tests. When measurement of the dependentvariable is not perfectly reliable, there is atendency for extreme scores to regress ormove toward the mean. The amount of statistical regression isinversely related to the reliability of the S MichaelInternal & External Validity -14 Examples: Threats to Internal Validity (f) Statistical regression:In an experimentinvolving reading instruction, subjectsgrouped because of poor pre-test readingscores show considerably greater gain thando the groups who scored average and highon the S MichaelInternal & External Validity -15 Threats to Internal Validity (g) Selection:Refers to selecting participants for thevarious groups in the study.

8 Are the groupsequivalent at the beginning of the study? If subjects were selected by random sampling andrandom assignment, all had equal chance of being intreatment or comparison groups, and the groups areequivalent. Were subjects self-selected into experimental andcomparison groups? This could affect the dependentvariable. Selection is not a threat for the one group design butit is a threat for the two group S MichaelInternal & External Validity -16 Threats to Internal Validity (h) Experimental Mortality:Differential loss ofparticipants across groups. Did some participants drop out? Did this affect theresults? Did about the same number of participants makeit through the entire study in both experimentaland comparison groups?

9 Is a threat for any design with more than S MichaelInternal & External Validity -17 Threats to Internal Validity (i) Testing:Did the pre-test affect the scores on thepost-test? A pre-test may sensitize participant inunanticipated ways and their performance on thepost-test may be due to the pre-test, not to thetreatment, or, more likely, and interaction of thepre-test and treatment. Is a threat to the one group design. Not a threat to the two group design. Both groupsare exposed to the pre-test and so the differencebetween groups is not due to S MichaelInternal & External Validity -18 Examples: Threats to Internal Validity (j) Selection:The experimental group in aninstructional experiment consisted of a high-abilityclass, while the comparison group was an average-ability class.

10 Experimental Mortality:In a health experimentdesigned to determine the effect of variousexercises, those subjects who find the exercise mostdifficult stop participating. Testing:In an experiment in which performance on alogical reasoning test is the dependent variable, apre-test cues the subjects about the S MichaelInternal & External Validity -19 Threats to Internal Validity (k) Instrumentation:Did any change occur during thestudy in the way the dependent variable wasmeasured? (Is a threat to the one group design; notto the two group design. Why?) Design contamination:Did the comparison groupknow (or find out) about the experimental group? Dideither group have a reason to want to make theresearch succeed or fail?