Validity - CAS

1 Research Skills for Psychology Majors: Everything You Need to Know to Get StartedValidityValidity, the deeper and more complicated twin of Reliability, is outlined in a cursory manner in this chapter. Validity is indeed a complex topic, so for a more complete treatment of the subject an advanced methods book should be consulted. You may have this opportunity in a later , like reliability, takes many forms and perhaps the term is used too broadly. Generally, Validity is used in two contexts: (1) evaluating the quality of a measure-ment instrument or method (a true twin of reliability); and (2) evaluating the qual-ity of a research study, especially an experiment (maybe a cousin).

2 This chapter will cover both uses of the term. These two sides of Validity have in common the question, is it really what it says it is? . In other words, Validity is an issue in inter-preting the real meaning of the research and focuses on the relationships among what the researcher had in mind from the start, what the researcher did and what 2003 W. K. Gabrenya Jr. Version: Contents of This ChapterValidity in for Squid and Anchovy in (Defined)..4 Threats to internal Effects and Demand Experiment (Reviewed)..6 Regression (Biased Sampling)..7 Carry-Over Loafing (Overview)..10 Page 2the outcome really means.

3 Reliability, while certainly necessary, is a less sweeping in MeasurementMeasurement takes many forms, and although the most common form of mea-surement in social and behavioral science seems to be the self-report survey instrument, we measure constructs in a variety of ways. For example, classic research on interpersonal aggression operationalizes aggression by measuring the number and intensity of electric shocks that a person gives to another. Effort in group process experiments is assessed by measuring how loudly a research subjects are willing to shout in a sound-proofed room. Attitudes can be measured by determining the diameter of the pupil (hole in the middle of your eye).

4 All such measures beg the question: what is really being measured? Construct ValidityThe question, what is really being measured? is the central issue of construct Validity . Of all types of measurement Validity , construct Validity is the most im-portant. A construct is a theoretical concept whose existence means something in the context of a hypothesis or theory. For example, when we study the effect of anger on aggression, anger and aggression are theoretical constructs that must be operationalized represented by something we can measure and quantify in order to be studied. Constructs, hypotheses, and operationalization are discussed in detail in chapters 5 and ValidityWhenever a measure is used in research, the first question is whether or not it is a valid representation of the construct.

5 Establishing the construct Validity of a measure can be very dif-ficult. The easiest way is to rely on face Validity : the measures seems valid on the face it, , it just looks right. For example, if we want to measure attitudes toward anchovy pizza, the author s favorite (but the squid is good too, if you can find it in America), we might ask,Anchovy pizza tastes (check one): Delicious ___ ___ ___ ___ ___ ___ ___ AwfulOften face-valid measures like this one have sufficient construct more complex method for assessing construct Validity is to compare the measure in question to other measures that are supposed to measure the same thing, and to measures that are not supposed to measure the same thing.

6 A valid measure should evidence high correlations with the former, and low correlations with the latter. For example, the anchovy pizza measure should show a moderate correlation with a measure Recipe for Squid and Anchovy PizzaRecipe for Squid and Anchovy Pizza (in Japanese and English): ~ PizzaPage 3of appreciation for seafood, one would suppose. Another approach is to admin-ister the measure to groups of people who, based on some other research, are expected ahead of time to differ in a certain direction on the measures. Italians like the author should appreciate anchovies and squid more than people from bor-ing northern places.

7 If the results don t come out as expected, the measure may have a ValidityContent Validity is closely related to construct Validity . Sometimes it is important that a measure assess a sufficiently broad or comprehensive range of the parts or components of a construct. Some constructs are very simple, like attitude toward anchovy pizza while others are complex and multifaceted, like knowledge of psychology research methods. To be valid, a measure of a complex construct must be sufficiently comprehensive, picking up most of its parts or components. This is another way of saying that sometimes a measure can t be considered to have construct Validity unless it meets this comprehensiveness criterion.

8 For example, a measure of a knowledge domain such as psychology research methods, would not have construct Validity unless its content were valid, , as-sessed a broad range of all the knowledge and skills involved in doing psychological research. A final exam in such a course that only focused on research design but ignored all the other aspects of methods would lack content Validity for lack of sufficient breadth. It would therefore lack construct Validity because the construct knowledge of research methods would not be adequately covered. Students refer to tests like this as stupid and unfair. Criterion ValidityAnother way to judge the Validity of a measure is whether it does a good job predicting something with which is ought to be related, termed a criterion.

9 A criterion can be just about anything that is, itself, high in construct Validity . For ex-ample, we might want to predict the starting salary of new psychology graduates, a variable that indicates success to many people. Starting salary is the criterion. The Validity of a measure can be assessed by how well it predicts this criterion. In this example, we might be interested in Grade Point Average as the predictor mea-sure. The criterion Validity of GPA is evaluated on the basis of how well it predicts salary just after graduation. Predicting a criterion from a previously measured variable like GPA is termed predictive Validity .

10 When the predictor and the criterion are assessed at the same time, the term concurrent Validity is used. Criterion Validity depends on the practical value of the measure more so than its theoretical value. In other words, the measure is only as good as its ability to predict something interesting or useful. Construct Validity may in fact be poor in a successful, criterion-valid measure because sometimes we don t have to know exactly what construct the measure is assessing for it to be useful in predicting a criterion. If animals howl more just before an earthquake, consistently, we need not know why, we just have to get out of the house with our clothes author s youthful attempts to build equations that predict the outcome of horse races is a discouraging case in point.