THE OUTCOMES AND IMPACTS - NCSALL

1 THE OUTCOMES AND IMPACTSOF ADULT LITERACY EDUCATIONIN THE UNITED STATESbyHal BederRutgers University, Graduate School of EducationNCSALL Reports #6 January 1999 The National Center for the Study of Adult Learning and LiteracyHarvard Graduate School of Education101 Nichols House, Appian WayCambridge, MA 02138 The work reported herein is supported by the Educational Research and Development CentersProgram, Award Number R309B60002, as administered by the Office of Educational Researchand Improvement/National Institute on Postsecondary Education, Libraries, and LifelongLearning, Department of Education, through contract to Harvard University. However, thecontents do not necessarily represent the positions or policies of the National Institute onPostsecondary Education, Libraries, and Lifelong Learning; the Office of Educational Researchand Improvement.

2 Or the Department of Education, and you should not assume endorsementby the Federal Report #6 January 19992 TABLE OF CONTENTSE xecutive SummaryPrefaceChapter 1: IntroductionFraming the IssueAccountabilityProgram Improvement and PlanningGoalsMeasurementCapacityJudgment Chapter 2: MethodologyGoalsResearch StrategyChapter 3: AnalysisNational StudiesNational Evaluation of Adult EducationPrograms (NEAEP), 1990-1994 The 1980 National Evaluation: An Assessmentof the State-Administered Program of theAdult Education ActA Longitudinal Evaluation of the Adult BasicEducation Program, 1973 The 1968 EvaluationState StudiesThe Washington Workforce Training StudyThe Tennessee Longitudinal StudyThe California Adult Learner ProgressEvaluation (CALPEP)The New Jersey StudyThe Utah StudyThe Maryland StudyThe 1980 Tennessee StudyThe Ohio StudyThe Wisconsin StudyWelfare StudiesThe California GAIN StudyThe Texas JOBS Program EvaluationSteps to SuccessWorkplace LiteracyMikulecky and LloydManufacturing and Financial Services inthe Chicago AreaWorkplace Literacy across the Three Phases of Textile Manufacturing Wisconsin Workplace PartnershipTraining Program 4 6 7 8 8 8 9911121313131515152123262931343436394142 4345484851535555575859 NCSALL Report #6 January 19993 Central Labor Council and the Consortiumfor Worker EducationFamily LiteracyThe National Even Start ProgramThe National Center for Family LiteracyChapter 4.

3 ConclusionsHow Effective is the Adult Literacy Education Program?EmploymentBetter Jobs for Those Who Are EmployedIncreased EarningsContinued EducationTermination/Reduction of Welfare DependenceSelf-Reported Gains in Basic SkillsTested Basic Skills Learning GainGED AcquisitionSelf-imageImpact on Children s EducationAttainment of Personal GoalsConceptual, Design, and Methodological ProblemsInaccurate and Incomplete DataSelf-report DataLack of ControlsValid and Reliable TestsPoor Quality ReportsLack of StandardsImplications for PolicyFinding the AnswerEconomic ImpactBasic SkillsSelf-imageChildren s EducationPersonal Goal AttainmentAccountability Standards and JudgmentsRecommendationsAn Effective outcome and ImpactReporting SystemA National Longitudinal AssessmentImproved State and Local outcome StudiesReferences61636366686870717171717 2727273737374757576767677777777787878797 97980 NCSALL Report #6 January 19994 EXECUTIVE SUMMARYThis study investigated the OUTCOMES and IMPACTS of adult literacy educationthrough a qualitative assessment of the OUTCOMES and IMPACTS research conductedsince the late 1960s.

4 OUTCOMES are the changes in learners that occur as a result oftheir participation in adult literacy education. IMPACTS are the changes that occur inthe family, community, and larger society as a consequence of goals of the study were to make reasoned inferences about the effectiveness of adult literacyeducation in the United States; to identify common conceptual, design, and methodological problemsinherent in the outcome studies conducted; to raise and discuss issues for policy; and to makerecommendations for the design and conduct of future OUTCOMES a comprehensive literature search, approximately 115 OUTCOMES and IMPACTS studies wereidentified. All were obtained in either hard copy or microfiche and the 68 of those that were found to havean OUTCOMES component were abstracted. Each study was then evaluated according to the following criteria:the study included an outcome / impact component; the report was adequately documented with respect todesign and methods; there was an adequate number of cases; the sampling plan was adequate; datacollection procedures were adequate ( , were not tainted by substantial attrition or biased by otherfactors); objective measures, rather than self-report, were used to measure OUTCOMES ; measures, especiallytests, were valid and reliable; the research design included a control or comparison group; and inferenceslogically followed from the design and on this evaluation, 23 studies were selected as being the most credible, and case studieswere prepared for each.

5 Studies are presented in five categories: national, state-level, welfare, familyliteracy, and workplace literacy. From the 23 studies, inferences about program effectiveness are EffectivenessThe 23 case studies represented evidence rather than proof of impact , and,like evidence in a trial, their findings were weighed in order to reach reasonableconclusions. Weighing had two dimensions. The first was the extent to which thevarious studies converged or diverged in respect to their findings on specificoutcome/ impact variables. Consensus across studies pointed towardeffectiveness/ineffectiveness, while lack of consensus suggested an inconclusiveresolution. The second dimension was the credibility of the individual studies. Whenarriving at conclusions, more credible studies were weighed more heavily than lesscredible conclusions set forth are deemed to be reasonable inferences from the findings reported in thecase studies.

6 They do not represent proof. Indeed, it is unlikely that any conceivable study or studies couldarrive at certainty. Table 1 presents the data used for this interpreting Table 1 and the conclusions made from it, three caveats are in order. First, thevariables included are those studied by a sufficiently large number of studies to enable reasonableconclusions. However, variable definitions and their units of measure vary among studies. In some studies,for example, learning gain is measured by the CASAS, while in others the TALS or TABE are , if a given study reported a gain, the gain is listed as positive (y) in the table irrespective of the sizeof the gain or the quality of the study s methodology. In some cases the gains reported as positive are quitesmall, and in some cases the limitations of the study render claims of gains suspect.

7 Third, the totals areaggregates of studies conducted at different times and on different populations of adult literacy learners,welfare clients and employees being examples. Drawing conclusions from such aggregates presumes thatdoing so is both valid and Report #6 January 19995 From the case studies, as summarized in Table 1, the following conclusions were made about theeffectiveness of the adult literacy education program in the United States:1. In general, it is likely that participants in adult literacy education receive gains in In general, participants in adult literacy education believe their jobs improve over time. However, thereis insufficient evidence to conclude that participation in adult literacy education causes In general, it is likely that participation in adult literacy education results in earnings In general, adult literacy education has a positive influence on participants continued Although the evidence suggests that participants in welfare-sponsored ( JOBS Program)

8 Adultliteracy education do experience a reduction in welfare dependence, the evidence is inconclusive as towhether adult literacy education in general reduces welfare dependence for Learners perceive that participation in adult literacy education improves their skills in reading, writing,and As measured by tests, the evidence is insufficient to determine whether or not participants in adultliteracy education gain in basic In general, adult literacy education provides gains in GED acquisition for participants entering at theadult secondary (ASE) Participation in adult literacy has a positive impact on learners According to learners self-reports, participation in adult literacy education has a positive impact onparents involvement in their children s Learners perceive that their personal goals are achieved through participation in adult the final chapter, conceptual, design, and methodological problems inherent in thestudies are discussed and implications for policy are presented, includingrecommendations for:1.

9 Relevant and measurable outcome standards and a feasible impact reporting system;2. a comprehensive national longitudinal evaluation measuring long-term impact ; and3. systematic funding and improvement of state and local outcome Report #6 January 19996 PREFACEThis report has four chapters. The first, the Introduction, frames the issues that provide focus tothe report. The second describes the study s methodology. The third chapter, which is by far the longest,presents case studies of 23 studies of OUTCOMES and IMPACTS . The final chapter presents conclusions,implications, and case studies presented in Chapter 3 represent the data upon which the study s conclusions arebased. While from a researcher s perspective it is critical that they be included in the report, it isrecognized that this material may be of more interest to researchers than to many policy makers andpractitioners.

10 For those who are less interested in the case studies, it is recommended that Chapter 3 beread Report #6 January 19997 THE OUTCOMES AND IMPACTS OF ADULT LITERACY EDUCATIONIN THE UNITED STATESCHAPTER 1: INTRODUCTIONFRAMING THE ISSUEThis paper examines the OUTCOMES and IMPACTS of adult literacy education in the United this examination can begin in earnest, however, it is important to frame the issue. What areoutcomes and IMPACTS ? Why is it important to examine them? OUTCOMES are the changes that take place in learners as a result of their participation in adultliteracy education. OUTCOMES imply cause and effect participation in adult literacy education is the cause;measurable changes in knowledge, skills, attitudes, and behavior are the effects.