Economics of Education Review

1 The impact of online learning on students course outcomes:Evidence from a large community and technical collegesystemDi Xu*, Shanna Smith JaggarsCommunity College Research Center, Teachers College, Columbia University, Box 174, 525 West 120th Street, New York, NY 10027,United States1. IntroductionFor two decades, state financing of higher Education hasbeen on the decline (Kane, Orszag, & Gunter, 2003). Publicpostsecondary institutions have responded by raisingtuition, increasing class sizes, cutting programs, andotherwise seeking to reduce costs and improve the same time, colleges have sharply increased theirdistance Education offerings through online coursework though often with an intent to improve access andconvenience for students rather than to reduce costs. Inthe wake of the recent recession, policy leaders in severalstates, assuming that online courses must be more cost-effective than face-to-face courses, have championedfurther expansions in online learning ( , Chen, 2012;Fain & Rivard, 2013; Texas Higher Education CoordinatingBoard, 2011).

2 The notion that online courses are more cost-effective than traditional, face-to-face courses is predicat-ed on two assumptions: first , that online course sectionsare consistently less expensive; and second, that they yieldfairly comparable student it may seem self-evident that online coursesare consistently cheaper than face-to-face courses, there issurprisingly little evidence on online and face-to-facecourse costs. Most research on the topic is dated ( ,Hawkes & Cambre, 2000; Jewett, 2000; Jung, 2003; Levine& Sun, 2002; Rogers, 2001; Virginia Community CollegeSystem, 2001; Whalen & Wright, 1999), and the conclu-sions drawn from relevant studies are mixed. Rumble(2003) discussed the complexities involved in makinggeneralizations about costs across different types ofcourses and institutions and concluded that there can beno clear-cut answer as to whether online courses areEconomics of Education Review 37 (2013) 46 57A R T I C L E I N F OArticle history:Received 27 November 2012 Received in revised form 1 August 2013 Accepted 5 August 2013 JEL classification:I23I28I21 Keywords.

3 Online learningCommunity collegesStudent performanceInstrumental variable analysisA B S T R A C TUsing a large administrative dataset from a statewide system including 34 community andtechnical colleges, the authors employed an instrumental variable technique to estimatethe impact of online versus face-to-face course delivery on student course travel distance between each student s home and college campus served as aninstrument for the likelihood of enrolling in an online section of a given course. In addition,college-by-course fixed effects controlled for within- and between-course selection yield robust negative estimates for online learning in terms of both coursepersistence and course grade, contradicting the notion that there is no significantdifference between online and face-to-face student outcomes at least within thecommunity college setting. Accordingly, both two-year and four-year colleges may wish tofocus on evaluating and improving the quality of online coursework before engaging infurther expansions of online learning.

4 2013 Elsevier Ltd. All rights reserved.* Corresponding author. Tel.: +1 212 678 3044; fax: +1 212 678 addresses: Xu), ( Jaggars).Contents lists available at ScienceDirectEconomics of Education Reviewjo u rn al h om epag e: ww o m/lo c at e/eco n ed ur ev0272-7757/$ see front matter 2013 Elsevier Ltd. All rights cheaper. Schiffman (2005) noted that developmentcosts for online courses varied across institutions from$10,000 to $60,000 per course. Based on interviews withpresidents, provosts, and other senior academic leaders atmore than 25 higher Education institutions,1 Bacow,Bowen, Guthrie, Lack, and Long (2012) reported that mostinstitutions provided distance Education to better servestudent needs rather than to save on costs. In fact, manyinterviewees believed that online courses were at least asexpensive as traditional courses, not only due to theirsubstantial start-up costs ( , investments in technology,course design, and instructor training) but also due torecurring costs ( , those resulting from increasedcoordination demands and technical support).

5 Moreover,studies of online course costs have not taken into accountthe quality or effectiveness of the courses examined, and itis possible that online courses with high completion ratesand strong learning outcomes require substantial invest-ments to design and second assumption underlying the cost-effective-ness argument that online courses produce studentoutcomes comparable to those produced by face-to-facecourses is also based on relatively weak dozens of studies have compared studentperformance between online and face-to-face courses,most have been descriptive studies, with no controls forstudent self-selection. Moreover, the majority have fo-cused on populations ( , K-12 students) or contexts ( ,hour-long educational modules) that are not relevant tothe typical online college course. Only a few random-assignment or quasi-experimental studies have focused onsemester-length college courses (Caldwell, 2006; Cavus &Ibrahim, 2007; Coates, Humphreys, Kane, & Vachris, 2004;Figlio, Rush, & Lin, 2010; LaRose, Gregg, & Eastin, 1998;Mentzer, Cryan, & Teclehaimanot, 2007; Odell, Abbitt,Amos, & Davis, 1999; Peterson & Bond, 2004; Schoenfeld-Tacher, McConnell, & Graham, 2001).

6 Results of thesestudies are mixed, leading many college leaders toconclude that online learning at least does no harm . However, two considerations limit the usefulness of , nearly all previous studies have focused onlearning outcomes among students who completed thecourse, and thus have disregarded the potential impact ofonline delivery on course withdrawal. Ignoring coursewithdrawal may be reasonable within the context ofselective four-year institutions, which typically have lowcourse withdrawal rates. In the community collegecontext, however, descriptive studies have typicallyreported course withdrawal rates in the 20 30% range,with higher withdrawal rates for online courses (Beatty-Guenter, 2002; Carr, 2000; Chambers, 2002; Moore,Bartkovich, Fetzner, & Ison, 2003). Course persistenceand completion is a particularly important issue incommunity colleges, where most students are low-income, many are working or have dependents, and fewcan readily afford the time or money required to retake acourse they did not successfully complete the first time(Adelman, 2005; Bailey & Morest, 2006; Planty et al.)

7 , 2009).Thus, studies that focus solely on course completers areminimally helpful to community college administratorscontemplating the potential costs and benefits of expand-ing online course , it is unclear whether the sets of coursesexamined in previous research represent the larger bodyof online courses available in the postsecondary setting,and particularly in the community college setting. Eachstudy in the literature tends to focus on one or two specificcourses, which in some cases are selected because they arethought to represent high-quality examples of onlinecoursework. Moreover, each course included in therigorous research cited above was conducted within aselective college or university (Jaggars & Bailey, 2010) institutions that are not representative of the less-selectiveor open-access colleges that make up the bulk of thenation s postsecondary sector. Qualitative research con-ducted in the community college setting has revealed thatmost online instructors simply convert their face-to-faceinstructional materials to printed handouts and text-heavyslide presentations, with few of the interactive technolo-gies that may effectively engage students in onlinelearning (Cox, 2006; Edgecombe, Barragan, & Rucks-Ahidiana, 2013).

8 Although no parallel studies have beenconducted in the four-year sector, these findings raise thequestion of how high-quality the typical or average online college course may order to understand student performance in thetypical online course within a given sector, it would bemost useful to compare a large and representative set ofonline courses against a similar set of face-to-face far, only one study has done so: Using a datasetincluding hundreds of course sections from 23 colleges inVirginia s community college system, Xu and Jaggars(2011) found that students fared significantly worse inonline courses in terms of both course persistence and end-of-course grades. However, the study was limited to entry-level English and math courses in community colleges inone state, raising the question of whether the results applyto other academic subjects and other state , although Xu and Jaggars controlled for a widearray of student, course, and institutional characteristicsusing multilevel propensity score matching, they could notcontrol for unobserved influences on students courseselection, such as employment status, actual workinghours, educational motivation, and academic , the results could have remained subject to selectionbias.

9 Indeed, using an endogenous switching model, Coateset al. (2004) found that online students tended to have higher levels of unobservable ability that improves theirperformance under both types of instruction (p. 543).Thus, failure to account for unobservables underlyingstudent self-selection may underestimate any negativeimpacts of the online format on student course paper builds on Xu and Jaggars (2011) study ofVirginia community colleges by focusing on a differentregion of the country and using an instrumental variable(IV) technique to control for unobserved confounding1 The institutions included public and private research universities,four-year colleges, and community Xu, Jaggars / Economics of Education Review 37 (2013) 46 57 47variables. Using a large administrative dataset fromWashington State s community and technical collegesystem, we used the distance from a student s home tocollege as an instrument for the likelihood of enrolling inan online rather than a face-to-face section of a givencourse.

10 We augmented the IV strategy using college-by-course fixed effects, which allowed us to compare studentswho took the same course but were enrolled in sectionswith different delivery formats, potentially controlling forbiases related to within- and between-course selection. Toassess the effects of taking a course online rather than face-to-face, we explored two course outcomes: (1) coursepersistence, or whether a student remained in the coursethrough the end of the semester; and (2) final course gradeamong those who persisted to the end of the course. Ouranalyses yielded robust estimates of negative impacts ofonline learning on both course persistence and coursegrade. Moreover, our IV estimates were consistentlystronger than the corresponding OLS estimates across allmodel specifications, lending support to the Coates et al.(2004) argument that students tend to be positivelyselected into online coursework, which may bias thenegative impacts of online learning toward zero whenstudent self-selection is not well Data and institutional characteristicsThe study used an administrative dataset of studentswho initially enrolled in one of Washington State s 34 two-year public community or technical colleges during the fallterm of 2004.