How Video Production Affects Student Engagement: An ...

1 How Video Production Affects Student Engagement: An Empirical Study of MOOC VideosPhilip J. GuoMIT CSAIL / University of KimMIT are a widely-used kind of resource for online learn-ing. This paper presents an empirical study of how videoproduction decisions affect Student engagement in online ed-ucational videos. To our knowledge, ours is the largest-scalestudy of Video engagement to date, using data from mil-lion Video watching sessions across four courses on the edXMOOC platform. We measure engagement by how long stu-dents are watching each Video , and whether they attempt toanswer post- Video assessment main findings are that shorter videos are much more en-gaging, that informal talking-head videos are more engaging,that Khan-style tablet drawings are more engaging, that evenhigh-quality pre-recorded classroom lectures might not makefor engaging online videos, and that students engage differ-ently with lecture and tutorial upon these quantitative findings and qualitative in-sights from interviews with edX staff.

2 We developed a setof recommendations to help instructors and Video producerstake better advantage of the online Video KeywordsVideo engagement; online education; MOOCACM Classification Information Interfaces and Presentation ( HCI):Multimedia Information SystemsINTRODUCTIONE ducators have been recording instructional videos for nearlyas long as the format has existed. In the past decade, though,free online Video hosting services such as YouTube have en-abled people to disseminate instructional videos at scale. Forexample, Khan Academy videos have been viewed over 300million times on YouTube [1].Videos are central to the Student learning experience in thecurrent generation of MOOCs from providers such as Cour-sera, edX, and Udacity (sometimes called xMOOCs [7]).

3 These online courses are mostly organized as sequences ofinstructor-produced videos interspersed with other resourcesPermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee.

4 Request permissions from @S 2014, March 4 5, 2014, Atlanta, Georgia, is held by the owner/author(s). Publication rights licensed to 978-1-4503-2669-8/14 $ 1. Video Production style often Affects Student engagement inMOOCs. Typical styles include: a.) classroom lecture, b.) talkinghead shot of an instructor at a desk, c.) digital tablet drawing formatpopularized by Khan Academy, and d.) PowerPoint slide as assessment problems and interactive demos. A studyof the first edX course ( , Circuits and Electronics)found that students spent the majority of their time watch-ing videos [2, 13].

5 Also, a study of three Coursera coursesfound that many students are auditors who engage primarilywith videos while skipping over assessment problems, onlinediscussions, and other interactive course components [9].Due to the importance of Video content in MOOCs, videoproduction staff and instructional designers spend consider-able time and money producing these videos, which are oftenfilmed in diverse styles (see Figure 1). From our discussionswith staff at edX, we learned that one of their most pressingquestions was:Which kinds of videos lead to the best stu-dent learning outcomes in a MOOC?A related question thataffects the rate at which new courses can be added is howto maximize Student learning while keeping Video productiontime and financial costs at reasonable a step toward this goal, this paper presents an empiricalstudy of students engagementwith MOOC videos, as mea-sured by how long students are watching each Video , andwhether they attempt to answer post- Video assessment prob-lems.

6 We choose to study engagement because it is a neces-sary (but not sufficient) prerequisite for learning, and becauseit can be quantified by retrospectively mining user interactionlogs from past MOOC offerings. Also, Video engagementis important even beyond education. For instance, commer-cial Video hosting providers such as YouTube and Wistia useengagement as a key metric for viewer satisfaction [6, 16],which directly drives videos are much more heavily in pre- Production lesson planning tosegment videos into chunks shorter than 6 that intersperse an instructor s talking headInvest in post- Production editing to display thewith slides are more engaging than slides s head at opportune times in the produced with a more personal feel couldTry filming in an informal setting.

7 It might not bebe more engaging than high-fidelity studio to invest in big-budget studio tablet drawing tutorials are moreIntroduce motion and continuous visual flow intoengaging than PowerPoint slides or code , along with extemporaneous high quality pre-recorded classroom lecturesIf instructors insist on recording classroom lectures,are not as engaging when chopped up for a should still plan with the MOOC format in where instructors speak fairly fast andCoach instructors to bring out their enthusiasm andwith high enthusiasm are more that they do not need to purposely slow engage differently with lectureFor lectures, focus more on the first-watch experience;and tutorial videosfor tutorials, add support for rewatching and 1.

8 Summary of the main findings and Video Production recommendations that we present in this importance of scale: MOOC Video producers currentlybase their Production decisions on anecdotes, folk wisdom,and best practices distilled from studies with at most dozensof subjects and hundreds of Video watching sessions. Thescale of data from MOOC interaction logs hundreds ofthousands of students from around the world and millions ofvideo watching sessions is four orders of magnitude largerthan those available in prior studies [11, 15].Such scale enables us to corroborate traditional Video engage-ment research and extend their relevance to a modern onlinecontext.

9 It also allows MOOC Video producers to make morerigorous decisions based on data rather than just , it could enable our findings and recommendations togeneralize beyond MOOCs to other sorts of informal onlinelearning that occurs when, say, hundreds of millions of peoplewatch YouTube how-to videos on topics ranging from cook-ing to paper makes two main contributions: Findingsfrom an empirical study of MOOC Video engage-ment, combining data analysis of million Video watch-ing sessions in four edX courses with interviews with sixedX Production staff. The left column of Table 1 summa-rizes our seven main findings. To our knowledge, ours isthe largest-scale study of Video engagement to date.

10 Recommendationsfor instructional designers and videoproducers, based on our study s findings (see the right col-umn of Table 1). Staff at edX are already starting to usesome of these recommendations to nudge professors to-ward cost-effective Video Production techniques that leadto greater Student WORKTo our knowledge, our study is the first to correlate videoproduction style with engagement at scale using millions ofviewing closest related work is by Cross et al., who studied someof these effects in a controlled experiment [4]. They createdKhan-style (tablet drawing) and PowerPoint slide versions ofthree Video lectures and surveyed 150 people online abouttheir preferences.