Understanding Understanding Source Code with …

1 Understanding Understanding Source code withFunctional magnetic resonance ImagingJanet Siegmund , Christian K stner , Sven Apel , Chris Parnin , Anja Bethmann ,Thomas Leich , Gunter Saake , and Andr Brechmann University of Passau, Germany Carnegie Mellon University, USA Georgia Institute of Technology, USA Leibniz Inst. for Neurobiology Magdeburg, Germany Metop Research Institute, Magdeburg, Germany University of Magdeburg, GermanyABSTRACTP rogram comprehension is an important cognitive process that in-herently eludes direct measurement. Thus, researchers are strug-gling with providing suitable programming languages, tools, orcoding conventions to support developers in their everyday this paper, we explore whetherfunctional magnetic resonanceimaging (fMRI), which is well established in cognitive neuroscience,is feasible to more directly measure program comprehension.

2 In acontrolled experiment, we observed 17 participants inside an fMRIscanner while they were comprehending short Source - code snip-pets, which we contrasted with locating syntax errors. We found aclear, distinct activation pattern of five brain regions, which are re-lated to working memory, attention, and language processing allprocesses that fit well to our Understanding of program comprehen-sion. Our results encourage us and, hopefully, other researchers touse fMRI in future studies to measure program comprehension and,in the long run, answer questions, such as: Can we predict whethersomeone will be an excellent programmer? How effective are newlanguages and tools for program Understanding ? How should wetrain developers?

3 1. INTRODUCTIONAs the world becomes increasingly dependent on the billionslines of code written by software developers, little comfort can betaken in the fact that we still have no fundamental Understanding ofhow developers understand Source program comprehension is not limited to theorybuilding, but can have real downstream effects in improving educa-tion, training, and the design and evaluation of tools and languagesfor programmers. If direct measures of cognitive effort and diffi-culty could be obtained and correlated with programming activity,then researchers could identify and quantify which types of activi-ties, segments of code , or kinds of problem solving are troublesomeor improved with the introduction of a new language or studying programmers, decades of psychological and observa-tional experiments have relied on indirect techniques.

4 Such as com- This author published previous work as Janet to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a 20XX ACM X-XXXXX-XX-X/XX/XX ..$ Screen Head coil while .. That s a currValue = (a) Prestudy(b) fMRI measurement(c) Activation pattern(d) Cognitive proc.(e) Other fMRI studies(f) Interpretation forprogram comprehensionFigure 1: Workflow of our fMRI task performance or having programmers articulate theirthoughts in think-aloud protocols.

5 Each method, when skillfullyapplied, can yield important insights. However, these commontechniques are not without problems. In human studies of program-ming, individual [13] and task variance [18] in performance oftenmask any significant effects hoping to be found when evaluating,say, a new tool. Think-aloud protocols and surveys rely on self-reporting and require considerable manual transcription and analy-sis that garner valuable but indefinite and inconsistent the past few decades, psychologists and cognitive neurosci-entists have collectively embraced methods that measure physio-logical correlates of cognition as a standard practice. One suchmethod isfunctional magnetic resonance imaging (fMRI), a non-invasive means of measuring blood-oxygenation levels that changeas a result of localized brain this paper, we report on results and experience from applyingfMRI in a program-comprehension experiment.

6 While our experi-ment is a first step toward measuring program comprehension withfMRI, and as such inherently limited, we believe this study can il-luminate a path toward future studies that systematically explorehypotheses and that can be used to build stronger theories of pro-gram accepted at ICSE our experiment, 17 participants performed two kinds of taskswhile in an fMRI scanner. In the first kind (referred to ascompre-hension tasks), developers should comprehend code snippets andidentify the program s output. In the second kind (referred to assyntax tasks), developers identified syntax errors in code snippets,which is similar to the comprehension tasks, but does not requireactual Understanding of the program.

7 As a result of our study, wefound: evidence that distinct cognitive processes were used whenperforming the comprehension tasks, activation of functional areas related to working memory, at-tention, and language comprehension, and a left-hemisphere results provide direct evidence of the involvement of work-ing memory and language processing in program comprehension,and suggest that, while learning programming, training workingmemory (necessary for many cognitive tasks), and language skills(which Dijkstra already claimed as relevant for programming),might also be beneficial for programming skills. Furthermore, ourresults can help to validate or invalidate particular theories of pro-gram comprehension.

8 Although a single study is not sufficient toanswer general questions, we can raise some further probing ques-tions: If program comprehension is linked to language comprehen-sion, does learning and Understanding a programming language re-quire the same struggles and challenges as learning another spokenlanguage? If program comprehension only activates the left hemi-sphere (often referred to as analytical), can we derive guidelines onhow to train students?Taking a broader perspective, our study demonstrates the fea-sibility of using fMRI experiments in software engineering. Al-though we believe this is only a first step, our experience and ex-perimental design is meant to be a template for other researchers toadopt and improve.

9 with decreasing costs of fMRI studies, we be-lieve that such studies will become a standard tool also in software-engineering are still many interesting, unanswered questions that canfollow this line of research: How do people use domain knowledgeduring comprehension? To what extent is implementing sourcecode a creative process? Can we train anybody to become an excel-lent programmer? How should we design programming languagesand tools for optimal developer support? Can software metrics pre-dict the comprehensibility of Source code ?In summary, we make the following contributions: We designed the first fMRI study to observe brain activityduring comprehension tasks. We share our design and expe-riences.

10 We conducted the study with 17 participants and observedactivation in five distinct brain regions. This demonstratesthe potential of fMRI studies in software-engineering re-search. We interpret how the identified cognitive processes con-tribute to program comprehension and discuss future re-search FMRI STUDIES IN A NUTSHELLR ationale of fMRI first goal in fMRI studies isto locate and isolate brain activity associated with a behavior. Toidentify a region of the brain, scientists use instruments with highspatial precision, such as fMRI scanners. After having establisheda general idea of where brain activity is occurring, scientists furthertry identifying the timing and interaction of brain activity betweendifferent brain locations.