Reviewing Conference Papers - Mark Bernstein

1 !eviewing Conference PapersMar" Bernstein , Eastgate Systems + 2008, I was program chair for WikiSym, the ACM Symposium for wikis. I think I was asked to serve in this post because WikiSym is a relatively young Conference and I ve had a good deal of experience on rel!ted progr!m I chose the WikiSym program committee, I made a special effort to include an exceptional diversity of professional and scholarly accomplishment in order to reflect the diverse interests of the Wiki community. That meant the Program Committee included a number of business leaders and professionals who don t usually participate in academic conferences, much less on program committees. To help them, and to keep everyone on the same page, I wrote this discussion of comp"ter science, conferences !re o#en the prim!ry medi"m of scientific comm"nic!tion, !nd Conference s"bmissions !re reviewed !t le!st !s stringently !s jo"rn!l p!pers. Despite the import!

2 Nce of the process, not m"ch h!s been written on the the Purpose of !efereeingThe prim!ry d"ty of the progr!m committee is to ens"re the integrity !nd the reli!bility of the rese!rch liter!t"re. People who !ttend the Conference , or who cons"lt its Proceedings in the f"t"re, m"st be confident th!t the res"lts reported here are honest, accurate, and may be relied th!t this is q"ite different from the d"ty of ! commerci!l Conference s"ch !s TED or CES, which m"st consider first wh!t might best !ttr!ct !nd entert!in its c"stomers. We wo"ld very m"ch like to h!ve !n exciting !nd !ttr!ctive progr!m, b"t it is m"ch more import!nt, for "s, th!t the rese!rch reported here be of the highest clear consequence is that we don t particularly care who wrote a paper or where they work. Some conferences pr!ctice blind r!vi!wing, withholding the n!me of !"thors from reviewers. I find this sometimes le!ds to "nprod"ctive spec"l!tion !nd th!t it is helpf"l in some c!

3 Ses to know who is writing, in order to "nderst!nd ex!ctly what they mean to say. DIFFERENCES FROM BOOK AND FILM REVIEWSN ewsp!per !nd m!g!zine critics help people spend their time efficiently, g"iding them tow!rd the most rew!rding books !nd films. Critics !lso help "s "nderst!nd how !rt f"nctions !nd g"ide cre!tors toward more useful and effective approaches. But most of all, critics sell Papers . The violence !nd vit"per!tion of newsp!per criticism is seldom productive FROM GRADING PAPERSC onference Reviewing has a certain pedagogical flavor. Many contributions are written, in whole or in part, by graduate students and postdocs. Others come from people new to the field. Part of the reviewer s role is to identify weaknesses in Papers that can be remedied through additional research or better writing, and also to indicate unproductive lines of work that are unlikely to yield useful the instructor s first job is to instruct the student.

4 The referee s job is to assemble the best available research, to show authors how it might be most effectually presented, and to help authors of unsatisfactory Papers to improve them or to understand why their approach needs to change. Copyright 2008 by Mark Bernstein . All Rights Not!bly the AC& Hypertext Conference on whose progr!m committee I ve served some seventeen times (incl"ding two stints !s co-ch!irConflicts of InterestBec!"se !ccept!nce or rejection of p!pers sometimes h!s signific!nt person!l or profession!l conseq"ences, it is import!nt th!t reviewers !void conflict of IS A CONFLICT?Yo" h!ve ! con'ict of interest if !n !"thor of ! p!per assigned to you is: !n employee of yo"r instit"tion or comp!ny (incl"ding s"bsidi!ries !nd other f!c"lties), or ! business partner ! coll!bor!tor or co-!"thor of ! book, ch!pter, or article within the past five years ! former doctor!l st"dent, s"pervisor, or direct report engaged in a professional rivalry or feud with you has close personal or family ties with youSho"ld yo" find yo"rself h!)

5 Ving ! con'ict of interest on ! p!per to which yo" h!ve been !ssigned, simply let me know. We h!ve ! l!rge !nd c!p!ble committee, !nd I sho"ld h!ve no tro"ble loc!ting !n alternate contrib"tions will be s"bmitted by members of the progr!m committee. The committee sho"ld t!ke c!re to ens"re th!t these p!pers !re neither f!vored nor disadvantaged by this Papers YOU FEEL UNABLE TO READ WITH INTELLIGENCE AND SYMPATHYIn assigning reviews, I ll do my best to take into account what I know of your interests and your background. If I have erred, and you find yourself assigned to review a paper that you really cannot read, please let me know and I ll arrange for a different !sion!lly, tho"gh, I m!y be seeking yo"r opinion specific!lly bec!"se ! p!per is not precisely "p yo"r alley. Obligations of a !eviewerSYMPATHYDo yo"r best to re!d p!pers with c!re !nd symp!thy. This is e!sy to do when the p!per is good, b"t is still desirable even when the paper is hours of work in some cases, years of work have gone into research and writing this paper .

6 If the paper is very bad indeed, this committee may be its only audience. Do your best to approach it with a broad interest and generous spirit. This does not !rg"e for lenience or l!xity, !nd cert!inly we c!nnot !ccept b!d p!pers or enco"r!ge defective research. A"thors !re most !pt to listen to yo"r reviews when they're tho"ghtf"l !nd constr"ctive r!ther th!n caustic and AN UNSYMPATHETIC COMMENT IS APPROPRIATEW hile cr"shing wit !nd spect!c"l!r den"nci!tion !re ! st!ple of the press critic, they !re "s"!lly "nhelpf"l in Conference Reviewing . An exception might be m!de, tho"gh, when the reviewer is cert!in th!t the !"thor is consist!ntl" eng!ged in !n erroneo"s co"rse, of which the p!per "nder review is merely the l!test ex!mple. Where ! mild correction might be too e!sily dismissed, emph!sis m!y !ttr!ct !ttention or provoke second is usually better to attempt this correction through private correspondence or, occasionally, through public discussion such as a Conference panel or a Q&A session.

7 But the anonymity of Conference Reviewing is, at times, an important safeguard of the reports !re !nonymo"s, !nd in writing them it is best to be co"rteo"s b"t fr!nk, c!ndid !nd direct. Reviews !re sent to the prim!ry !"thor of e!ch p!per, b"t the identity of the reviewer is not disclosed to few !c!demic reviewers prefer to sign their reviews; if yo" wish to do so, ple!se incl"de ! signature in the body of the review. Copyright 2008 by Mark Bernstein . All Rights pr!ctice, the !nonymity of reviews sho"ld not be too he!vily relied "pon. Altho"gh we t!ke c!re to !void !ccident!l disclos"re, mist!kes do h!ppen. &ore tellingly, !"thors m!y o#en be !ble to discern the identity of their reviewers thro"gh their interests, concerns, and TO THE COMMITTEEOcc!sion!lly, yo" m!y h!ve pertinent comments to m!ke to the committee which yo" do not wish to be bro"ght to the !ttention of the !"thors. A sep!r!te section of the review form is provided for this gener!

8 L, it is better to confine yo"r comments to those th!t c!n !ssist the !"thors in improving their work. Some "sef"l comments to the committee might be:I assume that equation 7, as derived in section , is correct. To be honest, I don t understand it, and I d be more comfortable, before the paper is accepted, if someone who remembers tensor analysis could check found the paper unobjectionable but extremely dull; accepting it will cause no harm, but rejecting it would not cause much harm, might not be clear !om the paper , which s"ms to have b"n adapted !om some sort of proposal, that the system h# actually b"n implemented. I have s"n it, and it $ actually quite prom$ing; if the paper cannot be salvaged, a demonstration would be very !pers s"bmitted to the Conference !re confidenti!l until they are not "se res"lts reported in these p!pers in yo"r work "ntil the p!per is p"blished, or comes to yo"r attention in some other way.

9 !ating PapersFor e!ch p!per, ple!se provide ! ! r!ting !nd ! re!son!bly complete comment. The comment is f!r more import!nt th!n the r!ting, b"t choosing the !ppropri!te r!ting will signific!ntly !ssist the : THEIR LIMITATIONSSome conferences pl!ce consider!ble weight on n"meric!l scores !nd r!ting. We won't do this; I find it leads to excessive emphasis on small some ratings have a specific meaning. The top r!ting is intended for very fine p!pers -- p!pers th!t !re !mong the best p!pers p"blished in the field, or th!t cont!in res"lts th!t dem!nd immediate publication. The lowest r!ting is me!nt for p!pers th!t !re !ctively pernicio"s. This wo"ld incl"de those th!t !re deceptive, d"plicito"s, pl!gi!rized, or fr!"d"lent. It would also include Papers that are simply former times (and still in a few conferences today), acceptance was contingent upon public endorsement of the work by a member of the committee. Imagine, if you would that you would be expected to stand at the side of the stage while the paper was presented.

10 You would rate a paper a "A" if you would be extremely proud to be associated, even in this small way, with the first presentation of this work; a D" would mean that you'd blush to have your friends a colleagues blame you for having to sit through additional marks are available to you to qualify your scores. An X represents exceptional confidence in your judgment, because the paper directly addresses topics of which you are particularly knowledgeable. A Z indicates that you mistrust your judgment in some degree, because aspects of the work lie outside the realms in which you feel : A remarkable result! One of my top students tried and failed to achieve th$ synthes$, and the route adopted here $ remarkably : The pattern language proposed in the conclusion will be influential and widely adopted; I have already begun to reconsider our firm s management practice. I admit, though, that I cannot follow the mathematics of Section and hope that another reader will check this section with : Prior to [Clump 96], it w# widely thought that wodgets would deteriorate in the field.