Writing reviews for systems conferences

1 Writing reviews for systems conferencesTimothy RoscoeETH Z urichMarch 20071 IntroductionThese notes arose from a conversation with Rebecca Isaacs about how to review asked me to write some notes that the 2007 SOSP Shadow Program Committeemight find is a bit presumptuous to claim to know how to write a good review for a systems con-ference. In particular, I m inherently unqualified to say whether I write good reviewsor not, other than that I continue to be asked to be on the occasional PC. I think I writebetter reviews now than I did when I was younger and less experienced, and some ofthe mistakes I ve tried to avoid in recent years are tackled , these notes should be read as, firstly, a description of how I aspire towrite reviews , and secondly, as a discussion of the kind of review I would like to receivefrom a PC.

2 Although they re written in an imperative style, they re not intended to What s a review for?A paper review actually serves a remarkable number of different purposes, though it srarely mentioned. Here are some examples: A review serves as a record which partially documents and justifies the commit-tee s decision to accept or reject the paper, and its reasonsfor doing do. A review provides feedback to the authors on how to improve the paper or oth-erwise proceed with their work, regardless of whether it s accepted or not.

3 A review communicates your thoughts about the paper to otherPC usually isn t time to talk about all the details of a review in the PC meet-ing itself, and reading fellow PC members reviews is an important time-savingtechnique. Typically, one reviewer will be called upon to summarize the paper,all the paper s reviews , and their personal opinion in the meeting, and otherschime in with other thoughts. A review is a chance to get your own thoughts on the paper straight by writingthem down. It s surprising how your opinion of a paper can change by being1forced to explain it.

4 I ve sometimes found a paper utterly repugnant on first andsecond reading, but in Writing down why I hated it, found it was mostly my ownprejudice and actually got to quite like it. Conversely, I ve found trying to writinga positive review for a paper I thought I liked turned me off it. People who havewritten down a review of a paper are generally better qualified to assess it thanthose who have merely read the that a review has several audiences: the paper authors,your fellow PC members(your peers in the research community), and, finally, Steps in reviewingEveryone has their own methodology for reviewing a dauntingpile of papers.

5 I do thefollowing, which is probably no better or worse than many, but works for me: As soon as you get the assignments, print them all out, and write the paper num-bers on the first page of the paper. If you re a real workaholic, write the con-ference name as well so you don t get mixed up between the manyPCs you recurrently serving on. Preferably (if you have lots of time before the reviewing deadline), read each onein turn quickly from start to finish to get a general sense of what it s like. I tend to process my papers in numerical order - I ve got to read them all anyway,so there s no point in cherry-picking.

6 Since paper numbers are often assigned se-quentially in submission order, often the first papers are submitted very early andcan be quite random, so don t get dispirited. The middle bunch are often quitesolid, and the final ones can sometimes be excellent, but can often be hurried aswell. This isn t the case where pre-registration of abstracts is required (as withSOSP, for example).If you like, randomize the paper order now, but then retain this order. I don t dothis because sooner or later I ll drop them on the floor and mixthem all up.

7 Starting as soon as possible, read each paper in turn carefully, scribbling noteson them in the margins with whatever comments you think of at the time. I tendto carry a bunch of papers around with me instead of a novel, and mark them upwhile on the train, plane, or sitting at home. You can do this in a fairly leisurelyfashion (in fact, it s best to do this while you re relaxed),and if you start earlythere won t be much of a feeling of time pressure. If you need to look up somereference or do some other research at this stage ( I m sure Multics didsomething like ), don t bother but just note it down.

8 It s better to keepmaking progress than get derailed into reading more papers that have alreadybeen published. Finally, start at the beginning of the pile of papers and typeup a full review ofeach paper (see below). By now, you will have read all the papers assigned toyou at least once (perhaps twice), and have a pretty good ideahow good the fieldis. The papers will also have steeped a bit in your subconscious for a while, andyour rage at reading the ones that don t cite your work (or, worse, that scoopit) will have subsided a bit, enabling you to think a bit more clearly and write2a more reasonable and constructive review .

9 This is the time to go off and lookup references that you might need to put in your review . If you re on a plane orotherwise separated from the Internet or your trusty copy ofOrganick, then skipthe paper and move on, and come back to it when you re better connected. Submit the reviews as you ve done them. The chair(s) of the conference willthank you for this, because they can see progress being frequentlyagonize over whether they will have enough reviews by the PC you can see, I personally tend to thread paper reviewing through the gaps in therest of my so-called life, though in practice the final Writing stage is often an intensivesession in front of the computer.

10 For this style of working, the offline review interfaceof many online review packages is really on a PC can be a serious time commitment. The time you spend on an in-dividual paper varies considerably based on the paper length and quality (very bad orvery good papers take less time, those in the middle take longer). In total, though, youshould expect to devote several days of your time (spread outover reviewing period) toreviewing papers for a good Structure of a reviewReviewing forms can have varying degrees of structure, according to the taste of theprogram chair, or his or her motivation to wrestle with configuring the review , it s a good idea to structure your review irrespective of whether the reviewform does this for you.