Writing Good Reviews

Reviewing papers is one of the most important roles of an academic. But yet, we provide surprisingly little guidance to our reviewers on the tone of their reviews. I tried to capture some of my thoughts in this email I sent to my reviewers when I ran RSS 2017.

Dear X,

Thank you, again, for serving as a reviewer for RSS 2017!

Reviewers are the life-blood of a conference. Your high quality reviews are critical not only for the authors but also for me and for the continued success of RSS. I know you have a lot of experience reviewing, so I will keep this very short and specific.

I’d like your reviews to be compassionate, constructive, and scholarly.

• Compassionate: invert your position and ask yourself how you’d feel if you received your review. If it makes your blood boil, take a break, and revise your review.
• Constructive: I’d like us to believe that every paper has a best paper award winner hidden in it. Your role is to guide the author to reveal it.
• Scholarly: cite your sources, do not make claims like “I do not feel this is true” [your feelings matter to me, but probably not to science]. Do not hawk your viewpoint. And please, absolutely no ad hominem attacks.

As a reviewer, your role is to comment on the paper’s quality to help the program committee reach an informed decision. So, please refrain from saying things like “This paper should be rejected from RSS” in the main body of the review. You’re welcome to add it to the private comments to the committee.

Finally, I am sure that if you try hard [or sometimes not that hard] to investigate the true identity of your authors, you would likely be able to do it [with high probability]. But, please do not use that as grounds for rejection. Instead of saying “I think that the author of this paper is Joe Biden, and this looks a lot like Joe’s 2015 paper”, just say “Here are the ways this paper is similar to Joe Biden’s 2015 paper”. In a similar vein, do not punish authors for not citing non peer reviewed content [like arXiv or workshop papers].

Your Area Chairs and I will be monitoring your reviews and providing you some feedback as this unfolds.

Happy reviewing!

Please contact me if you have any questions.

Thank you, again!

Sidd.

Conflicts of Interest

In addition to the usual conflict rules, I’d recommend another simple rule: if you feel like you cannot comment on the paper in an unbiased manner, decline to review it. I often find people vested in a research topic to be the worst reviewers, digging in stubbornly to their viewpoint and refusing to give other ideas a fair listen. The more knowledgable you are, the more enlightened you ought to be about other viewpoints.

Rapoport’s Rules of Criticism as stated by Dennet

Finally, I’d like to leave you with perhaps the most succinct summary on how to compose a critical commentary:

• You should attempt to re-express your target’s position so clearly, vividly, and fairly that your target says, “Thanks, I wish I’d thought of putting it that way.”

• You should list any points of agreement (especially if they are not matters of general or widespread agreement).

• You should mention anything you have learned from your target.

• Only then are you permitted to say so much as a word of rebuttal or criticism.