Writing Good Reviews

August 27, 2018

Reviewing papers is one of the most important roles of an academic. But yet, we provide surprisingly little guidance to our reviewers on the tone of their reviews. I tried to capture some of my thoughts in this email I sent to my reviewers when I ran RSS 2017.

Dear X,

Thank you, again, for serving as a reviewer for RSS 2017!

Reviewers are the life-blood of a conference. Your high quality reviews are critical not only for the authors but also for me and for the continued success of RSS. I know you have a lot of experience reviewing, so I will keep this very short and specific.

I’d like your reviews to be compassionate, constructive, and scholarly.

As a reviewer, your role is to comment on the paper’s quality to help the program committee reach an informed decision. So, please refrain from saying things like “This paper should be rejected from RSS” in the main body of the review. You’re welcome to add it to the private comments to the committee.

Finally, I am sure that if you try hard [or sometimes not that hard] to investigate the true identity of your authors, you would likely be able to do it [with high probability]. But, please do not use that as grounds for rejection. Instead of saying “I think that the author of this paper is Joe Biden, and this looks a lot like Joe’s 2015 paper”, just say “Here are the ways this paper is similar to Joe Biden’s 2015 paper”. In a similar vein, do not punish authors for not citing non peer reviewed content [like arXiv or workshop papers].

Your Area Chairs and I will be monitoring your reviews and providing you some feedback as this unfolds.

Happy reviewing!

Please contact me if you have any questions.

Thank you, again!

Sidd.

Conflicts of Interest

In addition to the usual conflict rules, I’d recommend another simple rule: if you feel like you cannot comment on the paper in an unbiased manner, decline to review it. I often find people vested in a research topic to be the worst reviewers, digging in stubbornly to their viewpoint and refusing to give other ideas a fair listen. The more knowledgable you are, the more enlightened you ought to be about other viewpoints.

Rapoport’s Rules of Criticism as stated by Dennet

Finally, I’d like to leave you with perhaps the most succinct summary on how to compose a critical commentary:

Writing Good Reviews - August 27, 2018 - Siddhartha Srinivasa