{"id":5563,"date":"2013-04-05T07:41:31","date_gmt":"2013-04-05T14:41:31","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=5563"},"modified":"2013-04-06T10:51:29","modified_gmt":"2013-04-06T17:51:29","slug":"anonymity","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=5563","title":{"rendered":"Anonymity"},"content":{"rendered":"
\n
The skilled adversarial reviewer can find reasons to reject any paper without even reading it. This is considered truly blind reviewing. [Cormode, G.<\/a>]<\/div>\n<\/blockquote>\n

Many conferences request that submitted papers be anonymized by removing the authors’ names, tweaking the references, removing mentions of the authors’ organization in the paper, etc. The goal of the double-blind review process is to reduce the bias (positive or negative) that reviewers might have based on their knowledge of who wrote the paper. SIGIR, for example, included the following<\/a> on their submission page for the 2013 conference:<\/p>\n

Anonymity. SIGIR reviewing is double-blind. Therefore, please anonymize your submission. This means that all submissions must contain no information identifying the author(s) or their organization(s): Do not put the author(s) names or affiliation(s) at the start of the paper, anonymize citations to and mentions of your own prior work that are directly related to your present work, and do not include funding or other acknowledgments. For example, if you are using your product that is well known in the domain and you think it will be easy for an expert to identify you or your company, we recommend that you use another name for your product (e.g., MyProduct_ABC, MyCompany_ABC). If your paper is accepted, then you will replace the original name in the final version for the proceedings.<\/p>\n

Papers that do not follow the above Style, Language, Anonymity instructions will be rejected without review. <\/strong>[emphasis mine]
\n<\/strong><\/p><\/blockquote>\n

And, apparently, in some cases, they followed through on this policy. In my opinion, this is too harsh.<\/p>\n

First, a desk rejection, given at the end of the review process, for a trivially-correctable error such as including the name of the company or product in the paper is just plain mean. It doesn’t serve the community, it doesn’t serve the authors, it doesn’t serve the SIGIR brand. It just serves bureaucracy. Furthermore, I am certain that this policy was not applied consistently, and that a number of papers (mine included) that were accepted for review did not meet the strict interpretation of this rule.<\/p>\n

Second, truly anonymizing self-references is just as likely to cause reviewers to complain about lack of relevant citations as it is to reduce the alleged bias. And if the reviewers do, in fact, know which paper was cited, then what’s the point of making it anonymous?<\/p>\n

Third, while in some cases it is certainly possible to obscure the authors’ names and institutional affiliation, in situations in which the submitted paper builds on prior work by that same group, it becomes very difficult to preserve anonymity and coherence at the same time. Sure, I can change the name of my system from Querium to Dhrevhz, but once I include a screenshot, any reviewer who has the background to review the paper will know which system is being described.<\/p>\n

The upshot of all of this is that true double-blind anonymization is difficult to achieve, and its necessity is unclear. Many conferences run review processes in which reviewers know the identity of the authors, and I haven’t seen any studies that attribute quality differences to this factor. Most journal reviewing also does not use double-blind anonymity and that process seems to work just fine. I think it’s time to stop worrying about anonymity and focus instead on more fundamental aspects such as\u00a0 novelty, creativity, thoroughness, etc.<\/p>\n","protected":false},"excerpt":{"rendered":"

The skilled adversarial reviewer can find reasons to reject any paper without even reading it. This is considered truly blind reviewing. [Cormode, G.] Many conferences request that submitted papers be anonymized by removing the authors’ names, tweaking the references, removing mentions of the authors’ organization in the paper, etc. The goal of the double-blind review […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/5563"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5563"}],"version-history":[{"count":5,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/5563\/revisions"}],"predecessor-version":[{"id":5565,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/5563\/revisions\/5565"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5563"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5563"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5563"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}