{"id":3427,"date":"2010-04-15T05:12:48","date_gmt":"2010-04-15T12:12:48","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=3427"},"modified":"2010-04-14T19:34:39","modified_gmt":"2010-04-15T02:34:39","slug":"3427","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=3427","title":{"rendered":"Reviewer Operating Characteristic"},"content":{"rendered":"

David Karger made an interesting proposal<\/a> on the Haystack blog about the efficiency of CHI reviewing. Using an ROC<\/a> analysis of reviewer scores for CHI 2010, he found that when there is consensus between the first two reviews that a paper in question scores below 2 of 5, that there is no need to solicit a third review for it.\u00a0 While this method would have caused the rejection of 6 of about 300\u00a0papers that weren’t actually rejected, it would save almost 500 reviews.<\/p>\n

The question is, is this tradeoff worth it?<\/p>\n

<\/p>\n

The answer is probably yes, particularly\u00a0if you consider the following points:<\/p>\n

    \n
  1. Having fewer papers to reviews should, on average produce better reviews by reducing the per-reviewer load and by decreasing the demand for reviewers (thus decreasing the number of poor reviewers)<\/li>\n
  2. Papers rejected based on two reviews are still entitled to a rebuttal, which would offer a chance to correct reviewing mistakes.<\/li>\n
  3. The associate chair responsible for the paper has the authority to request a third review even if the first two scores are low if the reviews are inadequate.<\/li>\n<\/ol>\n

    Another reason to believe this strategy will be successful is that it is already adopted by the SIGMOD conference. It would be interesting to obtain data from a couple of other (earlier) CHI\u00a0conferences to double-check the findings. If they hold up, this could be an easy fix to relieve pressure on an overloaded system.<\/p>\n

    Of\u00a0 course the adoption of this scheme does not preclude the incorporation of reviewer ratings<\/a>, the adoption of which should be used to improve the quality of reviews as well.<\/p>\n","protected":false},"excerpt":{"rendered":"

    David Karger made an interesting proposal on the Haystack blog about the efficiency of CHI reviewing. Using an ROC analysis of reviewer scores for CHI 2010, he found that when there is consensus between the first two reviews that a paper in question scores below 2 of 5, that there is no need to solicit […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[105],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/3427"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3427"}],"version-history":[{"count":8,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/3427\/revisions"}],"predecessor-version":[{"id":3435,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/3427\/revisions\/3435"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3427"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3427"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3427"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}