I’ve written about some alternatives to the current review process, and I believe one of ways in which the current process can be improved is by formal recognition of reviewers’ efforts. While many conferences and journals acknowledge reviewers by publishing their names, this does not reflect the quality of the effort put in by some reviewers. A more lasting and public recognition of quality reviewers may be one way to improve the quality of this volunteer effort.<\/p>\n
Interestingly, the APS<\/a> recently instituted a policy<\/a> of recognizing referees who review the articles submitted to the various APS journals.<\/p>\n The basis for choosing the honorees was the quality, number, and timeliness of their reports, without regard for membership in the APS, country of origin, or field of research. Individuals with current or very recent direct connections to the journals, such as editors and editorial board members, were excluded.<\/p><\/blockquote>\n <\/p>\n The ACM would do well to consider a scheme for recognizing excellence in reviewing as well. Consider the following proposal: during a review process for a conference, the program committee or a few meta-reviewers rate each review for quality on a five point (1-5) scale. Reviews that score a four accrue one point for the reviewer who wrote them; reviews that score a five accrue two points. (Perhaps reviews that score a 1 should accrue -1.) Scores are aggregated for each reviewer, and reviewers in the top 10% (or 15% or some other cutoff) get official recognition by the SIG or by the ACM, who are responsible for maintaining and publicizing the list of these excellent reviewers. To account for improvement (or lack of participation), the score on which reviewers are ranked could be a (potentially weighted) sum of contributions over the latest three years during which the person reviewed at least one submission.<\/p>\n These details are somewhat arbitrary, but I believe are representative of a scheme that recognizes outstanding reviewing. Making reviewing more visible to the community should encourage people to write more thoughtful reviews. While this scheme imposes some additional work on the program committee to vet reviews, the improvement in review quality\u00a0 should compensate by yielding a smoother-running and more reliable review process.<\/p>\n","protected":false},"excerpt":{"rendered":" I believe one of ways in which the current process can be improved is by formal recognition of reviewers’ efforts. This post proposes one such scheme.<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/2392"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2392"}],"version-history":[{"count":9,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/2392\/revisions"}],"predecessor-version":[{"id":2396,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/2392\/revisions\/2396"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2392"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2392"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2392"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}