In the aftermath of the CHI 2010 PC meeting, we had an interesting discussion of issues related to reviewing and managing the CHI conference submission process. Several interesting approaches to improving the outcomes were discussed, including reinstating mentoring, rating reviews, adding a desk-reject option for some papers, etc. The overall goal is to improve the quality of submissions and the quality of reviews. Simplifying the overall process was also brought up several times.<\/p>\n
Improving the quality of submissions had in the past been addressed with mentoring: Mentoring was a program that helped first-time authors structure their papers to conform with expectations of the CHI community. Prospective authors were matched with volunteer mentors who helped them prepare the paper for submission. The program ran for a few years in early 2000s, but was canceled due to difficulties in finding mentors. It would be great to restart the program again.<\/p>\n
Another way to improve the quality of papers that are subjected to review is to filter out those papers that contain obvious structural flaws. In the current CHI review process, papers are assigned to Associate Chairs (ACs) who recruit three external reviewers for each paper. In addition, one or more other ACs is assigned to read papers that have middling scores or high degree of disagreement among the external reviewers. Papers with scores toward the middle of the distribution (and those with considerable disagreement) are then discussed at the PC meeting. At any time, any AC can also nominate a paper for discussion regardless of its score, good or bad.<\/p>\n
The problem is that papers with obvious flaws that will result in a clear rejection still consume as much effort on the part of the external reviewers as do papers that have a chance of being accepted. This dilutes the reviewer pool, overloading reviewers, and making it harder to obtain quality reviews. One proposed solution to this problem is to screen\u00a0 submissions before they are reviewed. A number of ACs (say 3-4) read each submission to scan for obvious flaws that will preclude acceptance. If they all agree that a submission passes this inspection step, it gets added to the review pool; if not, it is returned to the authors with an explanation of what was lacking in the submission, along with a relevant example (based on a similar submission that was accepted to a prior conference). This process could be combined with the afore-mentioned mentoring process if the paper is submitted more than two months in advance of the regular deadline. The goal is to remove those papers that would wind up receiving uniformly low scores (less than 2 out of 5) from the pool of papers to which external reviewers will be assigned. The extra work up front by the ACs would result in a smaller number of papers that would need to be managed later. While this suggestion may seem unfair or undemocratic, it may improve the overall quality of the conference by allowing a more effective allocation of the scare reviewing resources.<\/p>\n
In addition to increasing the quality of submissions considered for review, more attention needs to be paid to the quality of reviewing. One of the challenges to obtaining quality reviews is to establish a good match between the reviewer and the paper being reviewed. Typically, papers are characterized by a number of author-selected keywords, and reviewers describe their expertise through the same keywords; the system can then match reviewers to papers to assure that expertise is being applies correctly. Unfortunately, this approach doesn’t always work, in part because reviewers’ self-characterization may not reflect their actual expertise.<\/p>\n
There are several possible ways to mitigate this mismatch.<\/p>\n
The CHI conference is growing (the number of submissions was up by 18% over last year), and it is important to make sure that we have an effective and efficient review process that balances the needs of the authors and the reviewers.<\/p>\n","protected":false},"excerpt":{"rendered":"
In the aftermath of the CHI 2010 PC meeting, we had an interesting discussion of issues related to reviewing and managing the CHI conference submission process. Several interesting approaches to improving the outcomes were discussed, including reinstating mentoring, rating reviews, adding a desk-reject option for some papers, etc. The overall goal is to improve the […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/2425"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2425"}],"version-history":[{"count":3,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/2425\/revisions"}],"predecessor-version":[{"id":2428,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/2425\/revisions\/2428"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2425"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2425"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2425"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}