{"id":4648,"date":"2010-09-21T07:12:06","date_gmt":"2010-09-21T14:12:06","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=4648"},"modified":"2010-09-20T23:53:58","modified_gmt":"2010-09-21T06:53:58","slug":"citing-best-papers","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=4648","title":{"rendered":"Citing Best Papers"},"content":{"rendered":"

Jeff Huang recently published a list of papers<\/a> from several major conferences that won Best Paper awards. It’s a nice collection of papers, highlighted in a way that is difficult to obtain from the ACM Digital Library. (Why that should be the case is a different story.)<\/p>\n

Clearly winning a Best Paper award is a significant achievement and authors of such papers should be proud of their work. But does this merit translate into impact? For example, do papers that win Best Paper awards get cited more frequently than other papers from the same conference?<\/p>\n

I wish I had access to the ACM metadata for the papers in question, as it would make a larger-scale analysis of citation rates straightforward. As it was, I resigned myself (again) to some manual analysis: I recorded the within-ACM citation counts for all full papers published in the SIGIR 2001-2005 conferences, as shown in Figure 1. (The data itself is here<\/a>.)<\/p>\n

\"\"<\/a>

Figure 1. Citation counts for full papers for SIGIR 2001-2005<\/p><\/div>\n

First, the general shape is not surprising: a few papers receive a lot of citations, and the rest trail off. Even though the number of papers accepted varied from year to year (46, 44, 46, 58, 71), the range of citations was quite comparable, so I didn’t bother normalizing the data.<\/p>\n

So where do the best papers lie? For the years 2001-2005, the best paper winners were cited 39, 68, 41, 39, and 36 times, respectively. This corresponds to ranks 17.5, 3, 13, 9, and 9. This is summarized in the following table:<\/p>\n\n\n\n\n\n\n\n\n
Year<\/td>\nPapers<\/td>\nRank of best
\npaper<\/td>\n		% of rank<\/td>\n# citations<\/td>\n% of max<\/td>\n<\/tr>\n
2001<\/td>\n46<\/td>\n17.5<\/td>\n38%<\/td>\n39<\/td>\n17%<\/td>\n<\/tr>\n
2002<\/td>\n44<\/td>\n3<\/td>\n6.8%<\/td>\n68<\/td>\n63%<\/td>\n<\/tr>\n
2003<\/td>\n46<\/td>\n13<\/td>\n28%<\/td>\n41<\/td>\n28%<\/td>\n<\/tr>\n
2004<\/td>\n58<\/td>\n9<\/td>\n16%<\/td>\n39<\/td>\n39%<\/td>\n<\/tr>\n
2005<\/td>\n71<\/td>\n9<\/td>\n13%<\/td>\n36<\/td>\n23%<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

This is not enough data to make strong conclusions, but it seems like the award winners achieve on average about 1\/3 the citation rate of the best-cited paper in a conference, and rank, on average in top quintile.<\/p>\n

In another attempt to visualize the data, I subtracted the citation count of the best paper in each year from all the other papers published that year, to produce Figure 2.<\/p>\n

\"\"<\/a>

Figure 2. Citation rates by year offset by the citation rate of the Best Paper award winner<\/p><\/div>\n

In this chart, we can see that 2002 was significantly better than the other years in predicting the best paper award winner (most of the papers were cited less), but the other years were largely indistinguishable.<\/p>\n

I suppose there are many reasons that a Best Paper award winner does not get high citation rates:<\/p>\n

    \n
  • some of this is due to chance (perhaps the high-cited papers were also considered for the award)<\/li>\n
  • some may be due to innovation (ground-breaking work may not get as much recognition as solid work in an established area)<\/li>\n
  • some may be due to selection bias within the nominating committee that may consider certain papers to be flawed or otherwise unacceptable, while the public at large may recognize something important in the paper anyway<\/li>\n
  • How well-attended a paper presentation is may also make a difference in generating buzz about the work.<\/li>\n<\/ul>\n

    The upshot, however, is that it’s quite difficult to predict the real impact of a paper at the time it is published. It might be interesting for specific communities to hold reviews of papers published five to ten years ago to assess the impact retrospectively. This should, of course, involve more than citation counts: it is important to look at the reasons papers get cited. A measurement that aggregates\u00a0 the degree of importance of a paper to the papers the cite it might approximate this longer-term impact, although such a measure might be quite subjective.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Jeff Huang recently published a list of papers from several major conferences that won Best Paper awards. It’s a nice collection of papers, highlighted in a way that is difficult to obtain from the ACM Digital Library. (Why that should be the case is a different story.) Clearly winning a Best Paper award is a […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[18],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4648"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4648"}],"version-history":[{"count":8,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4648\/revisions"}],"predecessor-version":[{"id":4657,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4648\/revisions\/4657"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4648"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4648"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4648"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}