{"id":1246,"date":"2009-07-01T11:37:19","date_gmt":"2009-07-01T18:37:19","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=1246"},"modified":"2009-07-01T11:37:19","modified_gmt":"2009-07-01T18:37:19","slug":"on-the-science-of-ir","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=1246","title":{"rendered":"On the Science of IR"},"content":{"rendered":"

Miles Efron<\/a> posted recently on his take on the progress of the IR field<\/a> in response to a question posted by Andrew Dillon<\/a> at the last ASIST<\/a> conference. Miles’ take was that progress was indeed being made for two reasons: the SIGIR<\/a> conference has become more competitive over the years, and the diversity of corpora in the TREC<\/a> umbrella has also increased. Unfortunately, I wasn’t there to hear the question or the subsequent discussion, but my guess as to what Andrew Dillon actually meant was not a question of statistical significance, but rather one of magnitude.<\/p>\n

Every year we see incremental improvements in Mean Average Precision<\/a> (MAP) scores reported in SIGIR (and in CIKM, and in other venues) for some narrow conceptions of the search task. The gains are real, but they may not matter. Similarly, Google recently reported<\/a> (thanks Jeremy<\/a>, thanks Greg<\/a>) that a change in latency from 100 msec to 400 msec reduced the number of queries people ran by about 0.5%. Statistically significant, yes. Important? Maybe not.<\/p>\n

The scientists among us like to measure things. That’s how we (and others) know we did something interesting. But it seems that what we really want to measure is difficult to observe, and so we settle on some plausible proxy. And so begins the slippery slope.<\/p>\n

It is certainly true that having ongoing improvement in indexing and retrieval algorithms is a good thing. But in some ways it has become a victim of its own success, and, like commercial agriculture, now produces decent commodity goods at ridiculously low cost. To continue with the analogy, we need to diversify our notion of information retrieval to include not only the supermarket (where any time of day you can find exactly the same product that you’ve always bought but without the ability to really understand or control what’s in the box) but also the farmer’s market, where you can find more variety, more surprises, and more interaction with the people who grow the food you will be eating.<\/p>\n

So there is still room in the field of information retrieval for progress, but the low-hanging fruit of precision-oriented search have been harvested. We now need to look to more difficult tasks<\/a>, to exploratory search<\/a>, to interaction<\/a>, to collaboration<\/a>.\u00a0 Looking beyond the ranked list is not only a pragmatic strategy for innovation, it’s also good science.<\/p>\n","protected":false},"excerpt":{"rendered":"

Miles Efron posted recently on his take on the progress of the IR field in response to a question posted by Andrew Dillon at the last ASIST conference. Miles’ take was that progress was indeed being made for two reasons: the SIGIR conference has become more competitive over the years, and the diversity of corpora […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1246"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1246"}],"version-history":[{"count":9,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1246\/revisions"}],"predecessor-version":[{"id":1255,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1246\/revisions\/1255"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1246"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1246"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1246"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}