So in our paper, we chose to evaluate our framework through experiments that involved relevance feedback.\u00a0 Relevance feedback is a long-studied, traditionally well-accepted interaction paradigm.\u00a0 The user runs a query, judges a few documents for relevance, and any relevant documents that are found during this process are saved or marked and fed back into the system to produce even better results on subsequent queries.\u00a0 Our results showed that the proposed framework is not only more effective than a robust, well-understood baseline, but that algorithms involved are up to an order of magnitude more efficient than traditional baselines.\u00a0 And speed is of utmost importance to interactive IR systems!<\/p>\n
We received three reviews…<\/p>\n
![\"\"](\"http:\/\/palblog.fxpal.com\/wp-includes\/js\/tinymce\/plugins\/wordpress\/img\/trans.gif\" "\\"More...\\"")
Reviewer #1<\/h3>\n
The first review, after summarizing our contribution, read in its entirety:<\/p>\n
The paper is well written and both the idea and the experimental part are sound.<\/p><\/blockquote>\n
This was accompanied by a 4\/6 recommendation score.\u00a0 Not much help.<\/p>\n
Reviewer #2<\/h3>\n
The second review’s worst criticism of the work was that the evaluation was incomplete:<\/p>\n
The idea is new & interesting, especially that it can make use of non-text query logs. One drawback of the paper, in this reviewer’s opinion, is incompleteness: why pseudo-relevance-feedback not considered as well, which is easy to do? Asking a user to judge documents until one gets 5 relevant may not be realistic. Even if PRF does not work, paper should present the results.<\/p>\n
…<\/p>\n
The idea … is new & interesting, especially that it can make use of non-text query logs. One drawback of the paper, in this reviewer’s opinion, is incompleteness: no study of employing pseudo-relevant docs. The impact would be small if one requires judged relevant docs.<\/p><\/blockquote>\n
This criticism is flawed on three counts:<\/p>\n
\n
- First, it was flat out wrong.\u00a0 We were not asking people to find five relevant documents<\/em>, we were asking them to make five judgments of relevance<\/em>. This was made very clear in the paper.\u00a0 Furthermore, even if making explicit judgments is difficult, there are many techniques for eliciting implicit (but not pseudo!) judgments of relevance (e.g., see Kelly and Belkin, 2001<\/a>).<\/li>\n
- The second reason for not doing pseudo-relevance feedback is that it is more likely to introduce noise and topic drift.<\/li>\n
- The third reason for avoiding PRF entirely is that it is unnecessary for interactive systems.\u00a0 Indeed, if a user is reading or saving (marking) documents, i.e. if a user is giving explicit judgments of relevance, decades of research have already shown those judgments to be much more effective than pseudo-judgments.<\/li>\n<\/ol>\n
The argument that somehow an evaluation is incomplete or meaningless — or that the impact is small — if it does not involve pseudo-relevance feedback is offensive in its narrow-mindedness.\u00a0 What it reflects, we believe, is the current bias of the field as a whole toward non-interactive web search-like experiences.\u00a0 In the web IR world, the commonly held understanding is that users are too lazy to engage in explicit relevance feedback, or else are engaged in a type of information seeking activity, such as navigation, that does not require any feedback, pseudo-relevant or otherwise.\u00a0 But web information retrieval is not all of information retrieval.<\/p>\n
This second reviewer gave us a 3\/6 recommendation score.<\/p>\n
Reviewer #3<\/a><\/h3>\n
The third reviewer, while stating that the work is novel, had two main concerns:<\/p>\n
\n
- Our chosen query expansion technique (selection and weighting of terms) was not convincing because many others were possible, using our framework.<\/li>\n
- We did not evaluate using pseudo-relevance feedback.<\/li>\n<\/ol>\n
For the first point, we intentionally chose the simplest implementation of our framework to show its strength and to intentionally make the fairest comparison possible.\u00a0 If our naive approach beats the quite reasonable baseline (20-30% increase in effectiveness, 10x speedup in efficiency) that should be enough; it is beyond the scope of a conference paper to exhaustively demonstrate its effectiveness for arbitrary schemes a reviewer might dream up. The naive approach worked. That’s a publishable result.<\/p>\n
That brings us again to the second point, which actually sounded like the stronger reviewer criticism: Pseudo-relevance feedback.\u00a0 Reading between the lines of the reviews, one gets the impression that the reviewers are well-versed in traditional web search: they mention log mining (a minor aside in our paper), and are obsessed with pseudo-relevance feedback. Those of us who were doing IR research before the late 1990s remember a time when intellectual efforts were not judged by standards applicable only to web search. The diversity of approaches, of metrics, and\u00a0 of applications of that era seem to have been reduced to the bleak outlines of precision-oriented page-at-a-time results lists, where interactivity is looked upon as a burden rather than an opportunity<\/em>.<\/p>\n