{"id":1716,"date":"2009-09-02T07:11:26","date_gmt":"2009-09-02T14:11:26","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=1716"},"modified":"2010-05-31T20:01:19","modified_gmt":"2010-06-01T03:01:19","slug":"have-queries-want-answers","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=1716","title":{"rendered":"Have queries, want answers"},"content":{"rendered":"<p>Sarah Vogel&#8217;s <a href=\"http:\/\/palblog.fxpal.com\/?p=1710#comment-7737\" target=\"_blank\">comment<\/a> on yesterday&#8217;s <a title=\"Open-source queries | FXPAL Blog\" href=\"http:\/\/palblog.fxpal.com\/?p=1710\" target=\"_blank\">post<\/a> got me thinking about recall-oriented search. She wrote about preferring Boolean queries for complex searches because they gave her a sense for when she really had exhausted a particular topic, something that&#8217;s often required for medical literature reviews. But we really have multiple problems here, that it may be useful to decouple: one is the issue of coverage (did we find all there was to find?) and the other is ranking (the order in which documents are shown).<\/p>\n<p><!--more-->A perfect query would retrieve all the required documents, and no others; at that point, order wouldn&#8217;t matter a whole lot. A slightly less perfect query might still retrieve all the required documents, but might also throw in some that aren&#8217;t. If it gets the order right, though, the all the useful documents will precede the others. But this outcome relies on two things: having the right query terms, and having the right ranking function. (Of course for most real topics, you&#8217;ll need multiple queries, but I&#8217;m building a straw-man.)<\/p>\n<p>Back to the <a title=\"PubMed Search Strategies Blog\" href=\"http:\/\/pubmedsearches.blogspot.com\/\" target=\"_blank\">PubMed Search Strategies<\/a> queries I wrote about yesterday: they represent considerable effort to construct an expression that gives good coverage for a particular concept, but when used with PubMed they suffer because the ranking isn&#8217;t based on the degree of match.<\/p>\n<h3>Predicting useful documents<\/h3>\n<p>One set of experiments I would like to be able to do is to analyze the final set of documents judged to be useful that were produced by a search that consists of several such concepts to understand how the different expressions of concepts contributed to that set of documents. My guess is that the expressions for each concept have some redundancy, and it would be interesting to understand that redundancy. In other words, given<\/p>\n<ul>\n<li>a decent ranking function that takes the quality of a match into account (even if the documents are selected using a Boolean expression), and<\/li>\n<li>sets of OR-ed terms that represent each concept (leaving the ANDs in place)<\/li>\n<\/ul>\n<p>how close can we get to the ideal set of results (as judged by the person who did the comprehensive search) by adding or removing terms from the sets characterizing the constituent concepts?<\/p>\n<p>Will we find that only a few terms are enough, taken in combination, to produce the required set of documents? Or will we find it difficult to get good recall without retrieving large sets of documents? Furthermore, will a solution to one combination of concepts be sufficiently similar to those of other sets so that we can generalize the results? If we cannot automate this process, will there still be clear points at which a person&#8217;s insight could guide the search? These questions are vague in the absence of concrete examples, but they are probably a good starting point for more concrete discussion.<\/p>\n<h3>Keywords vs. MeSH<\/h3>\n<p>Another class of questions I think could be answered with this data is related to the differences between keyword and MeSH expressions of concepts. Sarah Vogel commented that the quality of the index terms varies across MeSH. If the quality of the index could be characterized in some gross level, then it could be used as an independent variable in evaluating the effectiveness of MeSH for expressing queries.<\/p>\n<p>Also, it would be interesting to understand the differences between the MeSH queries and keyword queries on <a title=\"PubMed Search Strategies Blog\" href=\"http:\/\/pubmedsearches.blogspot.com\/\" target=\"_blank\">PubMed Search Strategies<\/a> site. While the two kinds of queries were often paired, they didn&#8217;t retrieve the same set of documents. If the person who created the queries thought them to be similar, are their differences important? Are the documents retrieved by both queries (keyword and MeSH) somehow more useful, more central to the concept?<\/p>\n<p>Related to this issue is the finding that there is no evidence that MeSH-based queries perform better on average than keyword-based queries, as I wrote in an earlier <a title=\"What a tangled MeSH we weave | FXPAL Blog\" href=\"http:\/\/palblog.fxpal.com\/?p=1666\" target=\"_blank\">post<\/a>. This means that MeSH queries and keyword queries retrieve about the same number of useful (relevant) documents. But do they retrieve the same documents, or different subsets? If the latter, what can we learn about the queries to figure out how to combine them to identify a larger subset of relevant documents? It would also be interesting to to see if conclusions from MeSH carry over in a meaningful way to other thesauri.<\/p>\n<p>The point of this rambling post is that where there are data, there are interesting questions to ask and answer, and this particular set of queries (with the promise of more to come) intrigues me. But these queries are mere shadows on a wall, and it would be considerably more interesting to engage with people who construct such expressions to understand better how they look for information, and how to make them more productive.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Sarah Vogel&#8217;s comment on yesterday&#8217;s post got me thinking about recall-oriented search. She wrote about preferring Boolean queries for complex searches because they gave her a sense for when she really had exhausted a particular topic, something that&#8217;s often required for medical literature reviews. But we really have multiple problems here, that it may be [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[112,209],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1716"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1716"}],"version-history":[{"count":8,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1716\/revisions"}],"predecessor-version":[{"id":3823,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1716\/revisions\/3823"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1716"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1716"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1716"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}