Jamie Callan of CMU gave an interesting and thought-provoking keynote talk at CIKM 2010. While traditionally search engines have been used in a more or less direct manner to identify useful documents that the user would then (manually) incorporate into other tasks, Jamie suggested a new class of applications that would use search engines for the purposes of identifying documents or parts of documents in some collection, but then would apply this information in pursuit of some other, more specialized, task.<\/p>\n
While the notion of using a search engine as a component of another system is not particularly novel, the kinds of requirements that his proposed use imposes on search engines would certainly push the envelope.<\/p>\n
He started by motivating his talk with three applications:\u00a0computer-assisted language learning, question-answering, and Never Ending Language Learner (NELL). The way such applications are constructed currently involves running some query, extracting information from the results, and throwing away the junk. Queries are typically keyword searches or simple patterns, but these often do not meet the true requirements of the applications that consume the results.<\/span><\/p>\n What he was proposing was a general solution that allows the search engine to know as much as possible about <\/span>the application’s information need and <\/span>the document contents. His claim is that current structured queries can handle simple document structure, but are too brittle and cannot handle more complex structures. His examples centered about using various NLP techniques to parse document structure and the kinds of failures that might break traditional approaches to index structured documents.<\/span><\/p>\n The best example of an alternative approach that Jamie described was based on work in indexing PubMed structure that was based on a relational schema that combined elements of a traditional document (author, title, abstract, \u00a0journal, etc.) with more meta-level information such as gene-t0-gene relationship information. (Unfortunately, I wasn’t able to catch the reference to this work; stay tuned.)<\/span><\/p>\n He concluded with a call for research on this new class of applications that combine <\/span>multiple forms of knowledge and language analysis and <\/span>metadata and structure of varying reliability. These applications pose m<\/span>any interesting unsolved core IR problems, require d<\/span>iverse information resources to exploit, and create opportunities for new retrieval models.<\/span><\/p>\n While I found the talk interesting and inspiring, I think some of the kinds of indexing and ranking algorithms that he wants to see already exist. Two examples come to mind: Ancestry.com and PowerSet. <\/span><\/p>\n Ancestry.com’s search interface implements a rich and flexible schema that incorporates exact matches and flexible best-match approach. While this is not exactly what Jamie suggested because it solves a direct search problem, it does address some of the search infrastructure requirements of flexible search capability.<\/span><\/p>\n As I understand it, PowerSet’s approach is to index (subject, verb, object) tripples are created from documents, and then indexed in a more traditional manner. This seems to be a step in the right direction, and fallback strategies for compensating for parser failures should make this more robust. Its incorporation into Bing for Wikipedia search is further indication of the viability of this approach.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":" Jamie Callan of CMU gave an interesting and thought-provoking keynote talk at CIKM 2010. While traditionally search engines have been used in a more or less direct manner to identify useful documents that the user would then (manually) incorporate into other tasks, Jamie suggested a new class of applications that would use search engines for […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[255],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4866"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4866"}],"version-history":[{"count":5,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4866\/revisions"}],"predecessor-version":[{"id":4871,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4866\/revisions\/4871"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4866"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4866"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4866"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}