{"id":1472,"date":"2009-07-29T07:16:36","date_gmt":"2009-07-29T14:16:36","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=1472"},"modified":"2009-07-27T00:01:24","modified_gmt":"2009-07-27T07:01:24","slug":"building-the-ivory-tower","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=1472","title":{"rendered":"Building the Ivory Tower"},"content":{"rendered":"

I recently read on Jeff Dalton’s blog<\/a> that a new open-source search engine, called\u00a0 Ivory<\/a>, has been released by Jimmy Lin<\/a>. Ivory is based on Hadoop, and is\u00a0 designed to handle terabyte-sized collections. Unlike Lucene, this is a research project, Jimmy Lin writes,<\/p>\n

aimed at information retrieval researchers who need access to low-level data structures and who generally know their way around retrieval algorithms. As a result, a lot of “niceties” are simply missing\u2014for example, fancy interfaces or ingestion support for different file types. It goes without saying that Ivory is a bit rough around the edges, but our philosophy is to release early and release often. In short, Ivory is experimental<\/strong>!<\/p><\/blockquote>\n

Still, it is interesting to see another tool made available publicly to help spur IR research. It joins a collection of search engines and search tooklits, including Lucene<\/a>, Lemur<\/a>,\u00a0 and Terrier<\/a>. At first glance, these efforts look similar. Certainly they all show their academic roots in being able to index TREC collections out of the box. It’s not clear (to me!) what the tradeoffs among the various tools are, and how well they can be applied to support interactive search.<\/p>\n

But it’s a trend worth following, and an experiment worth conducting to compare the various approaches with respect to the variety of information seeking tasks. Having that kind of a characterization would help those deciding which tool to use for a new research or deployment project. Dimensions on which the tools could be compared include the following:<\/p>\n