At Daniel Tunkelang’s suggestion, I revisited Google Squared, having written about it when it was first released. At the time, I tried a couple of queries (not a formal evaluation), and found some useful results, and some bogus ones. This time, I re-ran the same queries as before, and compared the results with my saved queries. For the query ‘airplane accidents’, the new results were considerably worse. For the query ‘acts of terrorism’, there were no initial results, but when I put in some instances (WTC attack, Oklahoma City bombing, Khobar towers, marine barracks) I got back a similar list to the one I had constructed in June.
An e-mail exchange with an old friend caused me to reflect on research in HCIR a decade ago: in the Hypertext conference series there was a lot of churn and innovation around ways to represent structure, about literary hypertext, and about novel interaction techniques that allow people to express information seeking intent in interesting ways. Much of that cottage garden research was swept away by the steam engine of the web, for better of for worse. The demands of scalability led to the abandonment of all sorts of niceties (such as link integrity, for example), including a rich model of interaction. SIGLINK, ACM’s SIG on hypertext, renamed itself SIGWEB in an attempt to stay relevant. The main impact of all that research seemed to be the idea that you could click on blue-underlined text to do something.
Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel have written what will undoubtedly be a controversy and discussion-inspiring paper for the upcoming CIKM 2009 conference. The paper compares over 100 studies of information retrieval systems based on various TREC collections, and concludes that not much progress has been made over the last decade in terms off Mean Average Precision (MAP). They also found that studies that use the TREC data outside the TREC competition tend to pick poor baselines to show short-term improvement (which is publishable) without demonstrating long-term gains in system performance. This interesting analysis is summarized in a blog post by William Webber.
Jeremy and I have been blogging about collaborative search for a while, and it is our pleasure to announce that Merrie Morris and we are organizing another workshop on Collaborative Information Seeking. The first workshop was held in 2008 in conjunction with the JCDL 2008 conference. We had a many interesting presentations and a lot of discussion about systems, algorithms, and evaluation.You can find the proceedings from the workshop on arXiv.org (metadata and papers) and on the workshop web site.
It’s time to revisit this topic, this time in conjunction with the CSCW 2010 conference. The workshop call for participation is here. Our goal is
to bring together researchers with backgrounds in CSCW, social computing, information retrieval, library sciences and HCI to discuss the research challenges associated with the emerging field of collaborative information seeking.
To participate, please submit a 2-4 page position paper in the ACM format by November 20th. The workshop will take place in February, in Savannah, Georgia. Hope to see you there!
The number of third-party tools for searching PubMed data seems to be increasing recently. As the NLM is about to roll out a new search interface, companies are starting to offer alternative interfaces for searching this important collection. The attraction is obvious: a large, motivated group of searchers, an important information need, and a manageable collection size. A decade ago, over 20 million searches were done monthly through the NLM site, and the numbers are surely higher today; the collection is large but not huge — currently over 17 million entries (some with full text), occupying somewhat more than 60GB of disk space. Thus we see an increasing number of sites offering search over this collection, including PubGet, GoPubMed, TexMed, and HubMed. The offerings range from basic to flashy, and appear to be aiming at different groups of searchers.
In a recent post, Miles Efron proposed a distinction between different kinds of information retrieval: “macro IR” that concerns with generic tasks such as searching the web, and “micro IR” that represents more focused interaction. My sense is that one key distinction between the two is the degree to which the system represents the context of the search, and therefore is able to act on the results. Miles’ examples–finding restaurants, books, music, people–have a transactional quality about them. The system has a sufficient representation of the task to both structure the query in an appropriate manner (e.g., Yelp! metadata about restaurants) and to act on the selected result (e.g., offer to make a reservation). Macro IR, on the other hand, lacks a strong contextual representation, and leaves it to the user to act on the retrieved information.
Ian Soboroff commented on yesterday’s blog post that although mental models were important, they were insufficient. He cited a paper that found that legal staff had experienced problems with using a full-text search engine to search (with a recall-oriented information need) a collection of documents in a legal discovery scenario. The paper concludes that coming up with effective keyword searches is difficult for non-search experts. The paper is interesting and worth reading, but I believe the authors conclusions are not warranted by their methodology.
A discussion among commenters on a post about PubMed search strategies raised the issue of how people need to make sense of the results that a search engine provides. For precision-oriented searches a “black box” approach may make sense because as long as the system manages to identify a useful document, it doesn’t matter much how it does that. For exploratory search, which may be more recall-oriented, having a comprehensible representation of the system’s computations is important to assess coverage of your results. This suggests the need to foster useful mental models, rather than relying on the system to divine your intent and magically produce the “right” result.
Sarah Vogel’s comment on yesterday’s post got me thinking about recall-oriented search. She wrote about preferring Boolean queries for complex searches because they gave her a sense for when she really had exhausted a particular topic, something that’s often required for medical literature reviews. But we really have multiple problems here, that it may be useful to decouple: one is the issue of coverage (did we find all there was to find?) and the other is ranking (the order in which documents are shown).
Every once in a while a Twitter query turns up something completely unexpected. I suppose that’s one reason for having them. My query on all things PubMed recently turned up the following gem: a blog entitled PubMed Search Strategies. What is it? A list of queries. What? PubMed Queries, in all the Boolean glory. The latest pair of posts are pharmacoepidemiology — keywords, and its paternal twin, pharmacoepidemiology — MeSH. The queries run for 39 and 13 terms, respectively. No average 2.3 word Web searches these.