We’ve been dabbling in the word of Mechanical Turk, looking for ways to collect judgments of relevance for TREC documents. TREC raters’ coverage is spotty, since it is based on pooled (and sometimes sampled) documents identified by a small number of systems that participated in a particular workshop. When evaluating our research systems against TREC data from prior years, we found that many of the identified documents had not received any judgments (relevant or non-relevant) from TREC assessors. Thus we turned to Mechanical Turk for answers.
Marti Hearst recently gave a talk at Google related to the themes in her book. She does a good job of explaining the challenges and opportunities related to interactive information seeking, including design, evaluation, query reformulation, integrating navigation and search, information visualization as it relates to search, and future trends. While most of this is music to the ears of HCIR types, her discussion of collaborative search (around minute 46) is particularly “relevant:” Marti spends a good deal of time on our paper on collaborative search, describing the various models of collaboration and showing some figures from our paper. The talk is on YouTube, the paper is on the web. Questions and comments are very welcome.
ps: Marti’s mention of Diane “Green” in minute 24 actually refers to Diane Kelly, whose well-received paper on query suggestion was presented at SIGIR 2009.
In 2001, when we were thinking about how to use e-books for legal research, we partnered with Lexis Nexis to study a moot court class in a law school. Without access to the documents that we obtained through Lexis, we would not have been able to engage the students and to explore potential designs for such devices.
But that was eight years ago. Today, we could resort to Google Scholar: A couple of days ago, Google announced on its blog that it will be including full text legal opinions from U.S. federal and state district, appellate and supreme courts in results returned by Google Scholar. In addition to each case, Google also returns citations of that case in other opinions. This service is unlikely to put West Publishers or Lexis Nexis out of business, but it does make it considerably easier for the average person (or researcher) to find these cases.
We are nearing the end of editing the Special Issue of Information Processing & Management, and are proud to announce the papers that will be in the issue. The Special Issue was the result of the 1st collaborative search workshop we organized at JCDL 2008; the next workshop is coming up soon! We had many submissions on a variety of related topics, including field work and other reporting that characterized instances of collaboration in information seeking, evaluation and models of collaborative episodes, and a number of system and algorithm papers.
Amazon’s Mechanical Turk is increasingly being used to obtain judgments of relevance that can be used to establish gold standards with which to evaluate information seeking experiments. The attraction is clear: for a few bucks, in a few days, you can obtain data that is every bit as useful for evaluating simulations and other off-line experiments as data collected in the lab from “live” participants, and may be a good substitute for TREC assessors’ judgments. And of course the scale of the enterprise is such that you can run complex multi-factor experiments and still retain enough power. If you’re not up to doing this yourself, companies such as Dolores Labs will do it for you.
Over the last year or so, Scott Carter, Jacob Biehl, and I have built and deployed an interesting system for managing whiteboard content. The system, ReBoard, consists of a camera that takes pictures of a traditional (or electronic, if you wanted) whiteboard when whiteboard content changes. The images captured by the camera are cleaned up by adjusting contrast and correcting for skew, and then saved into a database along with a bunch of metadata that identifies the changed region, the time and place the image was taken, and whether the content was likely created as a collaboration. Once captured, images can be shared with others and can be annotated by adding tags and notes.
A while ago, Google introduced Google Squared, an attempt to help people keep track of different aspects in their search results. I think that it’s an interesting HCIR idea that still lacks a good implementation, as I’ve written here and here. Recently, Google introduced a means of adding results informed by the searcher’s social network, which Google has dubbed “Social Circle.” I spent some time playing with it, and found it lacking.
Last week I was in DC at the HCIR 2009 workshop organized by Bill Kules, Daniel Tunkelang, and Ryen White. This was the third workshop in the series, and by far the biggest and most diverse in terms of attendees. Proceedings are available online. Daniel and Max Wilson have already given pretty good coverage to what happened at the workshop, so I will focus on my impressions, starting with Ben Shneiderman‘s keynote.
I am giving a talk today at NIST on collaborative search. Abstract:
In the library sciences, information seeking has long been recognized as a collaborative activity, and recent work has attempted to model group information seeking behavior. Until recently, technological support for group-based information seeking has been limited to collaborative filtering and “social search” applications. In the past two years, however, a new kind of technologically-mediated collaborative search has been demonstrated in systems such as SearchTogether and Cerchiamo. This approach is more closely grounded in the library science interpretation of collaboration: rather than inferring commonality of interest through similarity of queries (social search), the new approach assumes an explicitly-shared information need for a group. This allows the system to focus on mediating the collaboration rather than detecting its presence. In this talk, we describe a model that captures both user behavior and system architecture, describe its relationship to other models of information seeking, and use it to classify existing multi-user search systems. We also describe implications this model has for design and evaluation of new collaborative information seeking systems.
I’ve been going through some of the analytic Twitter tools/sites listed in oneforty. It’s a mixed bag, as should be expected, with some nice tools. In this post, I’ll talk about TweetReach, TweetStats, and Twitoaster; the rest will have to wait.
