In her talk at the IR Eval workshop at SIGIR 09, Sue Dumais called for an experimental platform for conducting research in information seeking (thanks Sakai-san!). She called it a Living Laboratory. This is a tremendous idea, the high tide that lifts all boats. Whether you’re interested in doing log analysis, interface design evaluation, building new indexing algorithms, or other kinds of research, having real data sets with real users and real information needs can move the field forward in ways that Cranfield-style experiments do not.

So what’s necessary to do this? Sue’s slide calls out APIs for web search, and also suggests collections such as the Wikipedia, Twitter, and scholarly publications. She also points out that “ranking experiments are somewhat limited,” and that “UX [is] perhaps more natural.” My sense is that in the absence of ground truth, and given the desire to evaluate exploratory as well as known-item and navigational search activity, we need to identify not only collections, but also users with information needs that may be satisfied by those collections. One such collection with a known set of users and information needs is the ACM Digital library that I wrote about earlier.

So a question to the SIGIR community is what kinds of resources would it take to create this kind of a “Living Laboratory”? Are we talking about 10s or 100s of thousands of dollars annually? Are there 100 institutions world-wide that would be willing to pay $1000/year to participate in this research? How about $5,000? Would that be enough? Can this laboratory be designed in such a way that it could be used by both UI and algorithms researchers? Would individual research efforts be siloed, or could I build a UI on top of someone’s particularly clever faceted search algorithm? Would this kind of effort be reduced to nothing more than a hosting service for some documents, or could an ecology of tools be developed to complement each other, to combine with each other?

This seems like a complex endeavor not only from the technical but also from social and political perspectives. And yet, the advantage of having this resource seems important enough that we should try to pursue it. My question to Sue and to the rest of the IR community is how do we start?

Share on: