On Monday I attended a crowd-sourcing Meetup with the funny (and as it turned out inaccurate) title of The Distributed Distributed Work Meetup. The idea was to hear talks about various crowd-sourcing topics from speakers in Seattle, San Francisco, Boston and New York. Technology didn’t cooperate, and were left to our own devices, which meant to eat, drink, and listen to fascinating and provocative talk by Alek Felstiner on a range of legal questions surrounding crowd-sourcing platforms such as Amazon’s Mechanical Turk (AMT).

I cannot do justice to the legal issues, in part because so many of them remain unresolved. I will, however, report on a number of questions raised during the talk and on some of the potential precedents for this kind of work. One reason to discuss this topic is that there are some concerns that the Turkers are being exploited by those who pay for the work, as the value of the work to the company sometimes seems much higher than the rate the worker is paid.

Yet many people chose to do this work freely, and seem to enjoy doing it, and certainly there are many companies and individuals, profit-oriented and academic, who benefit from this service. The questions raised in this talk bore on the legal relationship between the organizations or individuals requesting work and those providing it.

Continue Reading

The recent earthquake in Haiti has attracted attention from Twitter users and researchers. Twitter has been used to collect donations, to contact people on the ground, to coordinate relief efforts, etc. Recently, U. Colorado’s EPIC Group proposed a hash-tag-based syntax on top of Twitter messages to help automate the parsing of actionable messages, and to do so effectively and reliably. This is a noble effort, but as Manas Tungare points out, the proposed syntax is too complex for its intended users, who have more pressing issues than dealing with hash tags.

Continue Reading

We’ve been dabbling in the word of Mechanical Turk, looking for ways to collect judgments of relevance for TREC documents. TREC raters’ coverage is spotty, since it is based on pooled (and sometimes sampled) documents identified by a small number of systems that participated in a particular workshop. When evaluating our research systems against TREC data from prior years, we found that many of the identified documents had not received any judgments (relevant or non-relevant) from TREC assessors. Thus we turned to Mechanical Turk for answers.

Continue Reading

Amazon’s Mechanical Turk is increasingly being used to obtain judgments of relevance that can be used to establish gold standards with which to evaluate information seeking experiments. The attraction is clear: for a few bucks, in a few days, you can obtain data that is every bit as useful for evaluating simulations and other off-line experiments as data collected in the lab from “live” participants, and may be a good substitute for TREC assessors’ judgments. And of course the scale of the enterprise is such that you can run complex multi-factor experiments and still retain enough power. If you’re not up to doing this yourself, companies such as Dolores Labs will do it for you.

Continue Reading