We’ve been dabbling in the word of Mechanical Turk, looking for ways to collect judgments of relevance for TREC documents. TREC raters’ coverage is spotty, since it is based on pooled (and sometimes sampled) documents identified by a small number of systems that participated in a particular workshop. When evaluating our research systems against TREC data from prior years, we found that many of the identified documents had not received any judgments (relevant or non-relevant) from TREC assessors. Thus we turned to Mechanical Turk for answers.

Continue Reading

In the past 15 years, FXPAL has hosted a large number of interns, many of whom have become (even more) prominent in their fields. We will soon be recruiting a new crop of interns, and I thought it would be interesting to dig around a bit and see what people are up to these days. A few are now employees at FXPAL. What about the others?

Continue Reading

Jofish reports some interesting numbers regarding the role that the associate chairs play in the outcome of CHI paper reviews. He analyzes the CHI 2010 data to reach the following conclusions:

• Of the 302 submissions accepted, 57 or so were affected by decisions made at the meeting
• The 1AC (the primary meta-reviewer) was instrumental in getting a paper rejected 31 times, but was not able to prevent rejection 111 times, and represented reviewers’ consensus 1199 times.
• He also provides some more ammunition for the desk-reject debate.

It would be great to repeat this analysis on other years to see how reliable the patterns are.

An open question is whether the 57 or so papers whose fates were determined at the PC meeting deserved the outcome they received. (Obviously, the rejected papers’ authors would argue against this process.) It’s also interesting to note that it is not possible to replace the CHI PC process with an rule based on average scores, because both the reviewers and the ACs might then try to game the system by assigning extreme scores to marginal papers.

December 10. Could not open box. Tried several times.

December 11. Co-worker took it to get charged. When she brought it back, Walt Whitman’s picture replaced the lady who was displayed earlier.

December 12. Could not turn on device. Accidentally discovered cable and plug in the packaging. Charged overnight through my laptop.

Continue Reading

PARC hosted an NSF-funded workshop devoted to Technology Mediated Social Participation. This was an invitation-only workshop, coupled with a public panel discussion at the PARC Forum yesterday.

All around us, technology-mediated social participation has been harnessed for remarkable social benefits. New thrusts in basic research and engineering are likely to move beyond existing socio-technical media to produce new participatory systems spanning people, computation, communication and action. These developments could produce profound transformations in health care, community safety, disaster response, life-long learning, business innovation, energy sustainability, environmental protection, and other spheres of important national priorities.

The goal of the workshop is to foster a discussion of these topics, including:

Theoretical integration; Shareable Infrastructure, ethics, and protections; Social capital, social intelligence, and effective action; Design to motivate participation; Graduate Training; and – Unique challenges for government use of social media.

The panel consisted of Ben Shneiderman (U. Maryland), Amy Bruckman (Georgia Tech), Bernardo Huberman (HP Labs), and  Cameron Marlow (Facebook). Panelists argued that “it is our responsibility to shape the conversations on social media” (Ben), that “We want to build technology to enable communities to police themselves” (Cameron), and “Different spaces need different rules” (Amy).

Meanwhile, in Old Europe…

Continue Reading

The topic of implications of federal funding for research was brought up again recently in this Federal Register notice. The Office of Science and Technology Policy wants to receive public comment on a range of issues related to access to academic publications that were funded by Federal grants. The notice mentions the NIH model

One potential model, implemented by the National Institutes of Health (NIH) pursuant to Division G, Title II, Section 218 of Pub. L. 110-161 (http://publicaccess.nih.gov/) requires that all investigators funded by the NIH submit an electronic version of their final, peer-reviewed manuscript upon acceptance for publication no later than 12 months after the official date of publication.

and seeks comments on a range of issues regarding how to structure this broader policy, how to make articles available, how to ensure compliance, etc. This notice seems broader than the NSF-specific discussion I wrote about earlier because it appears to apply to all Federal agencies that fund open research.

Continue Reading

Yesterday, Lennart Nacke expressed the desire to act on the suggestion in a blog post I wrote to review the reviewers. So why not? I would like to see if we can collect some data to inform the debate about obtaining quality reviews for conferences such as CHI. The goal is to see if the availability of authors’ ratings on reviews of papers can be used to improve the reviewer selection process and to give direct feedback to reviewers as well.

Continue Reading

I’ve attended many Hypertext conferences in the past, and have spent a number of years at UofT, so it’s a double pleasure to point out that the Hypertext 2010 conference will be held at the University of Toronto June 13-16. The conference will cover topics including social computing, adaptive hypermedia, and hypertext in education and communication. The deadline for submission in January 18, 2010. For more info, please see the CFP.

Continue Reading

In the aftermath of the CHI 2010 PC meeting, we had an interesting discussion of issues related to reviewing and managing the CHI conference submission process. Several interesting approaches to improving the outcomes were discussed, including reinstating mentoring, rating reviews, adding a desk-reject option for some papers, etc. The overall goal is to improve the quality of submissions and the quality of reviews. Simplifying the overall process was also brought up several times.

Continue Reading

Developers have built applications for mobile phones to support a wide swath of activities, but I would argue that there is no better use for a mobile phone than for those tasks that are fundamentally mobile. And what is more mobile than running? While there have been a variety of research projects (such as UbiFit) designed to encourage exercise, I am more interested here in those applications that support folks who’ve already bought in. For us, smart phones that make it easy to track pace, distance, and even elevation (such as RunKeeper, SportsTracker, and MotionXGPS) have been killer apps. Research projects (such as TripleBeat) are also exploring how to increase competition using past personal results as well as results from other users. Other work has explored using shared audio spaces to allow runners to compete over distances.

How else might we use mobile technologies to improve the running experience? Continue Reading