{"id":5343,"date":"2011-11-10T05:24:18","date_gmt":"2011-11-10T13:24:18","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=5343"},"modified":"2011-11-10T07:14:46","modified_gmt":"2011-11-10T15:14:46","slug":"hcir-2011-keynote","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=5343","title":{"rendered":"HCIR 2011 keynote"},"content":{"rendered":"

HCIR 2011<\/a> took place almost three weeks ago, but I am just getting caught up after a week at CIKM 2011 and an actual almost-no-internet-access vacation. I wanted to start off my reflections on HCIR with a summary of Gary Marchionini<\/a>‘s keynote,\u00a0titled \u201cHCIR: Now the Tricky Part.\u201d Gary\u00a0coined<\/a> the term \u201cHCIR\u201d and has been a persuasive advocate of the concepts represented by the term.\u00a0The talk used three case studies of HCIR projects as a lens to focus the audience\u2019s attention on one of the main challenges of HCIR: how to evaluate the systems we build.<\/p>\n

The case study of the Open Video<\/a> introduced the notion of surrogates, which are representations of metadata designed for human consumption. One significant challenge, particularly for video objects, is to how to evaluate the effectiveness of surrogates. While most evaluations are done as laboratory studies, do these findings retain their validity in naturalistic settings?<\/p>\n

The second case study focused on the Relation Browser<\/a>, a faceted browsing tool. The Relation Browser is designed to reveal relationships across facets, and supports exploration through browsing rather than keyword search. This is a system that has undergone many revisions and redesigns over a decade. The lab\u2019s experience with the project raised several important issues that are applicable much more broadly: how does one capture the design rationale that goes into evolving the system over time? How to show the benefits of HCIR interfaces when users prefer interfaces with which they are already familiar?<\/p>\n

The third case study, based on the on-going Results Space<\/a> project, explored HCIR issues related to surrogates (are there better ways to represent search results than lists of snippets), and touched on issues of awareness in collaborative search.<\/p>\n

Having presented the case studies, Gary raised some broader issues for the HCIR field. While he admitted that most of the work that he has been involved with focused on lab studies, he recognized the need for field observations as well. In fact, he suggested that neither approach would be truly useful in isolation. A process that starts with qualitative observations to generate hypotheses that are then tested in the lab is one effective way to integrate the methods, but Gary also raised the possibility of proceeding in the opposite direction.<\/p>\n

He also identified the following challenges:<\/p>\n

1.\tQuery quality: can we assess the quality of a query, which is often the first signal of user behavior? Can this be done pre-retrieval as well as post-retrieval? How can we elicit human judgments of query quality?<\/p>\n

2.\tBehavior as evidence: what are the various ways in which searchers\u2019 behavior can be converted into implicit relevance feedback? Can we match behaviors to queries to infer what people might be interested in?<\/p>\n

3.\tHow do we design and evaluate surrogates that help people perceive, recognize, gist, understand, interpret, analyze and evaluate search results?<\/p>\n

4.\tHow do the tools we build affect searchers\u2019 cognitive load? As we add more tools, are we helping or hurting? Does collaboration simply increase the load further without offering tangible benefits, or is there a useful tradeoff?<\/p>\n

5.\tHow do we measure session quality and search quality? Recall and precision are point-based measures that don\u2019t capture the process characteristic of session-based search. How do we adapt them to the session as a unit of interaction (as opposed to a query)? What other metrics are appropriate?<\/p>\n

6.\tFinally, he touched on the tricky issue of recording and capturing experimental traces: annotated logs, video recordings, data sets, etc. Should researchers publish traces of system use to characterize participants\u2019 search activities?<\/p>\n

The talk also included an interactive interlude, in which Gary showed the audience the following graph:<\/p>\n

\"Slide<\/a>

Slide from Gary Marchionini's HCIR 2011 keynote<\/p><\/div>\n

The vertical axis was some kind of performance metric, and the horizontal axis represented queries over time. The dashed green line shows steady progress from lower-left to upper-right, while the solid blue line takes a more erratic route between the same starting and ending point. Gary\u2019s question to the audience was which line represented better, or perhaps more desirable, system performance. A number of suggestions were offered, including the benefits of steady progress, of relieving frustration though surprise (the aha! moment reflected in a sharp upward spike of the erratic curve), of achieving maximum performance if only for a short time, etc. There was some consensus that the line that went up and down reflected a learning process that is often characteristic of exploratory search. Which one represents more desirable performance, however, may well depend on the task and the measure selected.<\/p>\n

In short, the talk gave a great summary of some accomplishments of HCIR research and of some of the challenges ahead. We expect that next year\u2019s presentations will offer some insights into these thorny issues!<\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

HCIR 2011 took place almost three weeks ago, but I am just getting caught up after a week at CIKM 2011 and an actual almost-no-internet-access vacation. I wanted to start off my reflections on HCIR with a summary of Gary Marchionini‘s keynote,\u00a0titled \u201cHCIR: Now the Tricky Part.\u201d Gary\u00a0coined the term \u201cHCIR\u201d and has been a […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[94,295],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/5343"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5343"}],"version-history":[{"count":11,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/5343\/revisions"}],"predecessor-version":[{"id":5357,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/5343\/revisions\/5357"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5343"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5343"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5343"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}