{"id":1316,"date":"2009-07-10T07:50:15","date_gmt":"2009-07-10T14:50:15","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=1316"},"modified":"2009-07-13T10:53:58","modified_gmt":"2009-07-13T17:53:58","slug":"blueprint-for-information-seeking-evaluation","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=1316","title":{"rendered":"Blueprint for information seeking evaluation"},"content":{"rendered":"

I dodged being empaneled on a jury, and I made it to IBM Almaden to attend most of NPUC<\/a>. I did miss the talk by Brad Myers<\/a>, which I’ll have to view on video<\/a>, but got to see most of the other presentations and the poster\/demo session. One demo I found particularly interesting was Mira Dontcheva<\/a>‘s Blueprint<\/a> work. Blueprint is an Eclipse plugin for Flex programming that allows people to search for snippets of code directly from the IDE, and displays them in an overlay or side bar. Blueprint makes it possible to search the web with an interface similar to the typical auto-complete functionality. Furthermore, because it understands Flex syntax, its ranking should more accurate than a regular full-text index that happens to contain code. When you select a search result, Blueprint inserts it into your code, and automatically annotates the code to include the URL at which the snippet was found so that you can re-visit that page later.<\/p>\n

\"Adobe<\/a>

Adobe Blueprint screenshot<\/p><\/div>\n

This is a great interface for those of us used to Programming By Google, an interesting example of search that is integrated into a larger task, and is also a great opportunity for information seeking evaluation:\u00a0 first of all,\u00a0 you have real people performing real tasks. On top of that, (with suitable instrumentation and opt-in) you can measure how many queries people ran each time they used the search feature, and you can analyze the code that’s being created to see if any of the search results were incorporated either directly (through the built-in paste operation), or indirectly, through reuse of identifiers in the retrieved code. Such an experiment should indicate not only the usability of the approach (compared, for example, with searching in a browser), but also provide evidence that can be used to tweak the ranking algorithm.<\/p>\n

The above experiment, coupled with interviews of participants, should be able to answer all of\u00a0William Hersh’s<\/a> evaluation questions that I mentioned in an earlier post<\/a>, particularly the often hard-to assess “Did the system have an impact?” In short, a golden opportunity!<\/p>\n

Update<\/strong>: Mira pointed out to me that this work is part of Joel Brandt<\/a>‘s thesis work.<\/p>\n","protected":false},"excerpt":{"rendered":"

I dodged being empaneled on a jury, and I made it to IBM Almaden to attend most of NPUC. I did miss the talk by Brad Myers, which I’ll have to view on video, but got to see most of the other presentations and the poster\/demo session. One demo I found particularly interesting was Mira […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[15],"tags":[91],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1316"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1316"}],"version-history":[{"count":18,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1316\/revisions"}],"predecessor-version":[{"id":1388,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/1316\/revisions\/1388"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1316"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1316"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1316"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}