{"id":1802,"date":"2009-09-11T07:02:58","date_gmt":"2009-09-11T14:02:58","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=1802"},"modified":"2009-09-10T23:19:30","modified_gmt":"2009-09-11T06:19:30","slug":"perhaps-they-measured-the-wrong-thing","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=1802","title":{"rendered":"Perhaps they measured the wrong thing…"},"content":{"rendered":"

Ian Soboroff commented<\/a> on yesterday’s blog post<\/a> that although mental models were important, they were insufficient. He cited a paper <\/a>that found that legal staff had experienced problems with using a full-text search engine to search (with a recall-oriented information need) a collection of documents in a legal discovery scenario. The paper concludes that coming up with effective keyword searches is difficult for non-search experts. The paper is interesting and worth reading, but I believe the authors conclusions are not warranted by their methodology.<\/p>\n

The paper is interesting in that describes an experimental setup in which paralegals and attorneys collaborated in an information seeking task: the attorneys would specify information needs, the paralegals would do searches, and the attorneys would then evaluate the results for relevance; they were free to iterate until they thought they had found a query that identified 75% of the ‘vital’, ‘satisfactory’, and ‘marginally relevant’ documents. This setup was meant to simulate a realistic work scenario where groups of people worked on a recall-oriented, exploratory search.<\/p>\n

It turned out, however, that only about 20% of relevant articles were identified through this method. In their analysis, the authors argued that it is unrealistic to expect users without much training to find relevant documents using disjunctions of keywords when searching over large collections. But the authors’ analysis is flawed because in realistic scenarios, single queries are not expected to achieve high recall; rather the idea is that many queries run in an interactive, iterative manner will identify small clusters of relevant documents, the union of which may produce high recall. Ironically, the authors describe an elaborate, multi-step process they used to approximate the number of relevant documents. The process involved learning which vocabulary was useful for identifying some relevant documents, and learning new vocabulary from those documents. This, of course, is precisely how exploratory search is performed. So it is the expectation that people would be effective at this task without engaging in such a process that is unrealistic.<\/p>\n

The way the study structured people’s interactions with information fostered an inappropriate mental model in the users. Rather than expecting users to construct single high-precision, high-recall queries, the users should have been encouraged to be more interactive and exploratory. My sense is a more transparent approach would yield better results:<\/p>\n