Max Wilson recently commented on an article by Zhang et al on time series analysis of search logs.  (Thanks to Daniel for the web link.) This is a topic I’ve been interested in for a while, in particular for finding evidence of exploratory search in the logs.  Max notes that the average session length is just under three interactions, and concludes that this really amounts to a single query/selection interaction. If that’s so, then the average behavior  characterized in the Zhang paper is not in fact exploratory.

I was looking forward to reading the paper, but my excitement was short-lived:

• The abstract is wrong!  It claims that “results show that the average length of a searcher session is approximately 2.9 interactions” whereas what they are really measuring is the average number of query terms. How disappointing! Given that the authors had access to the IP address and cookie information associated with each query, they could have estimated session length, calculated the range of terms (and how many terms were repeated), and derived all sorts of other useful statistics about users’ behavior.
• The authors find that the rank of lowest-selected document ranges mostly from five to eight, meaning that searchers typically don’t get off the first page of results. This is not surprising for cognitive reasons. In fact, in my PhD research, I found that increasing the number of search results displayed per page increased the subset of the results set viewed by searchers (abstract, full paper).
• The correlation analysis suggested that for informational searches, “searchers who typed in the fewest query terms one period ahead (i.e., less than the average query length) were more likely to click higher ranked links (i.e., top ranked results) in the following period.”  This result is characteristic of naive searchers, or perhaps the authors’ classification of informational vs. navigational searches was not sufficiently accurate. The authors do not report any analysis of query terms, so it is difficult to know if there is a relationship between the types of search terms used and the exhibited search behaviors.

I hope that the authors revisit their analysis with an eye for capturing multiple metrics of users’ behaviors, and looks for patterns that indicate a diversity of searching styles and tactics. Another direction to take would be to classify search terms in some manner (e.g., part of speech, proper name, url, etc.) to see if these classifications predict behavior. Such a result could be operationalized by, for example, adjusting search interfaces based on a classification of query terms.

I tried to find the editor responsible for the issue in which this article appears, but could not find a reliable reference to it. The pre-print lists it as being published in 2008, but no issue of IP&M published in 2008 listed that article in its table of contents. I hope that the editors catch the error and at least have the authors edit the abstract—I would hate to pay money for this based on the abstract, only to have the article offer a completely different analysis.

Share on: