Daniel Tukelang wrote an interesting review/commentary on Clifford Nass and Corina Yen‘s new book on affective computing, where they cite many examples that biasing results toward one’s expectations can improve users’ satisfaction with the results. Another class of responses (that is also well-documented in the affective computing literature) is the tendency for people to anthropomorphize computers.
Daniel’s conclusion is that it’s relatively straightforward to use these techniques to deceive people, to subvert personalization, to mislead rather that to inform. I’ve got two reactions to this work, one related to system design, and one more specifically to information seeking.
First, from a designerly perspective, I’ve always been suspicious of the logic that says “people do X, therefore we should design systems to do X.” While that may be the easy solution, it may not be the responsible one. If we know, as designers, that people tend to anthropomorphize inanimate objects, and that leads them to have inappropriate mental models, we should strive to mitigate or prevent those tendencies. This may be easier to do with a hammer that does not have independent behavior compared with a computer, but that doesn’t mean that we shouldn’t try.
From an information retrieval perspective, these results bring into question the validity of user satisfaction as a meaningful measure of human-computer system performance. If the manner in which data are presented can affect people’s reactions to the data, those reactions may not be meaningful outside the context of the presentation.
Here’s a simple experiment I’d like to run:
- Create an interface through which people search for information.
- Give a participant a topic on which they will be expected to search, and have them study the topic.
- Ask them how many documents they expect to find that are pertinent to the topic.
- Next, have them type a query to retrieve those pertinent documents.
- Now comes the experimental manipulation: you have three conditions in which you return results to them:
- Return fewer than expected pertinent documents
- Return about as many as they expected
- Return many more than they expected
- Now, assess their satisfaction with the system.
- If you have enough subjects, you can run another factor that does not prime them with their expectations
If this experiment produces statistically-significant results that show that biasing a person’s expectations impacts satisfaction, that may in fact be used for good: if it’s possible to assess the likelihood that query results are poor before showing the results, then the interface might use some means to warn the person before showing the results. The nature of the warning depends very much on the expected users, on their tasks, and on how much else is known about them by the system. But my guess is that given an assessment of query quality, an up-front indication of that might improve satisfaction with the system.
I don’t know if this will work as I image, but it’s nice to think that persuasive techniques can be used for the benefit of the user, not just for insidious purposes. Of course there is more money in the latter…
Setting appropriate expectations seems like a good idea for optimizing both effectiveness and satisfaction. But I’m not sure I understand the proposed experiment. Is the goal to test whether biasing a person’s expectations impacts satisfaction in general (the literature seems pretty conclusive in the affirmative), or whether users have strong expectations about the number of results?
But I share your guess/hope that a reasonably accurate up-front indication of estimated result quality would help set more appropriate user expectations and thus improve user satisfaction with the system.
The goal of the experiment is to show specifically that the “user satisfaction” measure reported in some papers doesn’t reflect as much about system performance but rather the degree to which a user’s expectations were met.
There is also the correlated issue of confirmation bias — that users might be more satisfied with results that they expected to find rather than with something different. This might be particularly problematic in medical searches for lay people who may not be able to judge the medical applicability of the results, and may be biased to select overly-optimistic (or overly-pessimistic) results.
I agree that system performance should be judged more objectively, e.g., based on users’ effectiveness and efficiency at completing tasks. User satisfaction matters, but it’s certainly not equivalent to system performance. Ideally systems should optimize for both. I’m curious if anyone has explored the space of trade-offs.
One context in which I experienced user preferences trumping utility is in text to speech synthesis. At SpeechWorks, we were using the old-school segmental synthesizers, like DECTalk, because they were built into the telephony platforms. Then we acquired licenses to AT&T’s demi-phone synthesizers. The demi-phone models sounded much better if you didn’t try to understand them. But for pure utility of being able to transcribe, say, a set of street directions from MapQuest, the demi-phone models were useless.
No one believed this. They could hear how much better the demi-phone synthesis sounded compared to the robotic DECTalk.
This is also reminding me of the later post on Google Instant!