I have a question. It’s a general question, directed at anyone and everyone.
When one is building an Information Retrieval system, one uses target objective function(s) that give an indication of the performance of the system, and designs the system (algorithms, interfaces, etc.) toward those targets. Sometimes, those functions are open and well understood. Other times, those functions are proprietary and hidden.
My question is: Does it do the users of an IR system a service or disservice to hide from them the function that is being optimized? Or is it completely neutral? In other words, does the user have to understand, or at least be given the chance to understand, what it is that the system is trying to do for them in order to get the best value out of that system? Or can a user get results just as good without having to have a clear mental model of what the retrieval engine is trying to do? In short, does it matter if the user does not understand what the system is trying to do for him or her?
Can someone point me to research that may have looked at this question? If one were trying to publish original research on the topic, how would one go about designing an experiment in which both (1) this hypothesis is tested, and (2) done so in a way that generalizes, or at least hints at possible generalization?