Falk Scholer - Measuring Search Effectiveness: Bringing the User Back

As computer scientists, many of the systems and applications that we build are ultimately intended to be used by people to complete real world tasks. However, the experimental evaluation of what makes a system "good" can be difficult, and is sometimes not related to the end user. This talk will examine the effectiveness measurement of information retrieval systems, and demonstrate that the main evaluation methodology, using test collections, can lead to conclusions that don't match task-based outcomes of end users.

I will then present recent work that investigates some potential causes of the mismatch between experimental evaluation and end users. First, we will consider user variability, and in particular the impact that different search query formulations of the same underlying information need can have on measurement accuracy. Second, we will examine the issue of relevance perceptions and the technique of magnitude estimation, an approach from the field of psychophysics that aims to directly measure subjective user experiences.