Methods for Evaluating Interactive Information Retrieval Systems with Users
By Diane Kelly, School of Information and Library Science, University of North Carolina at Chapel Hill, USA, dianek@email.unc.edu
Abstract
This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.
Methods for Evaluating Interactive Information Retrieval Systems with Users
Methods for Evaluating Interactive Information Retrieval Systems with Users provides a foundation on which those new to Interactive Information Retrieval (IIR) can make more informed choices about how to design and conduct IIR evaluations with human subjects. The primary goal is to catalog and compile material related to the IIR evaluation method into a single source. Methods for Evaluating Interactive Information Retrieval Systems with Users begins by providing historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems. It proceeds to describe the major components of interactive information retrieval system evaluation and goes on to describe different experimental designs and sampling strategies. From there, it presents core instruments and data collection techniques and measures, explains basic data analysis techniques and reviews and discusses previous studies. It also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval. It concludes with a discussion of outstanding challenges and future research directions.
