Browse wiki

Jump to: navigation, search
Comparative analysis of clicks and judgments ir evaluation
Abstract Queries and click-through data taken from Queries and click-through data taken from search engine transaction logs is an attractive alternative to traditional test collections, due to its volume and the direct relation to end-user querying. The overall aim of this paper is to answer the question: How does click-through data differ from explicit human relevance judgments in information retrieval evaluation? We compare a traditional test collection with manual judgments to transaction log based test collections- by using queries as topics and subsequent clicks as pseudorelevance judgments for the clicked results. Specifically, we investigate the following two research questions: Firstly, are there significant differences between clicks and relevance judgments. Earlier research suggests that although clicks and explicit judgments show reasonable agreement, clicks are different from static absolute relevance judgments. Secondly, are there significant differences between system ranking based on clicks and based on relevance judgments? This is an open uestion, but earlier research suggests that comparative evaluation in terms of system ranking is remarkably robust. Copyright 2009 ACM. is remarkably robust. Copyright 2009 ACM.
Abstractsub Queries and click-through data taken from Queries and click-through data taken from search engine transaction logs is an attractive alternative to traditional test collections, due to its volume and the direct relation to end-user querying. The overall aim of this paper is to answer the question: How does click-through data differ from explicit human relevance judgments in information retrieval evaluation? We compare a traditional test collection with manual judgments to transaction log based test collections- by using queries as topics and subsequent clicks as pseudorelevance judgments for the clicked results. Specifically, we investigate the following two research questions: Firstly, are there significant differences between clicks and relevance judgments. Earlier research suggests that although clicks and explicit judgments show reasonable agreement, clicks are different from static absolute relevance judgments. Secondly, are there significant differences between system ranking based on clicks and based on relevance judgments? This is an open uestion, but earlier research suggests that comparative evaluation in terms of system ranking is remarkably robust. Copyright 2009 ACM. is remarkably robust. Copyright 2009 ACM.
Bibtextype inproceedings  +
Doi 10.1145/1507509.1507522  +
Has author Jaap Kamps + , Marijn Koolen + , Andrew Trotman +
Has extra keyword Clickthrough data + , Comparative analysis + , Comparative evaluations + , End users + , Relevance judgment + , Research questions + , System rankings + , Test Collection + , Transaction log + , Transaction log analysis + , Information retrieval + , Wikipedia + , Image retrieval + , Information services + , Online searching + , Research + , Search engine + , World Wide Web +
Has keyword Transaction log analysis + , Information retrieval + , Wikipedia +
Isbn 9781605584348  +
Language English +
Number of citations by publication 0  +
Number of references by publication 0  +
Pages 80–87  +
Published in Proceedings of Workshop on Web Search Click Data, WSCD'09 +
Title Comparative analysis of clicks and judgments ir evaluation +
Type conference paper  +
Year 2009 +
Creation dateThis property is a special property in this wiki. 7 November 2014 08:33:14  +
Categories Publications without license parameter  + , Publications without remote mirror parameter  + , Publications without archive mirror parameter  + , Publications without paywall mirror parameter  + , Conference papers  + , Publications without references parameter  + , Publications  +
Modification dateThis property is a special property in this wiki. 7 November 2014 08:33:14  +
DateThis property is a special property in this wiki. 2009  +
hide properties that link here 
Comparative analysis of clicks and judgments ir evaluation + Title
 

 

Enter the name of the page to start browsing from.