Browse wiki

Jump to: navigation, search
Overview of ResPubliQA 2009: Question answering evaluation over European legislation
Abstract This paper describes the first round of ReThis paper describes the first round of ResPubliQA, a Question Answering (QA) evaluation task over European legislation, proposed at the Cross Language Evaluation Forum (CLEF) 2009. The exercise consists of extracting a relevant paragraph of text that satisfies completely the information need expressed by a natural language question. The general goals of this exercise are (i) to study if the current QA technologies tuned for newswire collections and Wikipedia can be adapted to a new domain (law in this case); (ii) to move to a more realistic scenario, considering people close to law as users, and paragraphs as system output; (iii) to compare current QA technologies with pure Information Retrieval (IR) approaches; and (iv) to introduce in QA systems the Answer Validation technologies developed in the past three years. The paper describes the task in more detail, presenting the different types of questions, the methodology for the creation of the test sets and the new evaluation measure, and analyzing the results obtained by systems and the more successful approaches. Eleven groups participated with 28 runs. In addition, we evaluated 16 baseline runs (2 per language) based only in pure IR approach, for comparison purposes. Considering accuracy, scores were generally higher than in previous QA campaigns.ally higher than in previous QA campaigns.
Abstractsub This paper describes the first round of ReThis paper describes the first round of ResPubliQA, a Question Answering (QA) evaluation task over European legislation, proposed at the Cross Language Evaluation Forum (CLEF) 2009. The exercise consists of extracting a relevant paragraph of text that satisfies completely the information need expressed by a natural language question. The general goals of this exercise are (i) to study if the current QA technologies tuned for newswire collections and Wikipedia can be adapted to a new domain (law in this case); (ii) to move to a more realistic scenario, considering people close to law as users, and paragraphs as system output; (iii) to compare current QA technologies with pure Information Retrieval (IR) approaches; and (iv) to introduce in QA systems the Answer Validation technologies developed in the past three years. The paper describes the task in more detail, presenting the different types of questions, the methodology for the creation of the test sets and the new evaluation measure, and analyzing the results obtained by systems and the more successful approaches. Eleven groups participated with 28 runs. In addition, we evaluated 16 baseline runs (2 per language) based only in pure IR approach, for comparison purposes. Considering accuracy, scores were generally higher than in previous QA campaigns.ally higher than in previous QA campaigns.
Bibtextype inproceedings  +
Doi 10.1007/978-3-642-15754-7_21  +
Has author Penas A. + , Forner P. + , Sutcliffe R. + , Rodrigo A. + , Forascu C. + , Iñaki Alegria + , Giampiccolo D. + , Moreau N. + , Osenova P. +
Has extra keyword Cross-language evaluation forums + , European legislation + , Evaluation measures + , Information need + , Natural language questions + , QA system + , Question answering + , Question-answering evaluation + , Realistic scenario + , System output + , Test sets + , Validation technologies + , Wikipedia + , Linguistics + , Natural language processing systems + , Information retrieval +
Isbn 364215753X; 9783642157530  +
Language English +
Number of citations by publication 0  +
Number of references by publication 0  +
Pages 174–196  +
Published in Lecture Notes in Computer Science +
Title Overview of ResPubliQA 2009: Question answering evaluation over European legislation +
Type conference paper  +
Volume 6241 LNCS  +
Year 2010 +
Creation dateThis property is a special property in this wiki. 8 November 2014 03:20:39  +
Categories Publications without keywords parameter  + , Publications without license parameter  + , Publications without remote mirror parameter  + , Publications without archive mirror parameter  + , Publications without paywall mirror parameter  + , Conference papers  + , Publications without references parameter  + , Publications  +
Modification dateThis property is a special property in this wiki. 8 November 2014 03:20:39  +
DateThis property is a special property in this wiki. 2010  +
hide properties that link here 
Overview of ResPubliQA 2009: Question answering evaluation over European legislation + Title
 

 

Enter the name of the page to start browsing from.