Evaluation

From WikiPapers
Jump to: navigation, search

evaluation is included as keyword or extra keyword in 0 datasets, 0 tools and 31 publications.

Datasets

There is no datasets for this keyword.

Tools

There is no tools for this keyword.


Publications

Title Author(s) Published in Language DateThis property is a special property in this wiki. Abstract R C
Supporting navigation in Wikipedia by information visualization: Extended evaluation measures Wu I.-C.
Vakkari P.
Journal of Documentation English 2014 Purpose: The authors introduce two semantics-based navigation applications that facilitate information-seeking activities in internal link-based web sites in Wikipedia. These applications aim to help users find concepts within a topic and related articles on a given topic quickly and then gain topical knowledge from internal link-based encyclopedia web sites. The paper aims to discuss these issues. Design/methodology/approach: The WNavis application consists of three information visualization (IV) tools which are a topic network, a hierarchy topic tree and summaries for topics. The WikiMap application consists of a topic network. The goal of the topic network and topic tree tools is to help users to find the major concepts of a topic and identify relationships between these major concepts easily. In addition, in order to locate specific information and enable users to explore and read topic-related articles quickly, the topic tree and summaries for topics tools support users to gain topical knowledge quickly. The authors then apply the k-clique of cohesive indicator to analyze the sub topics of the seed query and find out the best clustering results via the cosine measure. The authors utilize four metrics, which are correctness, time cost, usage behaviors, and satisfaction, to evaluate the three interfaces. These metrics measure both the outputs and outcomes of applications. As a baseline system for evaluation the authors used a traditional Wikipedia interface. For the evaluation, the authors used an experimental user study with 30 participants. Findings: The results indicate that both WikiMap and WNavis supported users to identify concepts and their relations better compared to the baseline. In topical tasks WNavis over performed both WikiMap and the baseline system. Although there were no time differences in finding concepts or answering topical questions, the test systems provided users with a greater gain per time unit. The users of WNavis leaned on the hierarchy tree instead of other tools, whereas WikiMap users used the topic map. Research limitations/implications: The findings have implications for the design of IR support tools in knowledge-intensive web sites that help users to explore topics and concepts. Originality/value: The authors explored to what extent the use of each IV support tool contributed to successful exploration of topics in search tasks. The authors propose extended task-based evaluation measures to understand how each application provides useful context for users to accomplish the tasks and attain the search goals. That is, the authors not only evaluate the output of the search results, e.g. the number of relevant items retrieved, but also the outcome provided by the system for assisting users to attain the search goal. 0 0
Evaluation of WikiTalk - User studies of human-robot interaction Anastasiou D.
Kristiina Jokinen
Graham Wilcock
Lecture Notes in Computer Science English 2013 The paper concerns the evaluation of Nao WikiTalk, an application that enables a Nao robot to serve as a spoken open-domain knowledge access system. With Nao WikiTalk the robot can talk about any topic the user is interested in, using Wikipedia as its knowledge source. The robot suggests some topics to start with, and the user shifts to related topics by speaking their names after the robot mentions them. The user can also switch to a totally new topic by spelling the first few letters. As well as speaking, the robot uses gestures, nods and other multimodal signals to enable clear and rich interaction. The paper describes the setup of the user studies and reports on the evaluation of the application, based on various factors reported by the 12 users who participated. The study compared the users' expectations of the robot interaction with their actual experience of the interaction. We found that the users were impressed by the lively appearance and natural gesturing of the robot, although in many respects they had higher expectations regarding the robot's presentation capabilities. However, the results are positive enough to encourage research on these lines. 0 0
Predicting user tags using semantic expansion Chandramouli K.
Piatrik T.
Izquierdo E.
Communications in Computer and Information Science English 2012 Manually annotating content such as Internet videos, is an intellectually expensive and time consuming process. Furthermore, keywords and community-provided tags lack consistency and present numerous irregularities. Addressing the challenge of simplifying and improving the process of tagging online videos, which is potentially not bounded to any particular domain, we present an algorithm for predicting user-tags from the associated textual metadata in this paper. Our approach is centred around extracting named entities exploiting complementary textual resources such as Wikipedia and Wordnet. More specifically to facilitate the extraction of semantically meaningful tags from a largely unstructured textual corpus we developed a natural language processing framework based on GATE architecture. Extending the functionalities of the in-built GATE named entities, the framework integrates a bag-of-articles algorithm for effectively searching through the Wikipedia articles for extracting relevant articles. The proposed framework has been evaluated against MediaEval 2010 Wild Wild Web dataset, which consists of large collection of Internet videos. 0 0
SIGA, a system to manage information retrieval evaluations Costa L.
Mota C.
Diana Santos
Lecture Notes in Computer Science English 2012 This paper provides an overview of the current version of SIGA, a system that supports the organization of information retrieval (IR) evaluations. SIGA was recently used in Págico, an evaluation contest where both automatic and human participants competed to find answers to 150 topics in the Portuguese Wikipedia, and we describe its new capabilities in this context as well as provide preliminary results from Págico. 0 0
Tool-supported assessment of wiki-based assignments Kubincova Z.
Homola M.
Janajev R.
CSEDU 2012 - Proceedings of the 4th International Conference on Computer Supported Education English 2012 Integrating wiki-based activities into education encourages development of students' competencies that are important for their future professional life. As wikis support social learning and interaction, when working with them students learn to cooperate, plan and organize collaborative tasks, create concepts, express ideas, etc. Many teachers recognize undeniable benefits of a wiki in learning and try to involve it in their teaching. Since wikis were developed for collaborative work but not for use in education, they typically do not include proper tools for tracking and assessing the students' activities, which makes the evaluation difficult for the teacher. In this paper we present a tracking and assessment tool, which we have proposed and developed and report on our experience with evaluation of wiki-based assignments using this tool as well. Since the evaluation of wiki-based assignments is a nontrivial problem not only from the technical point of view but also from the point of methodology, the assessment methodology is also discussed here. 0 0
A empirical study on application of Wiki-based collaborative lesson-preparing Yingjie Ren
Chaohua Gong
Proceedings - 2011 International Conference of Information Technology, Computer Engineering and Management Sciences, ICM 2011 English 2011 Lesson-preparing is an important stage in the field of teaching activity. The aim of this paper was to explore the use of Eduwiki as a new effective collaborative lessonpreparing platform to support teachers' collaboration and teaching. Furthermore, to verify and explore how to integrate Eduwiki into teachers' daily lesson-preparing activities, a single-group post-test and interview were used in the experiments. The study showed that Eduwiki was effective in motivating peer-supported collaborative lesson-preparing activity, as well as for teachers' mutual development. School leaders' support was the first important motivator for implementing the experiments in Expriment1 and interested the collaboration support environment was the first important motivators for implementing the experiments in Experiment2. The external condition for teachers participating collaborative lesson-preparing was very easy for operation. It showed that those experienced teachers passed their experiences on to novices using Eduwiki, and made the novices achieved high performance by collaborative lesson-preparing. 0 0
Empirical Study on Application of Wiki Based Collaborative Lesson-Preparing Yingjie Ren
Chaohua Gong
ICM English 2011 0 0
Evaluation of OML and AERMOD Olesen H.R.
Berkowicz R.
Lofstrom P.
International Journal of Environment and Pollution English 2011 Results from an evaluation of three dispersion models are presented: the currently operational OML model, a new, improved 'Research Version' of OML, and the US AERMOD model. The evaluation is based on the Prairie Grass data set. For these data the OML Research Version appears superior to the other two models. Further, the paper discusses problems and pitfalls of the Prairie Grass data set. The criteria for exclusion of data have tremendous impact on evaluation metrics. A new Wiki on Atmospheric Dispersion has the potential to become a very useful focal point to pool and communicate experiences on data sets such as Prairie Grass. Copyright 0 0
The victorian web and the victorian course wiki: comparing the educational effectiveness of identical assignments in web 1.0 and web 2.0 George P. Landow HT English 2011 0 0
Towards identifying arguments in Wikipedia pages Hoda Sepehri Rad
Denilson Barbosa
World Wide Web English 2011 0 0
Wikipedia: A key tool for global public health promotion Heilman J.M.
Kemmann E.
Bonert M.
Chatterjee A.
Ragar B.
Beards G.M.
Iberri D.J.
Harvey M.
Thomas B.
Stomp W.
Martone M.F.
Lodge D.J.
Vondracek A.
De Wolff J.F.
Liber C.
Grover S.C.
Vickers T.J.
Mesko B.
Laurent M.R.
Journal of Medical Internet Research English 2011 The Internet has become an important health information resource for patients and the general public. Wikipedia, a collaboratively written Web-based encyclopedia, has become the dominant online reference work. It is usually among the top results of search engine queries, including when medical information is sought. Since April 2004, editors have formed a group called WikiProject Medicine to coordinate and discuss the English-language Wikipedia's medical content. This paper, written by members of the WikiProject Medicine, discusses the intricacies, strengths, and weaknesses of Wikipedia as a source of health information and compares it with other medical wikis. Medical professionals, their societies, patient groups, and institutions can help improve Wikipedia's health-related entries. Several examples of partnerships already show that there is enthusiasm to strengthen Wikipedia's biomedical content. Given its unique global reach, we believe its possibilities for use as a tool for worldwide health promotion are underestimated. We invite the medical community to join in editing Wikipedia, with the goal of providing people with free access to reliable, understandable, and up-to-date health information. 0 1
Anomalies in ontologies with rules Joachim Baumeister
Seipel D.
Journal of Web Semantics English 2010 For the development of practical semantic applications, ontologies are commonly used with rule extensions. Prominent examples of semantic applications not only are Semantic Wikis, Semantic Desktops, but also advanced Web Services and agents. The application of rules increases the expressiveness of the underlying knowledge in many ways. Likewise, the integration not only creates new challenges for the design process of such ontologies, but also existing evaluation methods have to cope with the extension of ontologies by rules. Since the verification of Owl ontologies with rule extensions is not tractable in general, we propose to verify ontologies at the symbolic level by using a declarative approach: With the new language Datalog{star operator}, known anomalies can be easily specified and tested in a compact manner. We introduce supplements to existing verification techniques to support the design of ontologies with rule enhancements, and we focus on the detection of anomalies that especially occur due to the combined use of rules and ontological definitions. © 2010 Elsevier B.V. All rights reserved. 0 0
Crowdsourcing a Wikipedia Vandalism Corpus Martin Potthast SIGIR English 2010 We report on the construction of the PAN Wikipedia vandalism corpus, PAN-WVC-10, using Amazon’s Mechanical Turk. The corpus compiles 32 452 edits on 28 468 Wikipedia articles, among which 2 391 vandalism edits have been identified. 753 human annotators cast a total of 193 022 votes on the edits, so that each edit was reviewed by at least 3 annotators, whereas the achieved level of agreement was analyzed in order to label an edit as “regular” or “vandalism.” The corpus is available free of charge. 6 1
Exploiting click-through data for entity retrieval Billerbeck B.
Gianluca Demartini
Firan C.S.
Tereza Iofciu
Ralf Krestel
SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval English 2010 We present an approach for answering Entity Retrieval queries using click-through information in query log data from a commercial Web search engine. We compare results using click graphs and session graphs and present an evaluation test set making use of Wikipedia "List of" pages. 0 0
Exploring the workplace communication ecology Turner T.
Qvarfordt P.
Biehl J.T.
Golovchinsky G.
Back M.
Conference on Human Factors in Computing Systems - Proceedings English 2010 The modern workplace is inherently collaborative, and this collaboration relies on effective communication among co-workers. Many communication tools - email, blogs, wikis, Twitter, etc. - have become increasingly available and accepted in workplace communications. In this paper, we report on a study of communications technologies used over a one year period in a small US corporation. We found that participants used a large number of communication tools for different purposes, and that the introduction of new tools did not impact significantly the use of previously-adopted technologies. Further, we identified distinct classes of users based on patterns of tool use. This work has implications for the design of technology in the evolving ecology of communication tools. 0 0
How geographic was GikiCLEF? A GIR-critical review Diana Santos
Nuno Cardoso
Cabral L.M.
Proceedings of the 6th Workshop on Geographic Information Retrieval, GIR'10 English 2010 In this paper we draw a balance of GikiCLEF as far as its appropriateness for the evaluation of GIR systems is concerned. We measure its degree of dealing with geographic matter, and offer GIRA, the final resource, for GIR evaluation purposes. Copyright 2010 ACM. 0 0
Overview of the INEX 2009 link the Wiki track Huang W.C.
Shlomo Geva
Andrew Trotman
Lecture Notes in Computer Science English 2010 In the third year of the Link the Wiki track, the focus has been shifted to anchor-to-bep link discovery. The participants were encouraged to utilize different technologies to resolve the issue of focused link discovery. Apart from the 2009 Wikipedia collection, the Te Ara collection was introduced for the first time in INEX. For the link the wiki tasks, 5000 file-to-file topics were randomly selected and 33 anchor-to-bep topics were nominated by the participants. The Te Ara collection does not contain hyperlinks and the task was to cross link the entire collection. A GUI tool for self-verification of the linking results was distributed. This helps participants verify the location of the anchor and bep. The assessment tool and the evaluation tool were revised to improve efficiency. Submission runs were evaluated against Wikipedia ground-truth and manual result set respectively. Focus-based evaluation was undertaken using a new metric. Evaluation results are presented and link discovery approaches are described. 0 0
Overview of the INEX 2009 link the wiki track Wei Che Huang
Shlomo Geva
Andrew Trotman
INEX English 2010 0 0
Upgrading a coldfusion-based academic medical library staff intranet Vander Hart R.
Ingrassia B.
Mayotte K.
Palmer L.A.
John Powell
Journal of Web Librarianship English 2010 This article details the process of upgrading and expanding an existing academic medical library intranet to include a wiki, blog, discussion forum, and photo collection manager. The first version of the library's intranet from early 2002 was powered by ColdFusion software and existed primarily to allow staff members to author and store minutes of library team meetings. Other ColdFusion-based applications and functions were subsequently added, as were various other library documents and procedures. As a follow-up to the library's strategic plan, a library Staff Intranet Team was organized in early 2008 to reorganize the content of the intranet and to identify software tools that would allow greater staff participation in maintaining and updating intranet content. Early steps in the process included brainstorming, a card-sorting exercise, product research, a staff survey, and paper prototyping. The team focused on implementing various open-source, ColdFusion-based tools in order to accommodate existing technology, available budget, and time constraints. Challenges in implementing the tools included bypassing or modifying existing authentication systems and applying modifications that led to loss of native functionality. Despite usability testing and staff training, library staff have not universally welcomed or adopted all the new tools. Notwithstanding these challenges, the renovated staff intranet has shown promise in furthering the goals in the library's strategic plan to improve communication and facilitate collaboration among library staff. 0 0
When the new application smell is gone: Traditional intranet best practices and existing Web 2.0 intranet infrastructures Yoose B. Journal of Web Librarianship English 2010 With the growth of Web 2.0 library intranets in recent years, many libraries are leaving behind legacy, first-generation intranets. As Web 2.0 intranets multiply and mature, how will traditional intranet best practices-especially in the areas of planning, implementation, and evaluation-translate into an existing Web 2.0 intranet infrastructure? This article explores traditional intranet theories and best practices in relation to the two generations of the Web 2.0 intranet at the Technical Services Department at the Miami University Libraries. The case study explores the evaluation of the first-generation Web 2.0 intranet (implemented in 2005) and the planning and implementation of the second-generation Web 2.0 intranet in 2008. While the technologies are different, the case study shows many of the best practices established for traditional intranets still hold true for the Web 2.0 intranets in the case study. Following many of the traditional best practices helped the second Web 2.0 intranet to succeed, while not following established practices hurt the first Web 2.0 intranet's ability to meet departmental expectations of information access and information currency. Nonetheless, there are issues specific to Web 2.0 intranets that are not covered in existing best practices. More input is needed before establishing Web 2.0-specific intranet best practices. 0 0
Community-legitimated e-testing:A basis for a novel, self-organized and sustainable (e)learning culture? Nestle F.
Nestle N.
CSEDU 2009 - Proceedings of the 1st International Conference on Computer Supported Education English 2009 Based on the assumption that educational standards can be operationally defined by pools of specific testing items properties of such item pools are discussed. The main suggestion of the paper is that pools of testing items defining a standard should be free accessible in internet, that they provide immediate feedback in form of scores and that certified results should be equivalent to results of classroom work. For the development of the item pools, web-2.0-type methods can be much more effective than closed expert groups and item evaluation by statistic methods.. Finally the consequences of such transparent community-legitimated standards for the future role of teachers and future forms of learning environments are discussed. 0 0
Creating User Profiles Using Wikipedia Krishnan Ramanathan
Komal Kapoor
ER English 2009 0 0
Design Alternatives for a MediaWiki to Support Collaborative Writing in Higher Education Classes Sumonta Kasemvilas
Lorne Olfman
Issues in Informing Science and Information Technology English 2009 Constructivist learning mechanisms such as collaborative writing have emerged as a result of the development of Web 2.0 technologies. We define the term mandatory collaborative writing to describe a writing activity where the group has a firm deadline. Our study focuses on how a wiki can fully support mandatory group writing. The motivation of this design science research study emerges from a graduate Knowledge Management class assignment to write a wiki book. The project outcome shows that the wiki instance used for the project, MediaWiki, could better facilitate the process with a set of extensions that support discussion, evaluation, and project management. We outline designs for these mechanisms: 1) a discussion mechanism that changes the way users discuss content on a wiki page and increases group awareness; 2) an evaluation mechanism that provides a tool for the instructor to monitor and assess students’ performance; and 3) a project management tool that increases awareness of the status of each component of the writing project and provides an overall summary of the project. A demonstration of the principles to a focus group provided a basic proof of the validity of these mechanisms. 16 1
Overview of videoCLEF 2008: Automatic generation of topic-based feeds for dual language audio-visual content Larson M.
Newman E.
Jones G.J.F.
Lecture Notes in Computer Science English 2009 The VideoCLEF track, introduced in 2008, aims to develop and evaluate tasks related to analysis of and access to multilingual multimedia content. In its first year, VideoCLEF piloted the Vid2RSS task, whose main subtask was the classification of dual language video (Dutch-language television content featuring English-speaking experts and studio guests). The task offered two additional discretionary subtasks: feed translation and automatic keyframe extraction. Task participants were supplied with Dutch archival metadata, Dutch speech transcripts, English speech transcripts and ten thematic category labels, which they were required to assign to the test set videos. The videos were grouped by class label into topic-based RSS-feeds, displaying title, description and keyframe for each video. Five groups participated in the 2008 VideoCLEF track. Participants were required to collect their own training data; both Wikipedia and general web content were used. Groups deployed various classifiers (SVM, Naive Bayes and k-NN) or treated the problem as an information retrieval task. Both the Dutch speech transcripts and the archival metadata performed well as sources of indexing features, but no group succeeded in exploiting combinations of feature sources to significantly enhance performance. A small scale fluency/adequacy evaluation of the translation task output revealed the translation to be of sufficient quality to make it valuable to a non-Dutch speaking English speaker. For keyframe extraction, the strategy chosen was to select the keyframe from the shot with the most representative speech transcript content. The automatically selected shots were shown, with a small user study, to be competitive with manually selected shots. Future years of VideoCLEF will aim to expand the corpus and the class label list, as well as to extend the track to additional tasks. 0 0
The importance of manual assessment in link discovery Huang W.C.
Andrew Trotman
Shlomo Geva
Proceedings - 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2009 English 2009 Using a ground truth extracted from the Wikipedia, and a ground truth created through manual assessment, we show that the apparent performance advantage seen in machine learning approaches to link discovery are an artifact of trivial links that are actively rejected by manual assessors. 0 0
The methodology of manual assessment in the evaluation of link discovery Huang W.C.
Andrew Trotman
Shlomo Geva
ADCS 2009 - Proceedings of the Fourteenth Australasian Document Computing Symposium English 2009 The link graph extracted from the Wikipedia has often been used as the ground truth for measuring the performance of automated link discovery systems. Extensive manual assessments experiments at INEX 2008 recently showed that this is unsound and that manual assessment is essential. This paper describes the methodology for link discovery evaluation which was developed for use in the INEX 2009 Link-the-Wiki track. In this approach both manual and automatic assessment sets are generated and runs are evaluated using both. The approach offers a more reliable evaluation of link discovery methods than just automatic assessment. A new evaluation measure for focused link discovery is also introduced. 0 0
VideoCLEF 2008: ASR classification with wikipedia categories Kusrsten J.
Richter D.
Eibl M.
Lecture Notes in Computer Science English 2009 This article describes our participation at the VideoCLEF track. We designed and implemented a prototype for the classification of the Video ASR data. Our approach was to regard the task as text classification problem. We used terms from Wikipedia categories as training data for our text classifiers. For the text classification the Naive-Bayes and kNN classifier from the WEKA toolkit were used. We submitted experiments for classification task 1 and 2. For the translation of the feeds to English (translation task) Google's AJAX language API was used. Although our experiments achieved only low precision of 10 to 15 percent, we assume those results will be useful in a combined setting with the retrieval approach that was widely used. Interestingly, we could not improve the quality of the classification by using the provided metadata. 0 0
GikiP: Evaluating geographical answers from wikipedia Diana Santos
Nuno Cardoso
International Conference on Information and Knowledge Management, Proceedings English 2008 This paper describes GikiP, a pilot task that took place in 2008 in CLEF. We present the motivation behind GikiP and the use of Wikipedia as the evaluation collection, detail the task and we list new ideas for its continuation. 0 0
GikiP: evaluating geographical answers from wikipedia Diana Santos
Nuno Cardoso
GIR English 2008 0 0
Lesson-preparing innovation: A new effective approach on implementing collaborative lesson-preparing activities within eduwiki Lei Wang
Chaohua Gong
Proceedings - International Conference on Computer Science and Software Engineering, CSSE 2008 English 2008 Lesson-preparing is an important stage in the teaching activity processes. This paper argues that collaborative lesson-preparing within Eduwiki environment is a new effective approach for teachers' collaboration and teaching. Further, the architecture of Eduwiki is effective in monitoring and recording processes of lesson-preparing. The paper also provides a case to show the mechanism of collaborative lesson-preparing activity. Teachers' evaluations have proved that Eduwiki is effective in motivating peer-supported collaborative lesson-preparing activity, as well as for teachers' mutual development. 0 0
Evaluating the comprehensiveness of wikipedia: The case of biochemistry Brendan Luyt
Kwek W.T.
Sim J.W.
Peng York
Lecture Notes in Computer Science English 2007 [No abstract available] 0 0