MC4WEPS (Multilingual Corpus for WEb People Search) corpus provides a real scenario to train and evaluate systems to disambiguate web people searches. The two main features of this corpus are: it includes multilingual results, and it keeps the social networking profiles.
The heterogeneity property of text evaluation measures states that the probability of a real (i.e. human assessed) similarity increase is directly related to the heterogeneity of the set of automatic similarity measures that corroborate such increase. This script implements a method for similarity measures that is based on the heterogeneity principle. The method is completely unsupervised (it does not use any kind of human assessments on the quality of the measures to be combined) and leads to top performing combined similarity measures in multiple tasks like Document Clustering, Textual Entailment, Semantic Text Similarity, and automatic MT and Summarization.
Many Artificial Intelligence tasks cannot be evaluated with a single quality criterion and some sort of weighted combination is needed to provide system rankings. A problem of weighted combination measures is that slight changes in the relative weights may produce substantial changes in the system rankings. This software implements the Unanimous Improvement Ratio (UIR), a measure that complements standard metric combination criteria (such as van Rijsbergen's F-measure) and indicates how robust the measured differences are to changes in the relative weights of the individual metrics. UIR is meant to elucidate whether a perceived difference between two systems is an artifact of how individual metrics are weighted.
Reliability and Sensitivity (extended BCubed) (New Version!)
Some key Information Access tasks -- Document Retrieval, Clustering, Filtering, etc. -- can be seen as instances of a generic "document organization" problem that establishes priority and relatedness relationships between documents. In this paper we propose two complementary evaluation measures -- Reliability and Sensitivity -- for the generic Document Organization task which are derived from a proposed set of formal constraints (properties that any suitable measure must satisfy).
For each of the tasks subsumed under the document organization problem, Reliability and Sensitivity satisfy more formal constraints than previously existing evaluation metrics. Their most characteristic feature, in addition, is their strictness: in order to reach high Reliability and Sensitivity values, a system must also achieve high values with all standard evaluation measures.
Two sets of annotations for evaluating the task of entity profiling in Microblog Posts. The first dataset is created using a pooling methodology, for which various methods for automatically extracting aspects from tweets that are relevant for an entity have been implemented. Human assessors have labeled each of the candidates as being relevant or not. The second dataset contains opinion targets for which annotators consider individual tweets related to an entity and manually identify whether the tweet is opinionated or not. If so, they annotate which part of the tweet is subjective and what the target of the sentiment is.
1,000 reviews extracted from booking.com.
5,496 words and 2,190 synsets from WordNet 2.1 labeled with an emotional category.
SentiSense Tagger and SentiSense Visualizer are included in the SentiSense Tools package.
Automatic Association Of Web Directories To Word Senses
The aim of this research is the development and application of algorithms to combine lexical information with web directories, in order to associate Wordnet word senses with ODP (Open Directory Project) directories.
Test-suite for Information Synthesis studies, made up of 72 manually-generated reports (topic-oriented summaries of large sets of relevant documents).
User logs capturing all the information relevant to user interaction with the search interface during the iCLEF 2008-2009 campaigns.
A corpus testbed for people searching algorithms.
A comparable corpus for multilingual news clustering evaluation.
Within the inter-universitary program in Cognitive Science (UNED/UAM/UCM).
Intelligent Information Access
Multilingual and Multimodal Information Access.
• Foreign-language search assistants for Document Retrieval, Question Answering and Information Synthesis tasks.
• Shallow and efficient NLP for Information Access.
• Intelligent organization, visualization and browsing of search results.
• Text Mining in Multilingual Collections and Social Media.
• On-line Reputation Management.
Evaluation of Information Access Systems.
• Co-organizers of the Cross-Language Evaluation Forum (CLEF).
• Co-organizers of the CLEF Cross-Language Question Answering track.
• Co-organizers of WePS, the Web People Search Evaluation Campaign (2007-2010).
• Co-organizers of iCLEF, the Interactive Cross-Language Evaluation Forum (2001-2009).
Acquisition and Representation of Lexical, Grammatical and Semantic Knowledge
Word senses, lexical databases, multilingual semantic networks, multilingual thesauri, and multimedia resources.
Word Sense Disambiguation.
Mining the Web for lexical information.
Extraction of Semantic Relations.
Knowledge Acquisition and Textual Inference.