[+] | [-]
CLEF QA Track logo

CLEF Question Answering Track

CLEF Association CLEF 2015
Question Answering, Natural Language, Linked Data, Semantic Indexing, Logical Inference, Textual Inference

General Scenario - QALD - BioASQ - Entrance Exams - Important Dates - Contact

General Scenario

In the current general scenario for the CLEF QA Track, the starting point is always a Natural Language question. However, answering some questions may need to query Linked Data (especially if aggregations or logical inferences are required), whereas some questions may need textual inferences and querying free-text, and finally, answering some queries may need both.

CLEF QA Track general scenario

As a matter of example related to CLEF eHealth, consider the use case where patients receive medical reports they don't understand. Given this report, patients have lots of questions. Some of them will need general definitions as one can find in wikipedia. Some might need more complex answers about the relations between symptoms, treatments, etc. We want to help them understand.

So, given this general scenario, CLEF QA Track will work on two instances: one targeted to (bio)medical experts (BioASQ Tasks) and a second targeted to Open Domains (QALD and Entrance Exams Tasks). In the first one, medical KBs, ontologies and articles must be taken into account. In the second one, general resources such as Wikipedia articles and DBpedia will be considered.


QALD: Question Answering over Linked Data

QALD-4 is the fourth in a series of evaluation campaigns on multilingual question answering over linked data, this time with a strong emphasis on interlinked datasets and hybrid approaches using information from both structured and unstructured data. The exercises are: Multilingual Question Answering (answers from an RDF data repository given a natural language question), Question Answering on Interlinked Data (questions require information from at least two interlinked biomedical datasets), and Hybrid Question Answering (questions over DBpedia that require a combination of structured information and raw text available in the DBpedia abstracts).

More info at:

BioASQ: Biomedical semantic indexing and question answering

The challenge (aka competition or shared task) will assess:

  • large-scale classification of biomedical documents onto ontology concepts (semantic indexing),
  • classification of biomedical questions onto relevant concepts,
  • retrieval of relevant document snippets, concepts and knowledge base triples,
  • delivery of the retrieved information in a concise and user-understandable form.

More info at:

Entrance Exams

Japanese University Entrance Exams include questions formulated at various levels of complexity and test a wide range of capabilities. The challenge of "Entrance Exams" aims at evaluating systems under the same conditions humans are evaluated to enter the University. In this first campaign we will reduce the challenge to Reading Comprehension exercises contained in the English exams. More types of exercises will be included in subsequent campaigns (2014-2016) in coordination with the "Entrance Exams" task at NTCIR. Exams are created by the Japanese National Center for University Admissions Tests. The "Entrance Exams" corpus is provided by NII's Todai Robot Project and NTCIR RITE.

More info at:

Important Dates

Registration at CLEFSoon
Release of BioASQ training dataDecember 09, 2014
Start of BioASQ Task on Semantic IndexingFebruary 03, 2015
Start of BioASQ Task on Question AnsweringMarch 05, 2015
Send the Entrance Exams agreement formby April 30, 2015
Release of Entrance Exams test dataMay 6, 2015
Run submissionsMay 15, 2015
Individual results to participantsMay 20, 2015
Submission of Working Notes PapersJune 7, 2015
CLEF Workshop in Toulouse, FRSeptember, 2015

Organizing Committee

Elena Cabrio (INRIA Sophia-Antipolis Mediterranee, Cedex, France)
Anastasia Krithara (NCSR "Demokritos", Greece)
Eduard Hovy (Carnegie Mellon University, USA)
Axel Ngonga (University Leipzig, Germany)
George Paliouras (NCSR "Demokritos", Greece)
Anselmo Peñas (UNED, Spain)
Alvaro Rodrigo (UNED, Spain)
Richard Sutcliffe (University of Essex, UK)
Christina Unger (CITEC, Universitat Bielefeld, Germany)


Anselmo Peñas (

Suported by: BioASQ logo BioASQ logo CHIST-ERA logo CHIST-ERA logo