If you are interested in participating, send us an email to diannibereval [at] gmail.com and we will reply you with detailed instructions for participation. All participants will be automatically subscribed to our mailing list. Information about the submission of results and their format will be available soon.
We invite potential participants to contact us, in order to be kept up to date with the latest news related to the task. The organizers will assist you for any potential issues that could be raised.
|15 March 2018||Trial data release|
|15 April 2018||Test data realease|
|3 May 2018||End of the evaluation period|
|15 May 2018||Results posted|
|1 June 2018||End of the period for the reception of Working notes|
|15 June 2018||Release of the Working notes reviews|
|29 June 2018||Release of the Working and overview of the task for the corresponding proceedings|
The dataset has been collected between 2017 and 2018. DIANN's corpus consists of a collection of 500 abstracts from Elsevier journal papers related to the biomedical domain. We have selected those abstracts for which there are available both, the Spanish and the English versions. We have annotated the disabilities appearing in these abstracts using the XML tag <dis>. Disabilities are usually expressed either with a specific word, such as "blindness", or as the limitation or lack of a human function, such as "lack of vision".
The boundaries among diseases, disabilities and signs are often unclear. In order to clarify what have been considered a disability in the annotations for this task, we provide lists of disability terms and lists of functions whose absence or limitation has been considered a disability.
|Communication||Behaviour||Day-to-day living activities|
|Capacidad intelectual||Capacidad ocupacional||Capacidades afectivas|
|Capacidades psicológicas||Capacidades sociales||Cognición|
|Desarrollo||Emociones||Habilidad para realizar actividades de la vida cotidiana|
Notice that these lists only include one representative term for each concept, but different variants of them can appear in the corpus. For example "sightlessness" does not appear because its representative term is "blindness". Functions can also appear in the corpus in a wide variety of forms. For example:
- Sight: One month later, she presented with complaints of systemic oedema and <dis>loss of vision </dis>.
- Movilidad: Fampridina-LP a dosis de 10mg cada 12 h es actualmente el único fármaco autorizado para mejorar el <dis>trastorno de la marcha </dis> en adultos con EM.
The modifiers affecting the disability have been annoated as part of it.
- aphasia / isolated aphasia / ...
- dementia / advanced dementia / ...
- afasia / afasia aislada / ...
- demencia / demencia avanzada / ...
Acronyms referring to a disability have been also annotated with the <dis> tag, but only if the extended version has also appeared in the same abstract.
It is necessary to take into consideration that disabilities that appear isolated in texts without any context (e. g., search queries) or disabilities that appear in names of associations or entities (e. g., Care Unit for Dementia Patients) have not been annotated.
Negation is annotated when it affects one or more disabilities. We have adopted the criteria from the Bioscope corpus for annotating negation in English. For Spanish we have annotated the scope corresponding to the English annotation. Negation is annotated with the <scp> tag. For the sake of clarity, the negation triggers has also been annoted with the <neg> tag.
Two types of matching will be used for the evaluation: partial and exact. For instance, if "severe cognitive impairment" is annotated in the goal standard, exact matching will be "severe cognitive impairment" and partial matching could be "cognitive impairment". For each system, the results obtained will be provided according to the matching type. The metrics used to evaluate the systems will be precision, recall and their harmonic mean F-measure.
Lourdes Araujo Serna
Juan Martínez Romo
Hermenegildo Fabregat Marcos
Please, if you have any questions regarding the task, please do not hesitate to ask through the email: diannibereval [at] gmail.com