News

Language-centric Artificial Intelligence applications are popping up all around us in our everyday lives. Voice assistants, TV show recommenders, automated translation services, to name a few, are used more and more. Especially in a crisis, like the pandemia Covid-19, their use is important for easy access to reliable sources of information.

The development of such powerful applications is fueled by three key factors: algorithms, computing power, and data. Nowadays, well-established deep learning algorithms are available and can be implemented via open-source machine learning libraries. In parallel, hardware IT infrastructures are continuously extended in terms of computing power and big data storage. The third factor concerns the acquisition of sizable data appropriate for the problem at hand (e.g. millions of translated sentence pairs in various language combinations for training Machine Translation engines).

Focusing on the goal of EU's multilingualism policy to give citizens access to EU procedures and information in their own languages, the Connecting Europe Facility - Automated Translation (CEF AT) aims at providing multilingual support to pan-European digital services, public administrations and SMEs in all EU Member States and EEA countries.

As a result, a continuous extension of the supported languages and quality improvement of the automated translation services is required. The European Language Resource Coordination (ELRC) contributes to this objective by coordinating the collection and processing of language resources pertinent to Machine Translation and by maintaining the language data repository (ELRC-SHARE), through which the collected data are made available not only to CEF AT but also, depending on their terms of use, to the general public.

To this end, the Institute for Language and Speech Processing / Athena Research Center, one of the founding partners of the ELRC initiative, has set up a workflow and developed a pipeline for parallel language data acquisition from the web, focusing on three domains, namely “Health”, “Culture” and “Scientific Research”. In the current COVID-19 crisis, the ELRC language data collection activities could not have been inert with respect to the growing demand for improved technology-enhanced multilingual access to COVID-19 information. As part of the data collection activities in the “Health” domain, efforts focused on identifying reliable sources of language data and on compiling dedicated resources on the pandemic. To this end,  the relevant MEDISYS metadata collections have been parsed and harvested in order to extract pairs of parallel sentences from comparable corpora by applying the above-mentioned workflow. Parts of these datasets were offered to the Covid-19 MLIA-Eval initiative.

The total number of TUs for EN-X language pairs that have been collected as part of the ELRC activities during the last two years amount to more than 40 million translation units in total. Further to the above, a considerable number of TUs have been identified for X-Y language pairs, where X and Y are CEF languages other than EN, while millions of TUs have been extracted from websites with multi-domain content, with the aim to cluster them in domain-specific subsets. In total, the constructed Language Resources comprise more than 80 million Tus, numbers that change constantly.