The LIG (Laboratoire d'Informatique de Grenoble) proposes the following
Master 2 level internship:

*Title*: Context-Aware Neural Machine Translation Evaluation II

*Description*:

Context-Aware Neural Machine Translation  (CA-NMT) [Tiedemann and
Scherrer, 2017; Laubli et al., 2018; Miculicich et al., 2018; Maruf et
al., 2019; Zheng et al., 2020; Ma et al. 2021; Lupo et al., 2022] is
currently one of the main research axes in NLP, with strong impact on
both academic and company research.

CA-NMT systems are evaluated with both "average-quality-measuring"
metrics such as BLEU [Papineni et al., 2002], and dedicated contrastive
test suites [Voita et al., 2019; Muller&Rios 2018; Lopes et al., 2020].

The latter have been designed to measure specifically to which degree
CA-NMT systems are able to exploit context while scoring sentences to
be translated in context. Indeed the average translation quality
measured by BLEU has been shown inadequate in this respect [Lupo et
al., 2022].

When evaluating models with contrastive test suites however, models are
only asked to score sentences and not to translate them. The ability of
models to use context is thus only implicitly evaluated.

With the work planned in this internship, and exploiting the work
already done during a previous internship on the same subject, we would
like to make a step ahead in the evaluation of CA-NMT systems.

The idea is to exploit annotated data like those already used for
[Muller&Rios 2018; Lopes et al., 2020] or for [Ekaterina et al., 2022]
to explicitly involve discourse phenomena, such like coreferences and
anaphora, in the evaluation procedure of CA-NMT models.

Such evaluation procedure will allow possibly to design more accurate
and adequate evaluation measures for "discourse-phenomena-aware" CA-NMT
systems.


*Practical Aspects*:

In this internship the student will use Machine Learning and Deep
Learning tools to automatically annotate parallel data (at least
English-French, but possibly also English-German and other language
pairs) used for NMT with discourse phenomena, as well as Neural Machine
Translation tools for automatically generating translations that will
be used for CA-NMT evaluation.

Based on the annotation of discourse phenomena, we will design an
adequate evaluation metric for CA-NMT systems, taking into account the
capability of the system to exploit discourse phenomena. Finally, the
evaluation metric will be tested by evaluating CA-NMT systems already
available [Lupo et al., 2022] or trained from scratch at LIG.


*Profile*:

    -   Master 2 student level in computer science or NLP
    -   Interested in Natural Language Processing and Deep Learning
        approaches
    -   Skills in machine learning for neural models
    -   Computer science skills:


    1.  Python programming. Some knowledge of deep learning libraries
        such like Pytorch and possibly Fairseq.
    2.  Data manipulation and annotation


The internship may last from 5 up to 6 months, it will take place at
LIG laboratory, GETALP team (http://lig-getalp.imag.fr/), starting from
January/February 2024.

The student will be tutored by Marco Dinarelli
(http://www.marcodinarelli.it), and Emmanuelle Esperança-Rodier
(https://lig-membres.imag.fr/esperane/)

Interested candidates must send a CV and a motivation letter to (both
adresses) marco.dinarelli@univ-grenoble-alpes.fr,
Emmanuelle.Esperanca-Rodier@univ-grenoble-alpes.fr.


*Bibliography*

[Tiedemann and Scherrer, 2017] Neural ma- chine translation with
extended context. Workshop on Discourse in Machine Translation 2017.

[Laubli et al., 2018] Has machine translation achieved human parity? a
case for document-level evaluation. EMNLP 2018.

[Miculicich et al. 2018] Document-level neural machine translation with
hierarchical attention networks. EMNLP 2018.

[Maruf et al., 2019] Selective attention for context-aware neural
machine translation. NAACL 2019.

[Zheng et al., 2020] Towards Making the Most of Context in Neural
Machine Translation. IJCAI 2020.

[Ma et al., 2021]  A Comparison of Approaches to Document-level Machine
Translation. arXiv pre-print 2021.

[Lupo et al., 2022] Divide and Rule: Effective Pre-Training for
Context-Aware Multi-Encoder Translation Models. ACL 2022.

[Papineni et al., 2022] Bleu: a method for automatic eval- uation of
machine translation. ACL 2002.

[Voita et al., 2019] "When a good translation is wrong in context:
Context-aware machine translation improves on deixis, ellipsis, and
lexical cohesion". ACL 2019.

[Muller&Rios 2018] "A large-scale test set for the evaluation of
context-aware pronoun translation in neural machine translation." CMT
2018

[Lopes et al., 2020] "Document-level neural MT: A systematic
comparison". EAMT 2020

[Ekaterina et al., 2022] "ParCorFull2.0: a Parallel Corpus Annotated
with Full Coreference". LREC 2022.