The LIG (Laboratoire d'Informatique de Grenoble) proposes the following
Master 2 level internship:


*Title*: Context-Aware Neural Machine Translation Evaluation


*Description*:

Context-Aware Neural Machine Translation  (CA-NMT)
[Tiedemann and Scherrer, 2017; Laubli et al., 2018; Miculicich et al.,
2018; Maruf et al., 2019; Zheng et al., 2020; Ma et al. 2021; Lupo et
al., 2022] is currently one of the main research axes in NLP, with
strong impact on both academic and company research.

CA-NMT systems are evaluated with both "average-quality-measuring"
metrics such as BLEU [Papineni et al., 2002], and dedicated contrastive
test suites [Voita et al., 2019; Muller&Rios 2018; Lopes et al., 2020].

The latter have been designed to measure specifically to which degree
CA-NMT systems are able to exploit context while scoring sentences to
be translated in context. Indeed the average translation quality
measured by BLEU has been shown inadequate in this respect
[Lupo et al., 2022].

When evaluating models with contrastive test suites however, models are
only asked to score sentences and not to translate them. The ability of
models to use context is thus only implicitly evaluated.

With the work planned in this internship we would like to make a step
ahead in the evaluation of CA-NMT systems.

The idea is to exploit annotated data like those already used for
[Muller&Rios 2018; Lopes et al., 2020] to explicitly involve discourse
phenomena, such like coreferences and anaphora, in the evaluation
procedure of CA-NMT models.

Such evaluation procedure will allow possibly to design more accurate
and adequate evaluation measures for "discourse-phenomena-aware" CA-NMT
systems.


*Practical Aspects*:

In this internship the student will use Machine Learning and Deep
Learning tools to automatically annotate parallel data (at least
English-French, but possibly also English-German and other language
pairs) used for NMT with discourse phenomena, as well as Neural Machine
Translation tools for automatically generating translations that will
be used for CA-NMT evaluation.

Based on the annotation of discourse phenomena, we will design an
adequate evaluation metric for CA-NMT systems, taking into account the
capability of the system to exploit discourse phenomena. Finally, the
evaluation metric will be tested by evaluating CA-NMT systems already
available or trained from scratch at LIG by the student.


*Profile*:

-   Master 2 student level in computer science or NLP
-   Interested in Natural Language Processing and Deep Learning
    approaches
-   Skills in machine learning for probabilistic models
-   Computer science skills:

    1.  Python programming. Some knowledge of deep learning libraries
        such like Pytorch (Fairseq would be a plus).
    2.  Data manipulation and annotation

The internship may last from 5 up to 6 months, it will take place at
LIG laboratory, GETALP team (http://lig-getalp.imag.fr/), starting from
January/February 2022.

The student will be tutored by Marco Dinarelli
(http://www.marcodinarelli.it), and Emmanuelle Esperança-Rodier
(https://lig-membres.imag.fr/esperane/)

Interested candidates must send a CV and a motivation letter to (both
adresses) marco.dinarelli@univ-grenoble-alpes.fr,
Emmanuelle.Esperanca-Rodier@univ-grenoble-alpes.fr.


*Bibliography*

[Tiedemann and Scherrer, 2017] Neural machine translation with
    extended context. Workshop on Discourse in Machine Translation
    2017.

[Laubli et al., 2018] Has machine translation achieved human parity? a
    case for document-level evaluation. EMNLP 2018.

[Miculicich et al. 2018] Document-level neural machine translation with
    hierarchical attention networks. EMNLP 2018.

[Maruf et al., 2019] Selective attention for context-aware neural
    machine translation. NAACL 2019.

[Zheng et al., 2020] Towards Making the Most of Context in Neural
    Machine Translation. IJCAI 2020.

[Ma et al., 2021]  A Comparison of Approaches to Document-level Machine
    Translation. arXiv pre-print 2021.

[Lupo et al., 2022] Divide and Rule: Effective Pre-Training for
    Context-Aware Multi-Encoder Translation Models. ACL 2022.

[Papineni et al., 2022] Bleu: a method for automatic evaluation of
    machine translation. ACL 2002.

[Voita et al., 2019] "When a good translation is wrong in context:
    Context-aware machine translation improves on deixis, ellipsis,
    and lexical cohesion". ACL 2019.

[Muller&Rios 2018] "A large-scale test set for the evaluation of
    context-aware pronoun translation in neural machine translation."
    CMT 2018

[Lopes et al., 2020] "Document-level neural MT: A systematic
    comparison". EAMT 2020