Aix-Marseille Université

Master Informatique

Parcours Science et Ingénierie des Données

Stage M2

"Using deep learning to study children's multimodal behavior in
face-to-face conversation"

Summary and skills required

The study of how children develop their conversational skills and how
these skills help them learn from others is an important scientific
frontier at the crossroad of social, cognitive, and linguistic
development with important applications in health (e.g., mitigating
communicative difficulties), education (e.g. improving teaching
practices), and child-oriented AI (e.g., virtual learning
companions). Recent advances in Natural Language Processing and
Computer Vision allow going beyond the limitations of traditional
research methods in the lab and advance formal theories of
conversational development in real-life contexts. In this internship,
we will leverage some of these recent techniques (e.g., multiscale
recurrent neural network, see [1]) to build a model that mimics how
children behave in face-to-face conversations with their caregivers
and how this behavior develops across middle childhood. The intern
will have access to the child-caregiver conversation data collected by
our team [2]. The data has already been hand-annotated for non-verbal
behavior (e.g., nods, smiles, and frowns) and is currently being
transcribed for verbal data and processed for extraction of
vocal/acoustic features.  The interns will contribute to the
development of a model (building on an existing pipeline in PyTorch)
that aims at studying how multimodal cues from the vocal, visual, and
verbal dimensions contribute to predicting the child's coordination
behavior in conversation (e.g., turn-taking management, negotiating
shared understanding with the interlocutor, and the ability for a
coherent/contingent exchange). The intern will collaborate closely
with several members of our team, involving computer scientists,
psychologists, and linguists (see our website www.cocodev.fr) as well
as members from the CoML team. Inquiries should be addressed to
Abdellah Fourtassi (abdellah.fourtassi@gmail.com).

[1] Roddy, M., Skantez, & Harte (2018). Multimodal Continuous
Turn-Taking Prediction Using Multiscale RNNs. In Proceedings of the
20th ACM International Conference on Multimodal Interaction

[2] Bodur, K., Nikolaus, M., Kassim, F., Prévot, L., & Fourtassi,
A. (2021). ChiCo: A Multimodal Corpus for the Study of Child
Conversation. In Proceedings of the International Workshop on Corpora
and Tools for Social Skills Annotation. 23rd ACM International
Conference on Multimodal Interaction

Supervisors:

-Abdellah Fourtassi (Aix-Marseille & INRIA Paris)
abdellah.fourtassi@univ-amu.fr
&
-Kevin El Haddad (University of Mons)
-Laurent Prévot (Aix-Marseille)
-Justine Cassell (INRIA Paris)

Other Détails
Stage au Laboratoire Informatique et Systèmes, équipes TALEP & CoCoDev
Durée du stage : 5 à 6 mois
Localisation : LIS
Gratification : 600 euros/mois

Email Abdellah Fourtassi (abdellah.fourtassi@univ-amu.fr) before
November 1st, 2021 if possible