Title: Information retrieval evaluation framework: towards continuous evaluation for industrial search engines
Supervision: P. Mulhem and L. Goeuriot
Starting date: July 2021
Duration: 18 months
Keywords: information retrieval, continuous evaluation, explainability

Context -  project KODICARE: 

Evaluating search systems requires setting up an evaluation
environment: select a paradigm, metrics, a dataset, etc. The choice of
an environment is rarely motivated objectively, and the impact of its
variations (choosing a dataset against another, altering one, etc.) is
rarely measured. Such objectivity comes from a quantifiable
understanding of the differences between datasets, documents, or test
queries. In KoDicare, we generically call such differences "knowledge
delta". Evaluation of several environments, knowing their knowledge
deltas, leads to measuring and qualifying "results deltas". Online
systems require continuous evaluation with a stable and meaningful
environment that guarantees the reproducibility and explainability of
systems results. A controlled environment quantifying both "knowledge
deltas" and "result deltas" will support such continuous evaluation,
and enable the provision of explanations for system engineers through
the analysis of related changes in the two "deltas". The theoretical
results will be confronted to real cases defined by a French company
that deploys a web search engine (Qwant). Currently, no such framework
dedicated to real continuous evaluation of information retrieval
systems exists, due to the numerous parameters that must be handled.

Aims of the project: 


- Definition and formalization of knowledge deltas
- Creation of use cases
- Creation of a framework allowing the measure of the results delta
- Evaluation and analysis: correlations between knowledge delta and
  result delta
- Revision of the framework, towards continuous evaluation of search
  engines

The objectives of the hired PostDoc will be to apply the theoretical
framework to the use cases. This evaluation framework will allow a
comprehensive analysis of the steps of a single stage
experiment. Analysis and meta analysis of the experiments will lead to
an improvement of the framework towards continuous evaluation.

The work expected is the participation in the modeling of result
deltas and to manage large scale continuous evaluation experiments
(data acquisition from the industrial partner, bringing the knowledge
and result deltas to scale in strong interaction with the industrial
partner).  It is expected for the postdoc to actively contribute to
benchmarking activities.

Required skills for the PostDoc job:
- Knowledge in IR and its experimental evaluation
- Knowledge in ML for IR
- Strong development skills, preferably in Python, Java
- Strong interaction and teamwork skills 
- Reporting and publishing skills

Offer details


- Hosting institution: One of the major research-intensive French
  universities, Univ. Grenoble Alpes enjoys an international
  reputation in many scientific fields, as confirmed by international
  rankings. The dynamic ecosystem, grounded on a close interaction
  between research, education and companies, has earned Grenoble to be
  ranked as the 5th most innovative city in the world. Surrounded by
  mountains, the campus benefits from a natural environment and a high
  quality of life and work environment. With 7000 foreign students and
  the annual visit of more than 8000 researchers from all over the
  world, Univ. Grenoble Alpes is an internationally engaged
  university. A personalized Welcome Center for international
  students, PhDs and researchers facilitates your arrival and
  installation. Grenoble Informatics Laboratory (LIG) is one of the
  largest laboratories in Computer Science in France. It is structured
  as a Joint Research Center (French Unité Mixte de Recherche - UMR)
  founded by the following institutions: CNRS, Grenoble Institute of
  Technology (Grenoble INP), Inria Grenoble Rhône-Alpes, Grenoble
  Alpes University. The mission of LIG is to contribute to the
  development of fundamental aspects of Computer Science (models,
  languages, methodologies, algorithms) and address conceptual,
  technological, and societal challenges.

- Funding: The PostDoc is funded by the ANR-FWF KODICARE project
  (2020-2023), involving the University Grenoble Alpes (UGA), the
  Technological University of Vienna (TUW) and Qwant company.

- Gross salary: from 2300¤/month to 2600¤/month depending on the
  candidate experience.


Application: 
Candidates should send their application (CV, cover letter, transcripts...) by May, 26th to philippe.mulhem@imag.fr and lorraine.goeuriot@imag.fr