Representations, Retrieval, and Evaluation in Knowledge-Intensive Natural Language Processing

Doctoral thesis, 2025

Several major advancements have recently been made within the field of Natural Language Processing (NLP). Nowadays, NLP systems based on language models (LMs) are readily available to the public in the form of chatbots, code assistants, writing assistants, etc. Any task that can be described in text can be, and is, addressed by NLP systems, covering the expected tasks as well as less expected tasks. While these advancements have highlighted many strengths of NLP systems, they have also highlighted weaknesses of NLP systems, hindering their use in certain scenarios. For example, modern NLP systems are neither reliable nor interpretable, limiting their usefulness for e.g. knowledge-intensive or high-risk tasks. In this thesis, we focus on the application of NLP systems to knowledge-intensive situations. We consider how methods leveraging different types of representations of information, such as the parametric memory of a model trained on multimodal information or retrieval-augmented generation (RAG), can be used to improve the systems. We find that RAG can be used to improve the stability of NLP systems for knowledge-intensive tasks, and bigger LMs generally are more efficient in leveraging the external information in RAG. We also develop datasets and methods to allow for more comprehensive and precise evaluations of NLP systems in knowledge-intensive situations. We find that insights gained from synthesised evaluation datasets are not guaranteed to transfer to real-world scenarios and that evaluation results are sensitive to how the knowledge under consideration interacts with the parametric memory of the LM. Taken together, the work included in this thesis improves our understanding of NLP systems for knowledge-intensive situations and highlights the important role of representations of information as well as realistic benchmarks for NLP.