Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?
Paper in proceeding, 2021

Large language models are known to suffer from the hallucination problem in that they are prone to output statements that are false or inconsistent, indicating a lack of knowledge. A proposed solution to this is to provide the model with additional data modalities that complements the knowledge obtained through text. We investigate the use of visual data to complement the knowledge of large language models by proposing a method for evaluating visual knowledge transfer to text for uni- or multimodal language models. The method is based on two steps, 1) a novel task querying for knowledge of memory colors, i.e. typical colors of well-known objects, and 2) filtering of model training data to clearly separate knowledge contributions. Additionally, we introduce a model architecture that involves a visual imagination step and evaluate it with our proposed method. We find that our method can successfully be used to measure visual knowledge transfer capabilities in models and that our novel model architecture shows promising results for leveraging multimodal knowledge in a unimodal setting.

Author

Tobias Norlund

Data Science and AI 1

Lovisa Hagström

Data Science and AI 1

Richard Johansson

University of Gothenburg

BlackboxNLP 2021 - Proceedings of the 4th BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP

149-162
9781955917063 (ISBN)

4th BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP 2021
Virtual, Punta Cana, Dominican Republic,

Subject Categories

Other Computer and Information Science

Information Science

Computer Science

More information

Latest update

10/23/2023