Gaze Based Human Intention Analysis
Doktorsavhandling, 2023

The ability to determine an upcoming action or what decision a human is about to take, can be useful in multiple areas, for example during human-robot collaboration in manufacturing, where knowing the intent of the operator could provide the robot with important information to help it navigate more safely. Another field that could benefit from a system that provides information regarding human intentions is the field of psychological testing where such a system could be used as a platform for new research or be one way to provide information in the diagnostic process. The work presented in this thesis investigates the potential use of virtual reality as a safe, measurable, and customizable environment to collect gaze and movement data, eye tracking as the non-invasive system input that gives insight into the human mind, and deep machine learning as one tool to analyze the data. The thesis defines an experimental procedure that can be used to construct a virtual reality based testing system that gathers gaze and movement data, carry out a test study to gather data from human participants, and implement artificial neural networks in order to analyze human behaviour. This is followed by two studies that gives evidence to the decisions that were made in the experimental procedure and shows the potential uses of such a system.

time series analysis

uncertainty estimation

Virtual reality (VR)

human intention prediction

deep machine learning

eye tracking

collaborative robots

psychological testing

EB-salen, Hörsalsvägen 11.
Opponent: Professor Mikael Ekström, Mälardalens universitet, Västerås, Sverige


Julius Pettersson

Chalmers, Elektroteknik, System- och reglerteknik

Human Movement Direction Classification using Virtual Reality and Eye Tracking

Procedia Manufacturing,; Vol. 51(2020)p. 95-102

Paper i proceeding

Intended Human Arm Movement Direction Prediction using Eye Tracking

Comparison of LSTM, Transformers, and MLP-Mixer Neural Networks for Gaze Based Human Intention Prediction

Cognitive Ability Evaluation using Virtual Reality and Eye Tracking


Paper i proceeding

Exploring the usability of Virtual Reality and Eye Tracking for Psychological Testing using Raven’s Progressive Matrices

Imagine an operator preparing some material while standing with her back against a robot, working on an assembly in a collaborative workspace. Suddenly the operator glances over her shoulder and carelessly turns around into collision course with the robot. However, the robot has already predicted that the operator is about to move and makes sure to stay out of her way until it is safe to continue working together. This scenario highlights the importance of research on prediction of human behaviour. Collaborative robots today are not interactive enough to act in such a way since they cannot yet interpret humans and adapt to their swift changes in behaviour as another human would. The main reason is that the robots today are limited in their senses and their awareness of the surrounding environment, which makes the human responsible for avoiding collision.

This thesis presents a system that uses artificial intelligence (AI) to predict in what direction a human is about to reach her/his arm, solely based on where they have been looking, their eye gaze. Measuring eye gaze is interesting from a behavioral perspective, because it is a noninvasive method that can give an insight into the individual’s problem solving, reasoning, and search strategies. The tests performed in this work, using virtual reality (VR), shows promising results using the combination of eye gaze and AI as a method to predict human movement intention. VR is shown as an important tool to be able to analyze eye gaze and at the same time ensure the safety of the human operator as these means of communicating with a robot are not yet possible in a real world setting.

Finally, the thesis focuses on how the AI can estimate its own uncertainty and how we can use that to determine whether to trust its predictions or not. However, as this research is in its early phase, there are several other points of uncertainty that still need to be addressed in the future before it is possible to use this work in the real world. There are also other areas that may benefit from analysis of human intention. One such area is also addressed in the current thesis, where the same set-up is utilized in the context of psychological testing. As AI technology continues to develop, it can be used to enhance the process of psychological testing by increasing the amount and variety of information that can be obtained during testing, increasing the accuracy of the test scores.


Elektroteknik och elektronik



Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5326



EB-salen, Hörsalsvägen 11.

Opponent: Professor Mikael Ekström, Mälardalens universitet, Västerås, Sverige

Mer information

Senast uppdaterat