Gaze Based Human Intention Analysis
Doctoral thesis, 2023
The ability to determine an upcoming action or what decision a human is about to take, can be useful in multiple areas, for example during human-robot collaboration in manufacturing, where knowing the intent of the operator could provide the robot with important information to help it navigate more safely. Another field that could benefit from a system that provides information regarding human intentions is the field of psychological testing where such a system could be used as a platform for new research or be one way to provide information in the diagnostic process. The work presented in this thesis investigates the potential use of virtual reality as a safe, measurable, and customizable environment to collect gaze and movement data, eye tracking as the non-invasive system input that gives insight into the human mind, and deep machine learning as one tool to analyze the data. The thesis defines an experimental procedure that can be used to construct a virtual reality based testing system that gathers gaze and movement data, carry out a test study to gather data from human participants, and implement artificial neural networks in order to analyze human behaviour. This is followed by two studies that gives evidence to the decisions that were made in the experimental procedure and shows the potential uses of such a system.
eye tracking
psychological testing
Virtual reality (VR)
deep machine learning
human intention prediction
collaborative robots
uncertainty estimation
time series analysis
EB-salen, Hörsalsvägen 11.
Opponent: Professor Mikael Ekström, Mälardalens universitet, Västerås, Sverige
Author
Julius Pettersson
Chalmers, Electrical Engineering, Systems and control
Human Movement Direction Classification using Virtual Reality and Eye Tracking
Procedia Manufacturing,; Vol. 51(2020)p. 95-102
Paper in proceeding
Intended Human Arm Movement Direction Prediction using Eye Tracking
International Journal of Computer Integrated Manufacturing,; Vol. In Press(2023)
Journal article
Comparison of LSTM, Transformers, and MLP-mixer neural networks for gaze based human intention prediction
Frontiers in Neurorobotics,; Vol. 17(2023)
Journal article
Cognitive Ability Evaluation using Virtual Reality and Eye Tracking
2018 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND VIRTUAL ENVIRONMENTS FOR MEASUREMENT SYSTEMS AND APPLICATIONS (CIVEMSA),; (2018)
Paper in proceeding
Exploring the usability of Virtual Reality and Eye Tracking for Psychological Testing using Raven’s Progressive Matrices
Imagine an operator preparing some material while standing with her back against a robot, working on an assembly in a collaborative workspace. Suddenly the operator glances over her shoulder and carelessly turns around into collision course with the robot. However, the robot has already predicted that the operator is about to move and makes sure to stay out of her way until it is safe to continue working together. This scenario highlights the importance of research on prediction of human behaviour. Collaborative robots today are not interactive enough to act in such a way since they cannot yet interpret humans and adapt to their swift changes in behaviour as another human would. The main reason is that the robots today are limited in their senses and their awareness of the surrounding environment, which makes the human responsible for avoiding collision.
This thesis presents a system that uses artificial intelligence (AI) to predict in what direction a human is about to reach her/his arm, solely based on where they have been looking, their eye gaze. Measuring eye gaze is interesting from a behavioral perspective, because it is a noninvasive method that can give an insight into the individual’s problem solving, reasoning, and search strategies. The tests performed in this work, using virtual reality (VR), shows promising results using the combination of eye gaze and AI as a method to predict human movement intention. VR is shown as an important tool to be able to analyze eye gaze and at the same time ensure the safety of the human operator as these means of communicating with a robot are not yet possible in a real world setting.
Finally, the thesis focuses on how the AI can estimate its own uncertainty and how we can use that to determine whether to trust its predictions or not. However, as this research is in its early phase, there are several other points of uncertainty that still need to be addressed in the future before it is possible to use this work in the real world. There are also other areas that may benefit from analysis of human intention. One such area is also addressed in the current thesis, where the same set-up is utilized in the context of psychological testing. As AI technology continues to develop, it can be used to enhance the process of psychological testing by increasing the amount and variety of information that can be obtained during testing, increasing the accuracy of the test scores.
This thesis presents a system that uses artificial intelligence (AI) to predict in what direction a human is about to reach her/his arm, solely based on where they have been looking, their eye gaze. Measuring eye gaze is interesting from a behavioral perspective, because it is a noninvasive method that can give an insight into the individual’s problem solving, reasoning, and search strategies. The tests performed in this work, using virtual reality (VR), shows promising results using the combination of eye gaze and AI as a method to predict human movement intention. VR is shown as an important tool to be able to analyze eye gaze and at the same time ensure the safety of the human operator as these means of communicating with a robot are not yet possible in a real world setting.
Finally, the thesis focuses on how the AI can estimate its own uncertainty and how we can use that to determine whether to trust its predictions or not. However, as this research is in its early phase, there are several other points of uncertainty that still need to be addressed in the future before it is possible to use this work in the real world. There are also other areas that may benefit from analysis of human intention. One such area is also addressed in the current thesis, where the same set-up is utilized in the context of psychological testing. As AI technology continues to develop, it can be used to enhance the process of psychological testing by increasing the amount and variety of information that can be obtained during testing, increasing the accuracy of the test scores.
Subject Categories
Electrical Engineering, Electronic Engineering, Information Engineering
ISBN
978-91-7905-860-9
Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie: 5326
Publisher
Chalmers
EB-salen, Hörsalsvägen 11.
Opponent: Professor Mikael Ekström, Mälardalens universitet, Västerås, Sverige