The robot as scientist: Using mental simulation to test causal hypotheses extracted from human activities in virtual reality
Paper in proceeding, 2020

To act effectively in its environment, a cognitive robot needs to understand the causal dependencies of all intermediate actions leading up to its goal. For example, the system has to infer that it is instrumental to open a cupboard door before trying to grasp an object inside the cupboard. In this paper, we introduce a novel learning method for extracting instrumental dependencies by following the scientific approach of observations, generation of causal hypotheses, and testing through experiments. Our method uses a virtual reality dataset containing observations from human activities to generate hypotheses about causal dependencies between actions. It detects pairs of actions with a high temporal co-occurrence and verifies if one action is instrumental in executing the other action through mental simulation in a virtual reality environment which represents the system's mental model. Our system is able to extract all present instrumental action dependencies while significantly reducing the search space for mental simulation, resulting in a 6-fold reduction in computational time.


Constantin Uhde

Technical University of Munich

Nicolas Berberich

Technical University of Munich

Karinne Ramirez-Amaro

Chalmers, Electrical Engineering, Systems and control

Gordon Cheng

Technical University of Munich

IEEE International Conference on Intelligent Robots and Systems

21530858 (ISSN) 21530866 (eISSN)

8081-8086 9341505
9781728162126 (ISBN)

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Las Vegas, USA,

Subject Categories

Human Computer Interaction


Computer Science



More information

Latest update