Embodied cognition and emotion in multisensory media

Licentiate thesis, 2007

Emotions underlie most events in our everyday life perception. They prepare ourselves for an optimum response to these events, either showing attraction or rejection to them. Likewise, media has the power to evoke in us similar affective processes as the ones created by real situations. Considering the growing number of new media applications and the amount of time we spend interacting with them, it is natural that an increasing body of research is now focusing on affective reactions and their impact on media perception. The research presented in this thesis concerns the role of emotions in media experiences. Special attention is paid to auditory-induced emotions even though contributions of other sensory modalities (visual, vibrotactile) to the affective media experience are also studied. In this work, emotions are approached from a perspective of embodiment. The concepts of embodied cognition and embodied emotion emphasize the idea that the body is strongly connected to the information processing. In line with this theory, the presented research used stimuli associated with a person's own body (self-representation sounds, e.g. heartbeat) or stimuli that occurred in a peripersonal space. It was hypothesized that stimuli resulting in the increase of body awareness might increase emotional experience and vice versa. Paper I represents a field-study in a cinema using different questionnaire-based measures of a subjective media experience. In addition, the influence of personality traits on the media experience was investigated. This paper explored how auditory information can compensate for the reduced visual information (frame rate) on a film. Paper II focuses on the specific effect of the perceived distance to stimuli on emotional experience. In particular, the influence of self-representation sounds on physiology, emotional responses to affective visual stimuli and memory for events was studied. Distant versus close sound reproduction conditions were used to identify whether an "embodied" experience (i.e. participants associating the self-sound with their own) can occur, which allows the emotional responses to be modulated. The multimodal contribution of vibrotactile information was also taken into account. Paper III concentrates on stimuli approaching one's body and multisensory integration of dynamic information. Specifically, this paper questions whether predictions of motion can be extrapolated from one sensory modality to other. In addition, it was examined whether providing ecological validity to the moving stimuli affects the nature of these dynamic links.