The Usability of Collaborative Virtual Environments: Towards an Evaluation Framework

Doctoral thesis, 2004

Collaboration is often a necessity in order to reach one's goals in society, organizations, or among individuals. Collaborative Virtual Environments (CVEs) can be used as a medium supporting communication and interaction between people. By allowing real-time co-work with graphical representations, they offer new possibilities to facilitate collaboration. Yet, while the use of CVEs is continuously increasing, there is still little research on understanding the consequences of using them or about their usability. By reviewing previous research, we can see that existing studies have focused almost exclusively on improving technical aspects for these environments. Problems originating from multiple users and the collaborative characteristics associated with the technology have been neglected. This thesis addresses these problems. The empirical background consists of five experimental studies that comparatively evaluated different CVE technologies and applications. The first three studies used quantitative evaluations to show how new immersive technologies, where the projected environments surround the users, have both benefits and drawbacks for use in comparison to desktop systems or for working with full-scale models. The evaluation focused on users' experiences and on their performance. It was found that although immersive technologies provided greater presence, effectiveness varied more with the tasks than with the technology used. The symmetry or asymmetry in the distributed settings was also found to have a major impact on effectiveness. The methods used for the first three studies were found to have certain limitations in analyzing collaboration. In the other two studies additional methods were therefore developed for identifying successes and failures in interaction with the technology, and social interaction. By focusing on social interaction, interaction via technology and problem solving separately, small sequences of successes and failures during collaboration were analyzed. Many of these sequences can be considered as patterns since they recurred during several tasks. The results show that problems, particularly in social interaction, have an impact on the efficiency of interaction via technology, and vice versa. Adding problem solving as a third process provides a means of developing a systematic evaluation framework. This framework, moreover, includes both qualitative and quantitative analyses. The findings presented in this thesis contribute to developing better usability evaluation methods for CVEs, and hence to the design of new CVEs. The results from the comparative evaluations from the experiments can be used to examine the social implications of using CVEs and also to support the selection and practical use of different types of CVE.