The Ultimate Display

Licentiate thesis, 2014

In 1965 Ivan E. Sutherland envisioned the Ultimate Display, a room in which a computer can directly control the existence of matter. This type of display would merge the digital and the physical world, dramatically changing how people interact with computers. This thesis explores flat displays, deformable displays, flexible materials, static, and mobile projection displays in dynamic environments. %Dynamic environments are inherent to human behavior, but pose big problems to Human-Computer Interaction since computing devices rely on many assumptions of the interaction. Two aspects of the dynamic environment are considered. One is mobile human nature -- a person moving through or inside an environment. The other is the change or movement of the environment itself. The initial study consisted of a mixed reality application, based on recent motor learning research. It tested if a performer's attentional focus on markers external to the body improves the accuracy and duration of acquiring a motor skill, as compared with the performer focusing on their own body accompanied by verbal instructions. This experiment showed the need for displays that resemble physical reality. Deformable displays and Organic User Interfaces (OUIs) leverage shape, material, and the inherent properties of matter in order to create natural, intuitive forms of interaction. We suggested designing OUIs employing depth sensors as 3D input, deformable displays as 3D output, and identifying attributes that couple matter to human perception and motor skills. Flexible materials were explored by developing a soft gripper able to hold everyday objects of various shapes and sizes. It did not use complex hardware or control algorithms, but rather combined sheets of flexible plastic materials and a single servo motor. The gripper showed how a simple design with a minimal control mechanism can solve a complex problem in a dynamic environment. It serves as an example application for merging the digital and the physical through flexible materials, embodied computation, and actuation. The next two experiments merge digital information with the physical dynamic environment by using mobile and static projectors. The mobile projector experiment consisted of GPS navigation using a bike-mounted projector, displaying a map on the pavement in front of the bike. We found out that if compared with a bike-mounted smartphone, the mobile projector yields a lower cognitive load for the map navigation task. A dynamic space emerges from the navigation task requirements, and the projected display becomes a part of the physical environment. In the final experiment, a person interacts with a changing, growing environment, on which digital information is projected from above using a static projector. The interactive space consists of cardboard building blocks, the arrangement of which are limited by the area of projection. The user adds cardboard blocks to the cluster based upon feedback projected from above. Concepts from artificial intelligence and architecture were applied for understanding the interaction between the environment, the user, the morphology, and the material of the physical building system.