Visual GUI Testing: Automating High-level Software Testing in Industrial Practice
Doctoral thesis, 2015

Software Engineering is at the verge of a new era where continuous releases are becoming more common than planned long-term projects. In this context test automation will become essential on all levels of system abstraction to meet the market's demands on time-to-market and quality. Hence, automated tests are required from low-level software components, tested with unit tests, up to the pictorial graphical user interface (GUI), tested with user emulated system and acceptance tests. Thus far, research has provided industry with a plethora of automation solutions for lower level testing but GUI level testing is still primarily a manual, and therefore costly and tedious, activity in practice. We have identfied three generations of automated GUI-based testing. The first (1st) generation relies on GUI coordinates but is not used in practice due to unfeasible maintenance costs caused by fragility to GUI change. Second (2nd) generation tools instead operate against the system's GUI architecture, libraries or application programming interfaces. Whilst this approach is successfully used in practice, it does not verify the GUI's appearance and it is restricted to specific GUI technologies, programming languages and platforms. The third (3rd) generation, referred to as Visual GUI Testing (VGT), is an emerging technique in industrial practice with properties that mitigate the challenges experienced with previous techniques. VGT is defined as a tool-driven test technique where image recognition is used to interact with, and assert, a system's behavior through its pictorial GUI as it is shown to the user in user-emulated, automated, system or acceptance tests. Automated tests that produce results of quality on par with a human tester and is therefore an effective complement to reduce the aforementioned challenges with manual testing. However, despite its benefits, the technique is only sparsely used in industry and the academic body of knowledge contains little empirical support for the technique's industrial viability. This thesis presents a broad evaluation of VGT's capabilities, obtained through a series of case studies and experiments performed in academia and Swedish industry. The research follows an incremental methodology that began with experimentation with VGT, followed by industrial studies that were concluded with a study of VGT's use at a company over several years. Results of the research show that VGT is viable for use in industrial practice with better defect-finding ability than manual tests, ability to test any GUI based system, high learnability, feasible maintenance costs and both short and longterm company benefits. However, there are still challenges associated with the successful adoption, use and long-term use of VGT in a company, the most crucial that suitable development and maintenance practices are used. This thesis thereby concludes that VGT can be used in industrial practice and aims to provides guidance to practitioners that seek to do so. Additionally, this work aims to be a stepping stone for academia to explore new test solutions that build on image recognition technology to improve the state-of-art.

Applicability and Feasibility

Software Engineering

Empirical Research

Visual GUI Testing

Automated Testing

Industrial Research

Opponent: Associate Professor Myra Cohen


Emil Alégroth

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

Denna avhandling presenterar och utvärderar användande av Visuell GUI Testning (VGT) i mjukvaruindustri med målet att undersöka teknikens effektivitet och kostnadseffektivitet. VGT är definierad som en automatiserad testteknik som utnyttjar verktyg med bildigenkänningsalgoritmer för emulation av mänskliga system- och acceptanstester.

This thesis presents and evaluates the use of Visual GUI Testing (VGT) in software industry with the goal of evaluating the technique's efficiency and cost-efficiency. VGT is defined as an automated test technique that uses tools with image recognition algorithms to emulate human system and acceptance tests.

Areas of Advance

Information and Communication Technology

Building Futures (2010-2018)


Driving Forces

Innovation and entrepreneurship

Subject Categories

Computer Systems



Doktorsavhandlingar vid Chalmers tekniska högskola. Ny serie


Opponent: Associate Professor Myra Cohen

More information