Towards Autonomous Testing Agents via Conversational Large Language Models
Paper in proceeding, 2023

Software testing is an important part of the development cycle, yet it requires specialized expertise and substantial developer effort to adequately test software. Recent discoveries of the capabilities of large language models (LLMs) suggest that they can be used as automated testing assistants, and thus provide helpful information and even drive the testing process. To highlight the potential of this technology, we present a taxonomy of LLM-based testing agents based on their level of autonomy, and describe how a greater level of autonomy can benefit developers in practice. An example use of LLMs as a testing assistant is provided to demonstrate how a conversational framework for testing can help developers. This also highlights how the often criticized 'hallucination' of LLMs can be beneficial for testing. We identify other tangible benefits that LLM-driven testing agents can bestow, and also discuss potential limitations.

large language model

software testing

machine learning

artificial intelligence, test automation

Author

Robert Feldt

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

Sungmin Kang

KAIST

Juyeon Yoon

KAIST

Shin Yoo

KAIST

Proceedings - 2023 38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023

1688-1693
9798350329964 (ISBN)

38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023
Echternach, Luxembourg,

Automated boundary testing for QUality of Ai/ml modelS (AQUAS)

Swedish Research Council (VR) (2020-05272), 2021-01-01 -- 2024-12-31.

Subject Categories

Language Technology (Computational Linguistics)

Software Engineering

Computer Science

DOI

10.1109/ASE56229.2023.00148

More information

Latest update

12/21/2023