Guidelines for empirical studies in software engineering involving large language models
Artikel i vetenskaplig tidskrift, 2027
Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines.org).
Large language models
Meta-science
Guidelines
Software engineering
Empirical research
Författare
Sebastian Baltes
Universität Heidelberg
Florian Angermeir
Blekinge Tekniska Högskola, BTH
Fortiss GmbH
Chetan Arora
Monash University
Marvin Muñoz Barón
Technische Universität München
Chunyang Chen
Technische Universität München
Lukas Böhme
Hasso-Plattner-Institut fur Softwaresystemtechnik GmbH
Universität Potsdam
Fabio Calefato
Universita degli Studi di Bari Aldo Moro
Neil A. Ernst
University of Victoria
Davide Falessi
Universita degli Studi di Roma Tor Vergata
B. Fitzgerald
University of Limerick
Lero - The Irish Software Engineering Research Centre
Davide Fucci
Blekinge Tekniska Högskola, BTH
Junda He
Singapore Management University
Christoph Treude
Singapore Management University
Marcos Kalinowski
Pontificia Universidade Catolica do Rio de Janeiro
Stefano Lambiase
Aalborg Universitet
Daniel Russo
Aalborg Universitet
M. Lungu
IT-Universitetet i Kobenhavn
Cristina Martinez Montes
Chalmers, Data- och informationsteknik, Interaktionsdesign och Software Engineering
Göteborgs universitet
Lutz Prechelt
Freie Universität Berlin
Paul Ralph
Dalhousie University
Rijnard van Tonder
Independent researcher
Stefan Wagner
Technische Universität München
Empirical Software Engineering
1382-3256 (ISSN) 1573-7616 (eISSN)
Vol. 32 2 64Ämneskategorier (SSIF 2025)
Programvaruteknik
Systemvetenskap, informationssystem och informatik
Datorsystem
DOI
10.1007/s10664-026-10922-3
Relaterade dataset
Guidelines [dataset]
URI: https://llm-guidelines.org/