Guidelines for empirical studies in software engineering involving large language models
Journal article, 2027
Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommendations (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the system and prompt design beyond the model; (4) report session traces, i.e., interaction logs and runtime traces; (5) use suitable baselines, benchmarks, and metrics; (6) include an open LLM as a baseline; (7) validate LLM outputs against human judgment; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines.org).
Large language models
Meta-science
Guidelines
Software engineering
Empirical research
Author
Sebastian Baltes
Heidelberg University
Florian Angermeir
Blekinge Tekniska Högskola, BTH
Fortiss GmbH
Chetan Arora
Monash University
Marvin Muñoz Barón
Technical University of Munich
Chunyang Chen
Technical University of Munich
Lukas Böhme
Hasso-Plattner-Institut fur Softwaresystemtechnik GmbH
University of Potsdam
Fabio Calefato
University of Bari Aldo Moro
Neil A. Ernst
University of Victoria
Davide Falessi
University of Rome Tor Vergata
B. Fitzgerald
University of Limerick
Lero - The Irish Software Engineering Research Centre
Davide Fucci
Blekinge Tekniska Högskola, BTH
Junda He
Singapore Management University
Christoph Treude
Singapore Management University
Marcos Kalinowski
Pontifical Catholic University of Rio de Janeiro
Stefano Lambiase
Aalborg University
Daniel Russo
Aalborg University
M. Lungu
IT University of Copenhagen
Cristina Martinez Montes
Chalmers, Computer Science and Engineering (Chalmers), Interaction Design and Software Engineering
University of Gothenburg
Lutz Prechelt
Freie Universität Berlin
Paul Ralph
Dalhousie University
Rijnard van Tonder
Independent researcher
Stefan Wagner
Technical University of Munich
Empirical Software Engineering
1382-3256 (ISSN) 1573-7616 (eISSN)
Vol. 32 2 64Subject Categories (SSIF 2025)
Software Engineering
Information Systems
Computer Systems
DOI
10.1007/s10664-026-10922-3
Related datasets
Guidelines [dataset]
URI: https://llm-guidelines.org/