Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines
Journal article, 2022

A meaningful and deep understanding of the human aspects of software engineering (SE) requires psychological constructs to be considered. Psychology theory can facilitate the systematic and sound development as well as the adoption of instruments (e.g., psychological tests, questionnaires) to assess these constructs. In particular, to ensure high quality, the psychometric properties of instruments need evaluation. In this article, we provide an introduction to psychometric theory for the evaluation of measurement instruments for SE researchers. We present guidelines that enable using existing instruments and developing new ones adequately. We conducted a comprehensive review of the psychology literature framed by the Standards for Educational and Psychological Testing. We detail activities used when operationalizing new psychological constructs, such as item pooling, item review, pilot testing, item analysis, factor analysis, statistical property of items, reliability, validity, and fairness in testing and test bias. We provide an openly available example of a psychometric evaluation based on our guideline. We hope to encourage a culture change in SE research towards the adoption of establishedmethods from psychology. To improve the quality of behavioral research in SE, studies focusing on introducing, validating, and then using psychometric instruments need to be more common.

Empirical software engineering

behavioral software engineering

questionnaire design




Daniel Graziotin

University of Stuttgart

Per Lenberg

University of Gothenburg

Robert Feldt

Chalmers, Computer Science and Engineering (Chalmers), Software Engineering (Chalmers)

University of Gothenburg

Stefan Wagner

University of Stuttgart

ACM Transactions on Software Engineering and Methodology

1049-331X (ISSN)

Vol. 31 1 7

Teambaserad utveckling för säkerhet och kvalitet av programvara till flygsystem

VINNOVA (2017-04874), 2017-11-10 -- 2020-12-31.


Marianne och Marcus Wallenberg Foundation (2017.0071), 2018-07-01 -- 2021-06-30.

Subject Categories

Psychology (excluding Applied Psychology)

Construction Management

Software Engineering



More information

Latest update