Influence of LLM Prioritizations on Human Decisions in Requirements Engineering

Paper i proceeding, 2026

Prioritizing software requirements is a critical yet subjective task in early-stage development. As large language models (LLMs) such as ChatGPT become increasingly integrated into software engineering workflows, it remains unclear how their suggestions influence human decision-making in tasks that require judgment and trade-offs. In particular, little is known about how the reasoning style of LLM-generated justifications, whether intuitive or analytical—affects user trust, confidence, and behavioral responses. This study explores how LLMs shape human prioritization decisions through a controlled survey. Participants ranked requirements for two hypothetical projects, then reviewed LLM-generated prioritizations framed using either intuitive (System 1) or analytical (System 2) reasoning. After exposure, they could revise their rankings and changes in confidence, cognitive effort, and trust in the LLM. Findings show that participants treated the LLM as a cognitive aid rather than an authority: most retained their original decisions, but confidence increased and perceived effort declined. Evaluations of the LLM’s accuracy and trustworthiness were generally moderate, with reasoning style having a limited effect. These results suggest that LLMs can support human reasoning in requirements engineering, not by replacing human judgment but by reinforcing it through structured external input.