Strategic decision points in experiments: A predictive Bayesian optional stopping method

Preprint, 2025

Sample size determination is a critical element in experimental design – not least in traffic and transport research. On one hand, without a sufficient sample size, an experiment cannot reliably answer its research question(s); on the other hand, too many samples are a waste of resources. Frequentist statistics require that the sample size be determined up front and the experiment be carried through to completion. That approach relies on a power analysis and its assumptions, which cannot be adjusted once the experiment has started. Bayesian sample size determination, with proper priors, can replace this approach. Bayesian optional stopping (BOS) iteratively analyzes collected data and enables stopping the experiments if the analyzed results show the statistical target (precision or significance) has been reached. We combined BOS with Bayesian rehearsal simulations in a method we call predictive Bayesian optional stopping (pBOS). Bayesian rehearsal simulations generate possible future data based on the Bayesian posterior distribution of some initially collected data and selected priors. Like BOS, pBOS iteratively analyzes the current experiment results and stops the experiment when a pre-defined statistical target is reached. Moreover, the rehearsal simulations also iteratively predict how the experiment might unfold, so that it can be stopped if the target is unlikely to be reached with some fixed maximum number of samples (typically constrained by available resources). While developing the method, we identified an inherent bias in the predictions. To correct for it, we employed a multiple linear regression model on the actual precision and predicted precision of the targets. We demonstrate the benefits and drawbacks of pBOS compared to BOS and frequentist power analysis, as well as providing guidelines for its use. The pBOS method shows a cost benefit up to 118% better than the cost benefit of the traditional BOS, depending on characteristics of the experiment, which in turn is better than the frequentist sample size determination. In summary, a researcher using pBOS can stop an experiment when the research question cannot be answered with the allocated resources, or when sufficient data have been collected—providing potential redirection of resources or cost savings compared with traditional frequentist sample size determination or BOS.