Planning of a holistic summative eHealth evaluation: The interplay between standards and reality

Licentiate thesis, 2018

Summative evaluation assesses outcomes, produces evidence, and advances knowledge of eHealth implementations in health care provisions. Therefore, its quality is essential to reap benefits from the results generated by evaluation studies. Standardization is considered as among the solutions to improve quality of evaluations as it creates methodological uniformity and enhances research credibility. However, standards are used insufficiently in eHealth evaluation practice.

The purpose of this thesis is to study the use of standards in eHealth evaluation planning practice, in order to provide insight to support knowledge development. This thesis takes a stance in research concerning eHealth evaluation planning with regards to the standardization, translation, and collaboration. This thesis is built on a systematic literature review assessing the extent to which eHealth evaluation frameworks are used in summative eHealth evaluations and a single case study of eHealth evaluation of a project concerning eHealth implementation within several health care contexts. It focuses on the evaluation planning process, assesses adequacy of eHealth evaluation planning guidelines to practice, and identifies reasons that hinder the use of and adherence to standards.

The thesis confirms that eHealth evaluation frameworks are not used in the empirical eHealth evaluations. In contrast, the frameworks and the evaluation planning guidelines are found to be adequate and beneficial to practice. The reasons hindering the use of standards and affecting adherence to them are the insufficient evaluator’s experience and resources using a standard, evaluator’s unawareness of a standard, inadequacy of a standard to address a target population or a disease, non-existence of a validated version of a standard in a particular location, and a lack of fit between a standard and a scope of the evaluation. A model is developed, suggesting that standards can be viewed as objects translated in a specific context and influenced by collaborative activities. The thesis suggests that adherence to standards in eHealth evaluation practice could be seen as a range that is caused by trade-offs made when standards are translated based on reality.