|Year : 2021 | Volume
| Issue : 3 | Page : 374-379
Approaches to assess e-health programs: A scoping review
Nitin Kumar Joshi1, Pankaj Bhardwaj1, Deepak Saxena2, Praveen Suthar3, Vibha Joshi4
1 Community Medicine and Family Medicine; School of Public Health, Health Technology Assessment, All India Institute of Medical Sciences, Jodhpur, Rajasthan, India
2 Department of Epidemiology, Indian Institute of Public Health, Gandhinagar, Gujarat, India
3 School of Public Health, Health Technology Assessment, All India Institute of Medical Sciences, Jodhpur, Rajasthan, India
4 Resource Center, Health Technology Assessment, All India Institute of Medical Sciences, Jodhpur, Rajasthan, India
|Date of Submission||10-May-2020|
|Date of Acceptance||21-Apr-2022|
|Date of Web Publication||13-Oct-2021|
Dr. Pankaj Bhardwaj
School of Public Health, All India Institute of Medical Sciences, Jodhpur, Rajasthan
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Context: With the increasing trend of adopting e-Health technologies, the need for evidence for assessing e-Health technologies has become crucial. The appraisal of the e-Health program is important as this could provide guidance on further e-Health investment and adoption. Aims: The aim of the study was to provide an articulated body of literature on the current state of knowledge about the assessment of e-Health interventions. Settings and Design: Scoping review was conducted based on the framework provided by Arksey and O'Malley (2005) and considering enhancements proposed by Peters et al. (2015). Subjects and Methods: We searched the electronic databases and available gray literature from inception until the last week of October 2019. PRISMA flowchart for the study selection process was used to guide reporting. Data extraction included information on study design, authors, year of study, country, and key findings in terms of approaches used for assessment of e-Health programs. Data were compiled and summarized narratively. Results: Searches were performed between October and December 2019. Seventy-one relevant papers published between the years until the last week of October 2019 were reviewed and analyzed. After considering all the eligibility criteria, 15 papers were included. We identified 15 approaches for assessing e-Health programs, which were summarized and tabulated. Conclusions: This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. e-Health interventions are highly contextual; therefore; the phase of maturity and objective of the assessment should be considered while carrying out the assessment of e-Health programs.
Keywords: Assessment, digital health, e-Health, evaluation, m-Health
|How to cite this article:|
Joshi NK, Bhardwaj P, Saxena D, Suthar P, Joshi V. Approaches to assess e-health programs: A scoping review. Indian J Community Med 2021;46:374-9
|How to cite this URL:|
Joshi NK, Bhardwaj P, Saxena D, Suthar P, Joshi V. Approaches to assess e-health programs: A scoping review. Indian J Community Med [serial online] 2021 [cited 2021 Dec 8];46:374-9. Available from: https://www.ijcm.org.in/text.asp?2021/46/3/374/328169
| Introduction|| |
The World Health Organization (WHO) defines e-Health as the use of information and communication technologies for health. In a broader sense, e-Health is about improving the flow of information through electronic means to support the delivery of health-care services and the management of the health systems. It encompasses a diverse set of informatics tools and processes that are designed with the view of improving public health and health care.
Many countries including India are making significant investments in e-Health systems as e-Health has the potential to revolutionize the health-care delivery. The digitalization of health system could play a catalytic role in improving the continuum of care. With the increasing trend of adopting e-Health technologies, the need for evidence for assessing e-Health technologies has become crucial. Currently, e-Health programs are implemented based on assumptions of the benefits of e-Health only and there is a paucity of studies and documents which provide comprehensive guidance on assessing the e-Health programs. The appraisal of e-Health program is important as this could provide guidance on further e-Health investment and adoption.
| Subjects and Methods|| |
This scoping review was conducted based on the framework provided by Arksey and O'Malley (2005) and considering enhancements proposed by Peters et al. The protocol of the study was exclusively developed and prepared by using inputs from public health research experts. We used a PRISMA flowchart for the study selection process to guide reporting. This review was conducted using the following steps: Identifying the research question, identifying relevant studies, study selection, charting the data, and collating, summarizing, and reporting the results.
Data sources and search
We searched the electronic databases and available gray literature from inception until the last week of October 2019. The search query consists of terms considered by the authors to describe the assessment of e-Health programs: e-Health, digital health, m-Health, assessment, and evaluation. The search strategy was not limited by the study design. An authentic database included English language literature indexed in Google Scholar, PubMed, and Scopus. We also searched for the articles included in the referencing list of the selected studies to avoid missing any literature. Other available literatures including academic papers dissertations, conference papers, and reports were also considered. This was done in a diverse format and audiences from other sources can provide invaluable information for our scoping review. Gray literature was also considered in this review as this type of literature could help significantly in reducing publication bias, increase reviews' comprehensiveness, and timeliness and foster a balanced picture of available evidence.
Study selection: Inclusion criteria
We developed explicit inclusion criteria for this review. The focus of this scoping review was to gather evidence on the practices of e-Health assessment. The Merriam Webster dictionary defines assessment as “the action or an instance of making a judgment about something: The act of assessing something.” Hence, e-Health assessment is defined as an act of making a judgment about whether the e-Health program is functioning and producing the effects as expected. In this context, e-Health program can be any Information, Communication, and Technology (ICT)-based application, service, or resource used by an organization, providers, and consumers in managing health. All studies conducted in any settings or country published in the English language were eligible for inclusion.
Study selection: Screening
Two reviewers independently reviewed titles and abstracts. Full texts of potentially relevant literature were sort and screened for inclusion criteria. Agreement between reviewers was achieved by common consensus. Conflicts were resolved by joint discussions with the third arbitrator reviewer. As this study was scoping review, no critical appraisal of included literature was performed.
A review of full-text article and extraction of relevant data was independently done by two reviewers. Data extraction included information on study design, authors, year of study, country, and key findings in terms of approaches used for assessment of e-Health programs.
Data summary and synthesis
Data were compiled in a single spreadsheet. Considering the heterogeneity of the literature, we summarized the data narratively and did not pool the results of the included studies.
| Results|| |
Searches were performed between October and December 2019. Seventy-one relevant papers published between the years until the last week of October 2019 were reviewed and analyzed. Nineteen duplicate papers were removed, 18 papers were excluded as they were not related to evidence on the practices of e-Health assessment, and 19 papers were removed as they were not related to ICT-based application, services used by providers, and consumers in managing health. After considering all the eligibility criteria, 15 papers were included. [Figure 1] and [Figure 2] illustrate framework used for conducting scoping review and literature search process, respectively.
|Figure 1: Flowchart (framework) showing strategies for conducting scoping review. Adapted from-Scoping studies: towards a methodological framework (Hilary Arksey & Lisa O'Malley-2005)|
Click here to view
The scoping review revealed different viewpoints on the assessment of e-Health interventions. Different authors tried to provide a novel framework for the evaluation of e-Health programs. Tobias Kowatsch et al., proposed a framework for the design and evaluation of digital health interventions (DEDHIs). This framework described life cycle of a DHI and recommended relevant evaluation criteria and implementation barriers to be considered for each phase of this life cycle. For each of the four phases of the DEDHI framework, corresponding evaluation tasks were mentioned. For the preparation phase, evaluation criteria were ease of use, adherence, personalization, safety, privacy, and security; for the optimization phase, evaluation criteria were effectiveness (individual components of the DHI), perceived benefit, content quality, personalization, perceived enjoyment, aesthetics, adherence, service quality, safety, privacy, and security; for the evaluation phase, evaluation criteria were effectiveness, perceived benefit, adherence, personalization, service quality, safety, privacy and security, and accountability; for the implementation phase, evaluation criteria were adherence, personalization, perceived benefit, content quality, ethics, service quality, safety, privacy and security, and accountability.
Report of the Expert Panel on Effective Ways of Investing in Health, 2019, mentioned that most of the developed evaluation frameworks for evaluating digital health services were guided by health technology assessment (HTA) methodology. Since HTA was developed in the context of curative health technologies, improvement in HTA methodology is needed in the context of evaluating digital health technologies. Since HTA does not address study aspects such as access, equity, patient empowerment, or goal orientation which are likely to be relevant in the context of digital health services, methods to systematically consider these aspects are needed to be developed for inclusion in HTA frameworks. For evaluating the digital health services, border perspective is needed and tailor-made evaluation should be designed to capture all relevant changes in the appropriate manner. This report also recommended that to select an assessment strategy for digital health services, full description of the technology, its use and aim, intent to use, cost, consequences, and relevant comparators should be addressed. The development phase and implementation are crucial elements in evaluation. The evaluation should focus on relevant outcomes in the context of the objective of the health system. All relevant cost benefits and broad impacts should be considered.
Høstgaard et al. developed a constructive e-Health evaluation method which offers a full evaluation framework to support and facilitate end-user involvement during the e-Health life cycle. This method offers guidance through all life-cycle phases and a modified summative evaluation. It provides corresponding objectives, activities, and methods for phases, namely preanalysis, research and planning, design, development, implementation and diffusion, and modified summative evaluation. Data for this evaluation were collected using questionnaires, observations, interviews, and insight gathered from relevant documents.
Biggs et al. identified priority areas for the development of the digital health benefit evaluation (BE) framework. They provided an overview approach for BE of the digital health system in the context of My Health Record System. Five benefit workstreams such as customer and market insights; behavioral economics; data analytics; impact evaluations; and health economics evaluations were introduced to evaluate digital health initiatives using qualitative, quantitative, and mixed-method design.
Murray et al. examined the challenges related to the assessment of the DHIs and outlined an evaluation strategy in terms of the research questions needed to appraise DHIs. These research questions included defining the problem and the likely benefit of the DHI, the causal model describing how the intervention will achieve its intended benefit, key components and how they interact with one another, estimating overall benefit in terms of effectiveness, cost-effectiveness, and harms.
Catwell and Sheikh proposed that e-Health interventions should be assessed while they are being designed, developed, and deployed. They also emphasized that continuous systematic evaluations of e-Health interventions are needed.
Nabukenya and Justus Ashaba employed a qualitative approach to establish perspectives of key informants with respect to e-Health evaluation practices and challenges faced. Most of the participants reported the system availability, system response speed, interoperability, usability, scalability, and availability of human resources to implement the e-Health initiatives as indicators that were currently considered during evaluations.
Vukovic et al. showed that available HTA reports on e-Health/m-Health technologies are heterogeneous in terms of transparency and thoroughness. These reports failed to tackle the relevant assessment elements, especially ethical, social, and organizational implications. They emphasized that HTA should provide a solid description of the technical, economic, clinical, legal, ethical, social, and organizational aspects related to the use of health technology.
The WHO's (2016) guideline on monitoring and evaluating DHIs grouped evaluation of DHIs into four categories: (i) qualitative approaches, (ii) quantitative approaches, (iii) mixed-methods approaches, and (iv) financial and economic evaluations (EEs). DHI projects typically begin with exploring basic questions of whether the intervention addresses the identified needs, including technical functionality and feasibility, followed by the assessment of user satisfaction, then move toward efforts to evaluate the effectiveness, attributable impact, and ultimately, “value for money” of the intervention.
Lau and Kuziemsky (2017) in Hand Book of e-Health Evaluation described six empirical frameworks that have been used for the evaluation of e-Health systems. These frameworks serve a similar purpose by providing an organizing scheme for e-Health practitioners to describe and predict the factors and processes that influence, design, implementation, use, and effect of e-Health system in a given health setting. BE framework, clinical adoption (CA) framework, CA meta-model framework, EE framework, pragmatic HIT evaluation framework, and holistic e-Health value framework were the proposed frameworks described in this handbook. [Table 1] illustrates the different approaches for e-Health assessment.
| Discussion|| |
In this scoping review, we performed a rigorous search of the literature to provide a narrative description of various approaches suggested to assess e-Health programs. We identified 15 types of approaches that were suggested by various authors. This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. We tried to provide a summary and tabulated these diverse approaches to provide an articulated body of literature on the current state of knowledge about the assessment of e-Health interventions. Assessment of e-Health can be carried out using a single approach or a combination of different approaches. The approach can be formative/summative, objective/subjective, and qualitative/quantitative., Formative assessment is conducted during the time of implementation. Information acquired through it could be used to shape the activity as it happens, but in contrast, summative assessment is conducted at the end of the implementation phase and is used to make decisions about the success or failure. Constructive e-Health evaluation presented by Høstgaard et al., provides an illustration of a full evaluation framework for the entire e-Health development life cycle involving both formative and summative assessments. According to the WHO, formative evaluations should be carried out internally by the organization itself so that its findings can be utilized to influence program plan and implementation. Summative evaluations, on the other hand, should be conducted by an external body so that it could help to generate evidence important for improvement, implementation, and expansion of the program. Objective assessment is described by measurable properties that are generally independent of the observer, whereas subjective assessment is based on the perception of the observers or users. DEDHI framework which provides assessment criteria based on the life cycle of DHI includes both subjective (e.g., perceived benefit of a DHI) and objective (e.g., adherence to a DHI) evaluation criteria. Qualitative assessment can be used to assess user satisfaction and border context of system workflow, whereas quantitative assessment tends to measure changes in process, outputs, and outcomes. These approaches were used by digital health benefits evaluation frameworks which assessed the five benefits workstream of the My Health Record System in Australia using qualitative, quantitative, and mixed-method design.
Assessment can also be performed considering organizational, technological, social, clinical, ethical, and economic aspects. These aspects can be considered during the various life-cycle phases of e-Health intervention and could differ in each phase depending on the activities of the phases. Continuous Systemic Evaluation proposed by Catwell and Sheikh from Centre for Population Health Sciences, Scotland, tried to provide a multifaceted and multidiscipline assessment approach to evaluate e-Health interventions taking account of political, social, organizational, and technical contexts. Various frameworks and approaches discussed in this review have its own strengths and limitations. For instance, the minimum HTA inspired framework covers most of the context mentioned above, but it is unable to consider interoperability, data quality, and accessibility and usability criteria. Similarly, the Infoway BE framework which assesses e-Health on three broad conceptual dimensions of health information system quality, use, and net benefits does not cover organizational and contextual factors.,
It is very difficult to evaluate all possible outcomes and to develop a standardized single approach to assess e-Health programs and intervention. This might be attributed to the complex nature of the health system and evolving health technologies. This review has a few limitations. Non-English language studies were not considered in this review as researchers were lacking competency in using the non-English language studies, quality appraisal of studies was not carried out and even after doing a comprehensive search, only 15 studies were identified which fulfilled the inclusion criteria.
| Conclusions|| |
This review showed that available literature on the assessment of e-Health programs is heterogeneous in terms of the methodology used. e-Health interventions are highly contextual; therefore; the phase of maturity and objective of the assessment should be considered while carrying out the assessment of e-Health programs.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Regional Strategy for Strengthening E-Health in the South-East Asia Region. Health Situation and Trend Assessment (HST) Department of Health Systems Development Department (HSD). Regional Office for South East Asia: WHO; 2014–2020.
Lau F, Kuziemsky C, editors. Handbook of eHealth Evaluation: An Evidence-based Approach. Victoria (BC): University of Victoria; 2017. Available from: https://www.ncbi.nlm.nih.gov/books/NBK481590/
. [Last accessed on 2019 Nov 15].
Enam A, Torres-Bonilla J, Eriksson H. Evidence-based evaluation of eHealth interventions: Systematic literature review. J Med Internet Res 2018;20:e10971.
Peters MDJ, Godfrey C, McInerney P, Baldini Soares C, Khalil H, Parker D. Chapter 11: Scoping reviews. In: Aromataris E, Munn Z, editors. Joanna Briggs Institute Reviewer's Manual. South Australia: The Joanna Briggs Institute; 2017. Available from: https://reviewersmanual.joannabriggs.org/
. [Last accessed on 2019 Nov 20].
Kowatsch T, Otto L, Harperink S, Cotti A, Schlieter H. A design and evaluation framework for digital health interventions. It Inform Technol 2019;61:253-63.
Høstgaard A, Bertelsen P, Nøhr C. Constructive eHealth evaluation: Lessons from evaluation of EHR development in 4 Danish hospitals. BMC Med Inform Decis Mak 2017;17:45.
Biggs JS, Willcocks A, Burger M, Makeham MA. Digital health benefits evaluation frameworks: Building the evidence to support Australia's National Digital Health Strategy. Med J Aust 2019;210 Suppl 6:S9-11.
Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al
. Evaluating digital health interventions: Key questions and approaches. Am J Prev Med 2016;51:843-51.
Catwell L, Sheikh A. Evaluating eHealth interventions: The need for continuous systemic evaluation. PLoS Med 2009;6:e1000126.
Vukovic V, Favaretti C, Ricciardi W, de Waure C. Health technology assessment evidence on E-Health/M-Health technologies: Evaluating the transparency and thoroughness. Int J Technol Assess Health Care 2018;34:87-96.
World Health Organization. Monitoring and Evaluating Digital Health Interventions: A Practical Guide to Conducting Research and Assessment. Switzerland: World Health Organization; 2016.
Lau F, Hagens S, Muttitt S. A proposed benefits evaluation framework for health information systems in Canada. Healthc Q 2007;10:112-6, 118.
Lau F, Price M, Keshavjee K. From benefits evaluation to clinical adoption: Making sense of health information system success in Canada. Healthc Q 2011;14:39-45.
Price M, Lau F. The clinical adoption meta-model: A temporal meta-model describing the clinical adoption of health information systems. BMC Med Inform Decis Mak 2014;14:43.
Joshi NK, Bhardwaj P, Suthar P, Jain YK, Joshi V, Manda B. Assessment of monitoring and online payment system (Asha Soft) in Rajasthan using benefit evaluation (BE) framework. J Family Med Prim Care 2020;9:2405-10. [Full text]
[Figure 1], [Figure 2]