Evaluation of global health capacity building initiatives in low-and middle-income countries: A systematic review

Hady Naal; Maria El Koussa; Melissa El Hamouch; Layal Hneiny; Shadi Saleh

doi:10.7189/jogh.10.020412

Low- and middle-income countries (LMICs) face increasing health challenges, coupled with a reduced ability to manage them [1,2]. According to the World Health Organization (WHO), there are around 57 LMICs experiencing significant health care crises and shortages of their health workforce [3,4]. LMICs are in dire need of a competent health workforce that has the capability to overcome the shortcomings of their fragile health care systems [1,2]. While global health is a multidisciplinary field that aims to resolve transnational health challenges for people worldwide, the health workforce refers to decision-makers and service providers who engage in action to enhance the health of populations [5,6]. The means to implement Global Health Capacity Building (GHCB) initiatives are becoming increasingly available given the ubiquitous advancements in technology and delivery methods [7,8]. GHCB initiatives are activities directed towards developing the abilities of individuals, organizations, and communities to manage global health-related issues [9,10]. These efforts are among the most important approaches to respond to such challenges in LMICs [11] since they have a large potential in improving health-related knowledge and skills, policies and practices, ultimately leading to changes in health care systems and health outcomes [11,12]. To develop, implement, and sustain these efforts, the use of adequate evaluation methods is necessary to gauge the effectiveness and impact of GHCB initiatives. However, little accounts of approaches used to evaluate GHCB initiatives in LMICs have been reported.

Current approaches to evaluate the impact of GHCB initiatives at the individual, organizational, and community levels have not been standardized. This is problematic because GHCB initiatives risk being ineffective and stagnant in the absence of adequate evaluation methods. This may also limit researchers’ and stakeholders’ ability to develop and upscale their own capacity building initiatives. Adequate evaluation is therefore a central component of GHCB initiatives and is an integral step to examine the impact associated with them [12]. In light of the aforementioned shortcomings, the present study systematically reviews and critically appraises the literature in order to identify and summarize commonly used evaluation methods for GHCB initiatives.

METHODS

Search strategy and selection criteria

This systematic review conforms to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) [13]. The search strategy was conducted by a medical librarian (LH) to identify relevant studies using the following databases: Medline (OVID), Embase.com, PubMed, and Scopus. The search was restricted by date only, and included articles published between January 1, 2009 until the day of the final search on August 19, 2019. According to the Best Evidence Medical Education (BEME) collaboration, a 10-year cut off is usually recommended for search strategies in systematic reviews [14]. The search strategy aimed to identify articles that addressed outcome evaluation methods used in GHCB initiatives in LMICs. Accordingly, Medical Subject Headings (MeSH) and keywords of the four concepts in our topic were combined in the search strategy (see Appendix S1 in the Online Supplementary Document), namely “Capacity Building”, “Global Health”, “Evaluation”, and “Low- and Middle-Income Countries”. We also searched Open Grey database for grey literature using a similar search strategy.

Studies were included if they were peer-reviewed journal articles, published in English. Articles were required to be related to GHCB, and specifically to report on outcome evaluation methods targeting at least one or more components of the GHCB learning modality (Table 1). Articles were excluded if they were editorials, commentaries, presentations, needs assessment or cross-sectional studies, prevention and intervention studies that did not evaluate the modality, poster abstracts, or books. Articles were also excluded if they were published before 2009, if they did not address an LMIC, if they were not in English, if they did not relate to GHCB, and if they did not report an outcome evaluation method. Finally, process evaluations, feasibility studies, and interventions that only reported intervention outcomes were excluded. While pre-and-post changes give insight into the effectiveness of the intervention, they do not tap into the learners’ reactions to and experiences with the GHCB initiative, which is this reviews’ primary focus.

Table 1. Definitions of key terms

Key terms	Definitions
Global Health	“Health problems, issues, and concerns that transcend national boundaries, which may be influenced by circumstances or experiences in other countries, and which are best addressed by cooperative actions and solutions” [15]
Capacity Building	“The development of knowledge, skills, commitment, structures, systems, and leadership to enable effective health promotion…[with] actions to improve health at three levels: the advancement of knowledge and skills among practitioners; the expansion of support and infrastructure for health promotion in organizations, and; the development of cohesiveness and partnerships for health in communities” [16]
Outcome Evaluation	An evaluation that measures changes the program has made in participants/clients based upon program objectives [17]
Learning Modality	Interventions, trainings, education, mentorships, delivered through face-to-face or in-person modes, online, or a combination of both [18].
Population Groups	Professional Personnel are individuals who received formal education and/or training in related global health fields (eg, doctors, nurses, professors, researchers etc…), Community Workers are individuals who have not received formal education and/or training (eg, community nurses, community health workers etc…), but who have some qualification in global health to practice within their community, and General Public (eg, community members, parents of school students etc…) includes individuals who do not have formal education and/or training, and who do not practice in any area related to global health [19]

LH conducted the search process, extracted all results into the Endnote software, and shared it with two reviewers (HN and MEH) who conducted the screening process after a calibration exercise. After excluding all duplicates, the two reviewers independently screened the titles and abstracts based on set criteria for eligibility. A third reviewer (MEK) was assigned to resolve disagreements. Next, full texts were retrieved, were scrutinized by the two initial reviewers, and conflicts were also resolved by the same third reviewer. Data were extracted into a pre-established Excel sheet in preparation for data synthesis and analysis, which was conducted on the Statistical Package for the Social Sciences (SPSS) (IBM, Armonk, NY, USA).

Data analysis

The data was extracted by one reviewer (HN) and was analysed by two reviewers (HN, MEK). Besides the bibliography (which included the date, name of first author, title, region, and country) and summaries of included studies (which included duration of the initiatives, sample size, key findings, conclusions, learning outcomes, and limitations), two main categories of data were extracted and reported in Table 2. The first related to the GHCB initiative, and included data regarding the global health theme, design of the study, learning modality, sample size, and learning outcomes. The second related to the evaluation approach, and it included data regarding the level of evaluation, target population, method of evaluation, evaluation tool, administration frequency (time point of data collection), the measured outcome, and the conclusions and limitations.

Table 2. Summary of findings

Region	Topic	Design	Population	Modality	Level of Evaluation	Method of Evaluation	Time Point
Africa
Asgary et al. (2016) [20]	NCD	Quantitative	Community Workers (Community health nurses)	Blended	Individual	Face-to-face	Post
Brantuo et al. (2014) [21]	SRH	Quantitative	Professional Personnel (health care providers; hospital nurses, doctors etc...)	Face-to-face	Individual & Organizational	Face-to-face & desk review	Pre-Post
Comeau et al. (2018) [22]	CD	Qualitative	Professional Personnel (Graduate students, postdoctoral trainees, academics etc...)	Blended	Individual	Face-to-face	Post
Crocker et al. (2016) [23]	Public health	Qualitative	Professional Personnel (Government officials)	Face-to-face	Individual & Organizational	Face-to-face	More than 3 mo
Davila et al. (2015) [24]	NCD	Mixed methods	Professional Personnel (epidemiologically trained hospital staff and ministry of health staff)	Face-to-face	Individual & Organizational	Face-to-face	More than 3 mo
Feldacker et al. (2014) [25]	Health system	Quantitative	Professional Personnel (Mid-level health care clinicians)	Face-to-face	Individual	Face-to-face	More than 3 mo
Garley et al. (2016) [26]	Health system	Quantitative	Professional Personnel (M&E Officers working in ministries of health, NGO, development projects etc…)	Blended	Individual	Face-to-face & online	More than 3 mo
Huber et al. (2014) [27]	Health system	Quantitative	Professional Personnel (Health practitioners, researchers, managers)	Face-to-face	Individual	Face-to-face	Post
Makanjuola et al. (2012) [28]	Mental health	Qualitative	Professional Personnel (mental health college teachers)	Face-to-face	Individual	Face-to-face	More than 3 mo
Matovu et al. (2013) [29]	Health system	Qualitative	Professional Personnel (Medical / nursing / clinical officers; social scientists etc…)	Face-to-face	Individual & Organizational	Face-to-face	Post
Mpofu et al. (2014) [30]	Health system	Mixed methods	Professional Personnel (health care students)	Face-to-face	Individual	Face-to-face	Post
Muchiri et al. (2016) [31]	NCD	Qualitative	General Public (adults diabetes patients)	Face-to-face	Individual	Face-to-face	More than 3 mo
Mutabaruka et al. (2010) [32]	Public health	Mixed methods	Professional Personnel (managers of immunisation programs)	Face-to-face	Individual & Organizational & Country	Face-to-face	Unspecified
O'Donovan et al. (2018) [33]	CD	Quantitative	Community workers (community health workers)	Blended	Individual	Face-to-face	Pre-Post
Plowright et al. (2018) [34]	SRH	Quantitative	Community workers (community health workers)	Face-to-face	Individual	Face-to-face	Pre-Post
Sarli et al. (2010) [35]	Health system	Qualitative	Community Workers (community health workers)	Face-to-face	Individual & Community	Face-to-face	More than 3 mo
Shimpuku et al. (2018) [36]	SRH	Mixed methods	General Public (pregnant women and family members)	Face-to-face	Individual	Face-to-face	Pre-Post
Sibeko et al. (2018) [37]	Mental health	Mixed methods	Community Workers (community health workers)	Face-to-face	Individual	Face-to-face	Pre-Post & up to 3 mo post
Tekola et al. (2019) [38]	Mental health	Qualitative	General Public (caregivers of children with developmental disorders) & Professional Personnel (Healthcare providers and other stakeholders)	Face-to-face	Individual	Face-to-face	Post
Wills et al. (2010) [39]	Health system	Qualitative	Community workers (Primary health care workers)	Face-to-face	Individual & Organizational	Face-to-face	Post
Witek-McManus et al. (2015) [40]	CD	Mixed methods	General Public (School teachers)	Face-to-face	Individual	Face-to-face	More than 3 mo
Fish et al. (2019) [41]	NCD	Mixed methods	Professional Personnel (health care providers; clinical and surgical oncologists)	Face-to-face	Individual	Online & Face-to-face	Pre-Post
Asia:
Abdel-All et al. (2018) [42]	NCD	Mixed methods	Community workers (accredited social health activists)	Face-to-face	Individual	Face-to-face	Pre-Post & up to 3 mo post
Carlos et al. (2015) [43]	CD	Quantitative	Professional Personnel (health care providers - doctors, nurses, medical technicians)	Face-to-face	Individual	Face-to-face	Pre-Post
Choi et al. (2016) [44]	Mental health	Quantitative	General Public (parents of adolescents)	Online	Individual	Online & Over the phone	Pre-Post & up to 3 mo post
Gao et al. (2018) [45]	Health system	Mixed methods	Professional Personnel (health workers, academics, researchers, government officials etc…)	Blended Learning	Individual	Online & Over the phone	Pre-Post & up to 3 mo post
Gyawali et al. (2018) [46]	NCD	Mixed methods	Professional Personnel (female community health volunteers)	Face-to-face	Individual	Face-to-face	Pre-Post
Jung et al. (2009) [47]	Health system	Mixed methods	General Public (community leaders)	Face-to-face	Individual	Face-to-face	Pre-Post
Kang et al. (2015) [48]	Health system	Quantitative	Professional Personnel (hospice and palliative health care experts)	Face-to-face	Individual	Face-to-face & Mailed survey	Post
Kusuma et al. (2019) [49]	CD	Quantitative	General Public (household samples)	Face-to-face	Community	Face-to-face	Pre-Post & up to 3 mo post
Limato et al. (2018) [50]	SRH	Qualitative	Community workers (community health workers) & General Public	Face-to-face	Individual & Community	Face-to-face	More than 3 mo
Monoto et al. (2018) [51]	SRH	Qualitative	Community Workers (Breastfeeding Peer Counsellors)	Blended Learning	Individual	Face-to-face	Pre-Post & up to 3 mo post
Negandhi et al. (2015) [52]	Health system	Mixed methods	Professional Personnel (Public health professionals)	Face-to-face	Individual	Online & Over the phone	Post
Nugroho et al. (2019) [53]	SRH	Quantitative	Community workers (outreach workers)	Face-to-face	Individual	Face-to-face	Pre-Post & up to 3 mo post
Shen et al. (2018) [54]	NCD	Quantitative	Community Workers (community nurses) & General Public (patients)	Face-to-face	Individual	Face-to-face	Pre-Post
Slakey et al. (2016) [55]	Surgical care	Qualitative	Professional Personnel (Healthcare providers)	Blended Learning	Individual	Online & Face-to-face	More than 3 mo
Sranacharoenpong et al. (2009) [56]	NCD	Quantitative	Community Workers (Community Healthcare Workers)	Blended Learning	Individual	Online & Face-to-face	Pre-Post
Wang et al. (2017) [57]	Mental health	Qualitative	Professional Personnel (Healthcare professionals - General practitioners and nurses	Face-to-face	Individual	Face-to-face	Pre-Post & up to 3 mo post
Xu et al. (2016) [58]	Health system	Quantitative	Professional Personnel (Undergraduate medical students)	Face-to-face	Individual	Face-to-face	Pre-Post
Yang et al. (2018) [59]	Mental health	Quantitative	Professional Personnel (Mental health professionals from colleges, hospitals, and health centers)	Face-to-face	Individual	Face-to-face	Pre-Post
Zhan et al. (2017) [60]	Health system	Quantitative	Community workers (Primary health care workers)	Blended Learning	Individual	Online & Face-to-face	Pre-Post
Zhang et al. (2012) [61]	Mental health	Quantitative	Professional Personnel (Physicians - pulmonogists)	Face-to-face	Individual	Face-to-face	Pre-Post
Zhao et al. (2019) [62]	SRH	Mixed methods	Professional Personnel (doctors, midwifes, directors, officers)	Face-to-face	Individual	Face-to-face	Post
Ziganshin et al. (2015) [63]	Health system	Quantitative	Professional Personnel (Physicians and medical trainees)	Face-to-face	Individual	Face-to-face	Post
Caribbean:
Cianelli et al. (2013) [64]	Mental health	Qualitative	Professional Personnel (health care workers - nurses, physicians etc…)	Face-to-face	Individual	Face-to-face	Post
Knettel et al. (2017) [65]	SRH	Mixed methods	Community workers (Community health workers)	Face-to-face	Individual	Face-to-face	Pre-Post
Rivera et al. (2018) [66]	NCD	Quantitative	Professional Personnel (Public health graduate students)	Face-to-face	Individual	Face-to-face	Pre-Post
Central America
Gonzalez et al. (2016) [67]	Public health	Quantitative	Professional Personnel (health care workers - physicians, nurses, nutritionists, respiratory technicians etc…)	Online	Individual	Online	Pre-Post
McConnell et al. (2017) [68]	Health system	Quantitative	Professional Personnel (nurses)	Online	Individual	Online	Pre-Post
Molokwu et al. (2016) [69]	NCD	Quantitative	General public (Primary care patients - women)	Face-to-face	Individual	Face-to-face	Pre-Post
MENA:
Abdelhai et al. (2012) [70]	SRH	Quantitative	Professional Personnel (Public Health Students)	Online	Individual	Online & Face-to-face	Pre-Post
Gholipour et al. (2018) [71]	Health system	Quantitative	Professional Personnel (heads of health centers, health managers, health deputies)	Face-to-face	Individual	Face-to-face	Pre-Post
Omar et al. (2009) [72]	Health system	Mixed methods	Professional Personnel (health managers)	Face-to-face	Individual & Organizational	Face-to-face	More than 3 mo
Siabani et al. (2016) [73]	NCD	Quantitative	General Public (patients with chronic heart failure)	Face-to-face	Individual	Face-to-face	Pre-Post & up to 3 mo post
Pacific:
Fung et al. (2015) [74]	Mental health	Qualitative	Professional Personnel (mental health clinicians)	Face-to-face	Individual	Online & Face-to-face	Pre-Post
Charlson et al. (2019) [75]	Mental health	Mixed methods	Professional Personnel (health care workers - nurses medical officers, counsellors)	Face-to-face	Individual	Face-to-face	Post
International (multiple):
Khan et al. (2018) [76]	NCD	Quantitative	Professional Personnel (nurses, researchers, medical assistants etc…) & Community Workers (Community health workers)	Face-to-face	Individual	Face-to-face	Post
South America
Gomes et al. (2015) [77]	NCD	Qualitative	Community Workers (community health workers)	Face-to-face	Individual	Face-to-face	More than 3 mo
Hull et al. (2012) [78]	Surgical care	Qualitative	Professional Personnel (post-graduate medical students)	Face-to-face	Individual	Face-to-face	Pre-Post
Joshi et al. (2011) [79]	Health system	Quantitative	Professional Personnel (health care providers - physicians, nurses, dentists) & Community workers (Community health workers)	Online	Individual	Online	Post
Monier et al. (2019) [80]	CD	Quantitative	Professional Personnel (post-graduate health care students)	Online	Individual	Online	Post
Pereira et al. (2015)[81]	Mental health	Quantitative	Professional Personnel (health care providers)	Blended Learning	Individual	Online	Pre-Post & up to 3 mo post
Vieira et al. (2014)[82]	Mental health	Quantitative	General Public (School teachers)	Face-to-face	Individual	Face-to-face	Pre-Post

CD – communicable disease, NCD – non-communicable diseases, SRH – sexual and reproductive health

Risk of bias and methodological quality of included studies was assessed using the McGill Mixed-Methods Appraisal Tool (MMAT) [83]. Since the aim of this review is to appraise evaluation approaches in GHCB, a meta-analysis was not applicable given that we did not examine effect sizes associated with the interventions. Instead, we opted for a qualitative and descriptive approach for data analysis.

RESULTS

Overview

Of the 8324 eligible articles, 200 studies met the inclusion criteria based on title and abstract screening and were eligible for full-text review, from which 63 articles were included for final analysis (Figure 1). Included studies (Table 2) reported initiatives conducted in Africa, Asia, Caribbean, Central America, Middle East and North Africa, Pacific, and South America. Global Health topics of the initiatives were categorized under mental health (19.0%), health system (28.6%), non-communicable diseases (20.6%), communicable diseases (9.5%) sexual and reproductive health (14.3%), public health (4.8%), and surgical care (3.2%).

Figure 1. PRISMA flowchart.

Characteristics of the studies

Three main population groups were addressed in the aforementioned initiatives, categorized into professional personnel (57.1%), community workers (20.6%), general public (14.3%) or mixed (7.9%). The most commonly adopted learning modality was face-to-face (74.6%), followed by blended (15.9%) and online learning (9.5%). Most evaluations were conducted on the individual level (84.1%), whereby only a handful of studies evaluated initiatives on the organizational, community, or country levels. The majority of studies relied on pre-and-post or only post intervention time intervals for evaluation (64.5%), and a small number reported up to 3 months follow-ups (16.1%) or more (19.4%). Capacity building initiatives were mostly evaluated using face-to-face methods (76.2%), and a limited number used online methods (7.9%) or a combination of online and other methods (15.9%). The most commonly used tools were questionnaires (63%), followed by open-ended or close-ended evaluation forms (49%), semi-structured interviews (35%), focus groups (25%), online surveys (22%), expert/peer rating (11%), case studies (9%), researcher observations (6%), desk reviews (3%), and reflective commentaries (2%).

Closer examination of the results revealed that African and Asian countries reported the most GHCB initiatives evaluations (69.8%). In comparison to other regions, they invested most in longer-term evaluations spanning 3 months or more (75%). African countries specifically, almost exclusively placed emphasis on evaluating modalities beyond the individual level. Across studies, GHCB initiatives were directed mostly towards professional personnel, and online evaluation methods were almost exclusively used for them. Online evaluation methods were only used for individual level evaluation, whereby other levels of evaluation such as organizational or community levels were only assessed face-to-face. Also, long-term evaluations spanning 3 months, or more, were conducted face-to-face. Results show that over the past 4 years, online methods of evaluation have been increasingly used in GHCB.

Evaluation of learning modalities

With regard to learning modalities, blended and online GHCB initiatives were only evaluated on the individual level, whereby organizational, community, and country level evaluations were done only for face-to-face trainings. Similarly, long-term evaluations of 3 months or more were done for face-to-face modalities, whereby only 2 blended-approach initiatives - that had a face-to-face component - were evaluated at 3 months follow-up or more, and none for online approaches. Blended and online GHCB initiatives were also exclusively conducted for community workers and professional personnel, with the exception of 1 study that trained a sample of the general public through an online approach. Finally, results show that online and blended modalities are starting to increase in number over the past 4 years.

Variables and indicators

In almost all capacity building initiatives, the main purpose was to improve the knowledge, capacity, skills, confidence, or performance of their participants. Evaluations and measured outcomes of the initiatives targeted (1) changes in knowledge, attitudes, skills, and performance, (2) perceptions, experiences, and satisfaction with the training, and (3) organizational, community, and country level impact in rare instances. Indicators for evaluations on the individual level focused on reactions, perceptions, satisfaction, experiences, and expectations, and on changes in knowledge, skills, practices, performance. Evaluations on the organizational level focused their indicators on assessing transfer (and barriers to application) of knowledge to the organization through learners’ technical input, performance, leadership, and relationship with colleagues, in addition to desk reviews of organizational-level changes such as policies, outputs, and practices. On the community-level, indicators mainly addressed the relationship between community members and health care services such as individuals’ knowledge of available services and related personnel, their health care practices, and their perceptions of health problems in the community.

Study design

Articles used a mixture of qualitative (25.4%), quantitative (49.2%), and mixed-methods (25.4%) approaches, and all online studies were quantitative. Pre-and-post intervention changes along with satisfaction with trainings were generally measured quantitatively, whereas experiences and perceptions of the training were measured qualitatively. Organizational, community, and country level impact were assessed using one or both approaches. Finally, the GHCB initiatives overall improved knowledge, skills, and practices, whereby interactive modalities and engaging approaches were most important for knowledge retention and application. Online and blended modalities were reported to be generally effective and accepted by population groups, especially those that provide increased engagement and interaction. Common stated limitations included the need for longer-term and wider evaluations beyond individual-level learning and reaction.

Risk assessment

Based on the MMAT tool used to evaluate the quality of studies in this review, a small number of studies (n = 9) scored low (2/4), and the rest (n = 54) scored higher (3/4 or 4/4) on quality assessment (Table 3).

Table 3. MMAT Risk of bias summary

Study ID (first author & year)	MMAT rating (out of ****)	Main limitations
Abdel-All, 2018	**	-Did not report response rate
Abdel-All, 2018	**	-Did not appropriately consider the limitations associated with integration of qualitative and quantitative data.
Abdelhai, 2012	***	-Unacceptable response rate
Asgari, 2016	***	-Measurements are not appropriate to answer the research question
Brantuo, 2014	**	-Measurements inappropriate.
Brantuo, 2014	**	-Did not report the response rate.
Carlos, 2015	****
Charlson, 2019	**	-Did not consider how findings relate to the context in which the data was collected
Charlson, 2019	**	-Did not consider how findings relate to the researchers influence.
Choi, 2016	***	-Unacceptable response rate
Cianelli, 2013	****
Comeau, 2018	****
Crocker, 2016	***	-Did not consider how findings relate to the researchers influence.
Davila, 2015	***	-Did not consider how findings relate to the researchers influence
Feldacker, 2015	****
Fish, 2019	**	-Did not report the response rate
Fish, 2019	**	-Did not take into account the difference between groups (did not list the demographic characteristics of the participant)
Fung, 2015	****
Gao, 2018	****
Garley, 2016	**	-Did not report the response rate
Gholipour, 2018	****
Gomes, 2015	**	-Did not consider how findings relate to the researchers influence.
Gomes, 2015	**	-Did not consider how findings relate to the context in which the data was collected
Gonzalez, 2016	***	-Unacceptable response rate
Gyawali, 2018	***	-The integration of quantitative and qualitative data are not relevant to answer the research question
Huber, 2014	***	-Did not take into account the difference between groups (did not list the demographic characteristics of the participant)
Hull, 2012	****
Joshi, 2011	**	Did not report the response rate
Joshi, 2011	**	-Did not take into account the difference between groups (did not list the demographic characteristics of the participant)
Jung, 2009	****
Kang, 2015	****
Khan, 2018	****
Knettel, 2017	****
Kusuma, 2019	****
Limato, 2018	****
Makanjuola, 2012	****
Matovu, 2013	***	-Did not consider how findings relate to the researchers influence
McConnell, 2017	***	-Did not report response rate
Molokwu, 2016	***	-Unacceptable response rate
Monier, 2019	***	-Did not take into account the difference between groups (did not list the demographic characteristics of the participant)
Monoto, 2018	**	-Process for analyzing qualitative data are not relevant to answer the research question
Mpofu, 2014	***	-Did not report response rate
Muchiri, 2016	***	-Did not consider how findings relate to the researchers influence
Mutabaruka, 2010	***	-Unacceptable response rate
Negandhi, 2015	****
Nugroho, 2019	****
O'Donovan, 2018	****
Omar, 2009	****
Pereira, 2015	****
Plowright, 2018	****
Rivera, 2018	***	-Did not report the response rate
Sarli, 2010	****
Shen, 2018	****
Shimpuku, 2018	****
Siabani, 2015	***	-Unacceptable response rate
Sibeko, 2018	****
Slakey, 2016	***	-Did not consider how findings relate to the researchers influence
Sranacharoenpong, 2009	***	-Did not report the response rate
Tekola, 2019	***	Did not consider how findings relate to the researchers influence
Vieira, 2014	***	-Sample not representative of the population.
Wang, 2017	***	-Did not consider how findings relate to the researchers influence
Wills, 2010	***	-Did not consider how findings relate to the researchers influence
Witek-McManus, 2015	***	-The integration of quantitative and qualitative data are not relevant to answer the research question
Xu, 2016	**	-Did not report the response rate
Xu, 2016	**	-Did not take into account the difference between groups (did not list the demographic characteristics of the participant)
Yang, 2018	***	-Did not report the response rate
Zhan, 2017	****
Zhang, 2012	***	-Did not report response rate
Zhao, 2019	***	-Did not report response rate
Ziganshin, 2015	****

MMAT – mixed-methods appraisal tool

DISCUSSION

To the best of our knowledge, this is the first systematic review on the approaches used to evaluate GHCB initiatives in LMICs. To-date, only one review addressed capacity building tools [84], however it was limited to those targeting health research, a subcomponent of GHCB. Another systematic review addressed the effectiveness of capacity building initiatives in the public health domain [12], however authors did not focus on evaluation approaches. Both reviews, although did not focus on LMICs, highlighted the need for the development and refinement of more consistent evaluation approaches to measure larger and long-term impact of capacity building in general. Considering this increasing interest in capacity building, along with the noticeable knowledge gaps, the urgent need to cover a wider and more comprehensive scope of GHCB with a focus on evaluation approaches is well-warranted, especially in LMICs. This is a timely study given the advancements in capacity building delivery and evaluation methods, the growing global health burdens in LMICs, and the increasing need for the development of a competent health workforce in LMICs [1,85]. Evaluations are essential to assess the impact of GHCB initiatives on different levels, to assess whether or not their goals were achieved, and to highlight areas that may require improvements. Adequate evaluations are also important to gauge the utility of, and strategically allocate funds. We advance the literature by critically appraising the studies on this topic, by synthesising their findings, and by recommending increased standardization in evaluation approaches.

Overall, our findings indicate that most of the initiatives are delivered and evaluated through face-to-face modalities and this is not surprising, given the low resource settings that characterize LMICs, and their documented slow adoption of technology [86]. Interestingly however, based on our review, there has been a shift over the past 4 years towards integrating technology in the delivery and evaluation of GHCB initiatives. Of those, blended approaches that still include a face-to-face component are more prevalent than purely online modalities, potentially because such a transition is complex and may require more time, technological literacy, adequate infrastructure and cultural acceptance [86,87]. Nevertheless, such a shift may offer increased practicality and accessibility for the delivery and evaluation of GHCB initiatives given the potential of technology in overcoming barriers associated with traditional capacity building delivery methods [88]. For example, LMICs tend to have limited access to global health education, and technology has the potential to address this problem by allowing the transfer of knowledge from developed to developing countries in an easier, more accessible and cost-effective manner [88,89]. Our results indicate that clinically related GHCB initiatives included a face-to-face component, whereby purely online initiatives were didactic and did not include a hands-on approach. This may be an important avenue for future research, namely to assess the delivery of clinical practicums through online modalities. Finally, only one online intervention addressed the general public, whereby professional personnel and community workers were predominantly the target of the reviewed GHCB initiatives.

Our results build on previous research and point towards a growing need for better evaluation approaches for GHCB initiatives [12,84], especially for online and blended modalities. Customarily, higher-income countries tend to place more emphasis on adequate evaluation approaches to assess higher level and longer-term impact of their initiatives on the individual, organizational, and community levels in comparison to LMICs [84]. In one systematic review, most of the included studies that assessed long-term organizational-level impact of capacity building were from higher-income countries [84]. Evidently, higher levels of evaluation are time consuming, and require more sophisticated and complex methods of measurement [90]. In this review, while a small number of face-to-face approaches evaluated their initiatives long-term, and assessed impact beyond individual level, none of the online initiatives did so, and only 2 blended GHCB initiatives conducted evaluations beyond the 3 months-time period. As a result, we were not able to draw further conclusions regarding blended and online modalities beyond the individual-level assessment of reaction, knowledge, attitudes, and practices, or their long-term efficacy. This is especially important since blended and online approaches were effective and accepted by the participants, which means that a shift toward technology-assisted GHCB may be promising [67,88]. There is thus an urgent need to refine evaluation approaches used in initiatives delivered through technology-assisted modalities in order to better assess their wider and longer-term impact. While individual-level assessment offers important first-level insight, it does not tap into higher levels of assessing organizational or community-level changes such as shifts in policy, practices, or access to health care services to name a few. One way to do that is through the standardization of evaluation approaches, a process by which assessments are developed to have uniform applications with consistent procedures across studies [91].

Standardization may simplify and provide a better structure for the evaluation of initiatives based on the GHCB modality, vis-à-vis population groups, evaluation methods, level of evaluation, frequency of evaluation, and evaluation tools, and may enhance comparability of outcomes. That said, we identified much variability in the evaluation methods and tools used across studies, whereby evaluations deviated from standardization in terms of the lack of consistency in the items/questions, set variables, indicators, and outcomes. Also, few studies reported the actual tools that were used, and this is problematic because it dissipates the efforts and resources allocated towards the evaluation of GHCB, in that authors tended to develop their own tools rather than use or adapt existing ones. It is important for future studies to be more consistent in their evaluation approaches. Ultimately, this means that GHCB initiatives in LMICs need standardized evaluation tools to measure the effectiveness of learning modalities beyond the well-addressed individual-level assessment of knowledge, attitude, and practice. These tools may be adapted to the context and approach of each initiative.

Results of this study should be interpreted in light of several limitations. Most significantly, the reviewed articles are not representative of GHCB initiatives among LMICs, given that we only included those evaluating the outcome of at least one component of the modality. Also, studies may have been missed due to the search parameters used, which is a common problem in the literature. However, we minimized this by exhausting the search process through the key terms, and search strategy employed across multiple academic and grey literature databases. In addition, many LMICs may prioritize reporting to donors as opposed to publishing their findings, which may affect the accurate representation of studies included in this review. Finally, we had a limited focus on demographic groups and other related capacity building data, because it was beyond the scope of the review: our main goal was to review the methods and evaluation approaches.

CONCLUSION

This review provides the first summary of evaluation approaches used in GHCB in LMICs and identifies several gaps that need to be addressed. Most significantly, because online and blended modalities are promising avenues for GHCB, more attention should be placed on evaluating them long-term beyond the individual level in order to maximize their impact and sustainability. In addition to that, standardized evaluation tools for GHCB are needed in order to enhance efficiency and comparability of results across initiatives.

Additional material

Online Supplementary Document

Acknowledgments

Authors thank Dina Muhieddine for her help compiling the risk assessment table, and the IDRC for funding this project.

Notes

[1] Funding: International Development Research Centre in Canada.

[2] Author contributions: HN, MEH, MEK, and SS designed the study. LH conducted the search process and wrote the first draft of the methods section. HN, MEH, MEK screened articles. HN analysed the data and wrote the manuscript with input from MEK. MEK and SS provided critical comments and revisions and oversaw the project.

[3] Competing interests: The authors completed the ICMJE Unified Competing Interest form (available upon request from the corresponding author), and declare no conflicts of interest.

REFERENCES

[1] E Nicol, E Turawa, and G Bonsu. Pre- and in-service training of healthcare workers on immunization data management in LMICs: a scoping review. Hum Resour Health. 2019;17:92 DOI: 10.1186/s12960-019-0437-6. [PMID:31791352]

[2] . Measuring performance on the Healthcare Access and Quality Index for 195 countries and territories and selected subnational locations: a systematic analysis from the Global Burden of Disease Study 2016. Lancet. 2018;391:2236-71. DOI: 10.1016/S0140-6736(18)30994-2. [PMID:29893224]

[3] World Health Organization. The Kampala declaraTion and agenda for global action. Geneva: WHO; 2008.

[4] World Health Organization. WHO global code of practice on the international recruitment of health personnel. Geneva: WHO; 2010.

[5] JP Koplan, TC Bond, MH Merson, KS Reddy, MH Rodriguez, and NK Sewankambo. Towards a common definition of global health. Lancet. 2009;373:1993-5. DOI: 10.1016/S0140-6736(09)60332-9. [PMID:19493564]

[6] WHO. World Health Report 2006: Working Together for Health Geneva World Health Organization 2006.

[7] J Wang, AS Abdullah, Z Ma, H Fu, K Huang, and H Yu. Building capacity for information and communication technology use in global health research and training in China: A qualitative study among Chinese health sciences faculty members. Health Res Policy Syst. 2017;15:59 DOI: 10.1186/s12961-017-0222-8. [PMID:28659155]

[8] D Hoffman, B Blasi, B C’ulum, Z Drags’ic, A Ewen, and H Horta. The methodological illumination of a blind spot: information and communication technology and international research team dynamics in a higher education research program. High Educ. 2014;67:473-95. DOI: 10.1007/s10734-013-9692-y

[9] R Kislov, H Waterman, G Harvey, and R Boaden. Rethinking capacity building for knowledge mobilisation: developing multilevel capabilities in healthcare organisations. Implement Sci. 2014;9:166 DOI: 10.1186/s13012-014-0166-0. [PMID:25398428]

[10] Paul S. Capacity Building for health Sector Reform. Geneva: World Health Organization; 1995.

[11] D Beran, P Byass, A Gbakima, K Kahn, O Sankoh, and S Tollman. Research capacity building—obligations for global health partners. Lancet Glob Health. 2017;5:e567-8. DOI: 10.1016/S2214-109X(17)30180-8. [PMID:28495256]

[12] K DeCorby-Watson, G Mensah, K Bergeron, S Abdi, B Rempel, and H Manson. Effectiveness of capacity building interventions relevant to public health practice: a systematic review. BMC Public Health. 2018;18:684 DOI: 10.1186/s12889-018-5591-6. [PMID:29859075]

[13] A Liberati, DG Altman, J Tetzlaff, C Mulrow, PC Gotzsche, and JP Ioannidis. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62:e1-34. DOI: 10.1016/j.jclinepi.2009.06.006. [PMID:19631507]

[14] Education BEM. The literature search. 2020. Available: https://www.bemecollaboration.org/LiteratureSearch/. Accessed:22 December 2019.

[15] IOM. America's vital interest in global health: Protecting our people, enhancing our economy, and advancing out international interests. Washington (DC): Institute of Medicine - National Academy Press 1997.

[16] BJ Smith, C Tang, and D Nutbeam. WHO health promotion glossary: new terms Health Promot Int. 2006;21:340-5. DOI: 10.1093/heapro/dal033. [PMID:16963461]

[17] McNamara C. Basic Guide to Program Evaluation (including outcome evaluation). Available: https://managementhelp.org/evaluation/program-evaluation-guide.htm. Accessed: 18 June 2020.

[18] UBC Learning Modalities Availablehttps://ets.educ.ubc.ca/learning-design/learning-modalities/. Accessed: 18 June 2018.

[19] WHO. Community health workers: what do we know about them? The state of the evidence on programmes, activities, costs, and impact on health outcomes of using community health workers. Geneva World health organization 2007.

[20] R Asgary, PB Adongo, A Nwameme, HVS Cole, E Maya, and M Liu. MHealth to Train Community Health Nurses in Visual Inspection with Acetic Acid for Cervical Cancer Screening in Ghana. J Low Genit Tract Dis. 2016;20:239-42. DOI: 10.1097/LGT.0000000000000207. [PMID:27030884]

[21] MN Brantuo, E Cristofalo, MM Mehes, J Ameh, NO Brako, and F Boahene. Evidence-based training and mentorship combined with enhanced outcomes surveillance to address the leading causes of neonatal mortality at the district hospital level in Ghana. Trop Med Int Health. 2014;19:417-26. DOI: 10.1111/tmi.12270. [PMID:24495284]

[22] DL Comeau, K Mishkin, N Tukvadze, Z Avaliani, RR Kempker, and L Sthreshley. Mentoring in Global Health: Formative Evaluation of Tuberculosis Research Training Programs in Ethiopia and Georgia. Am J Trop Med Hyg. 2018;99:565-77. DOI: 10.4269/ajtmh.17-0746. [PMID:30014815]

[23] J Crocker, KF Shields, V Venkataramanan, D Saywell, and J Bartram. Building capacity for water, sanitation, and hygiene programming: Training evaluation theory applied to CLTS management training in Kenya. Soc Sci Med. 2016;166:66-76. DOI: 10.1016/j.socscimed.2016.08.008. [PMID:27543683]

[24] EP Davila, Z Suleiman, J Mghamba, I Rolle, I Ahluwalia, and P Mmbuji. Non-communicable disease training for public health workers in low- and middle-income countries: lessons learned from a pilot training in Tanzania. Int Health. 2015;7:339-47. DOI: 10.1093/inthealth/ihu090. [PMID:25526907]

[25] C Feldacker, S Chicumbe, M Dgedge, G Augusto, F Cesar, and M Robertson. Mid-Level Healthcare Personnel Training: An Evaluation of the Revised, Nationally-Standardized, Pre-Service Curriculum for Clinical Officers in Mozambique. PLoS One. 2014;9:e102588. DOI: 10.1371/journal.pone.0102588. [PMID:25068590]

[26] A Garley, E Eckert, A Sie, M Ye, K Malm, and EA Afari. Strengthening individual capacity in monitoring and evaluation of malaria control programmes to streamline M&E systems and enhance information use in malaria endemic countries. Malar J. 2016;15:300 DOI: 10.1186/s12936-016-1354-y. [PMID:27233243]

[27] J Huber, D Bauer, M Hoelscher, J Kapungu, A Kroidl, and T Lennemann. Evaluation of health research capacity strengthening trainings on individual level: validation of a questionnaire. J Eval Clin Pract. 2014;20:390-5. DOI: 10.1111/jep.12143. [PMID:24828323]

[28] V Makanjuola, V Doku, R Jenkins, and O Gureje. Monitoring and evaluation of the activities of trainees in the ‘training of trainers’ workshop at Ibadan, south-west Nigeria. Ment Health Fam Med. 2012;9:25-32. [PMID:23277795]

[29] JK Matovu, RK Wanyenze, S Mawemuko, O Okui, W Bazeyo, and D Serwadda. Strengthening health workforce capacity through work-based training. BMC Int Health Hum Rights. 2013;13:8 DOI: 10.1186/1472-698X-13-8. [PMID:23347473]

[30] R Mpofu, PS Daniels, TA Adonis, and WM Karuguti. Impact of an interprofessional education program on developing skilled graduates well-equipped to practise in rural and underserved areas. Rural Remote Health. 2014;14:2671 [PMID:25178157]

[31] JW Muchiri, GJ Gericke, and P Rheeder. Impact of nutrition education on diabetes knowledge and attitudes of adults with type 2 diabetes living in a resource-limited setting in South Africa: A randomised controlled trial. J of Endocr J Endocrinol Metab Diab South Afr. 2016;21:26-34. DOI: 10.1080/16089677.2016.1200324

[32] E Mutabaruka, C Dochez, D Nshimirimana, and A Meheus. Evaluation of mid-level management training in immunisation in the African region. East Afr J Public Health. 2010;7:37-43. [PMID:21413570]

[33] J O’Donovan, K Kabali, C Taylor, M Chukhina, JC Kading, and J Fuld. The use of low-cost Android tablets to train community health workers in Mukono, Uganda, in the recognition, treatment and prevention of pneumonia in children under five: a pilot randomised controlled trial. Hum Resour Health. 2018;16:49 DOI: 10.1186/s12960-018-0315-7. [PMID:30231894]

[34] A Plowright, C Taylor, D Davies, J Sartori, GL Hundt, and RJ Lilford. Formative evaluation of a training intervention for community health workers in South Africa: A before and after study. PLoS One. 2018;13:e0202817. DOI: 10.1371/journal.pone.0202817. [PMID:30248100]

[35] L Sarli, E Enongene, K Bulgarelli, A Sarli, A Renda, and G Sansebastiano. Training program for community health workers in remote areas in Senegal. First experience. Acta Biomed. 2010;81:54-62. [PMID:20857853]

[36] Y Shimpuku, FE Madeni, S Horiuchi, K Kubota, and SC Leshabari. Evaluation of a family-oriented antenatal group educational program in rural Tanzania: a pre-test/post-test study. Reprod Health. 2018;15:117 DOI: 10.1186/s12978-018-0562-z. [PMID:29954398]

[37] G Sibeko, PD Milligan, M Roelofse, L Molefe, D Jonker, and J Ipser. Piloting a mental health training programme for community health workers in South Africa: An exploration of changes in knowledge, confidence and attitudes. BMC Psychiatry. 2018;18:191 DOI: 10.1186/s12888-018-1772-1. [PMID:29898705]

[38] B Tekola, F Girma, M Kinfe, R Abdurahman, M Tesfaye, and Z Yenus. Adapting and pre-testing the World Health Organization’s Caregiver Skills Training programme for autism and other developmental disorders in a very low-resource setting: Findings from Ethiopia. Autism. 2020;24:51-63. DOI: 10.1177/1362361319848532. [PMID:31094208]

[39] J Wills and M Rudolph. Health promotion capacity building in South Africa. Glob Health Promot. 2010;17:29-34. DOI: 10.1177/1757975910375167. [PMID:21495439]

[40] S Witek-McManus, DP Mathanga, A Verney, A Mtali, D Ali, and J Sande. Design, implementation and evaluation of a training programme for school teachers in the use of malaria rapid diagnostic tests as part of a basic first aid kit in southern Malawi. BMC Public Health. 2015;15:904 DOI: 10.1186/s12889-015-2228-x. [PMID:26377070]

[41] M Fish, J Parkes, N Dharsee, S Dryden-Peterson, J Efstathiou, and L Schnipper. POETIC (Program for Enhanced Training in Cancer): An Initial Experience of Supporting Capacity Building for Oncology Training in Sub-Saharan Africa. Oncologist. 2019;24:1557-61. DOI: 10.1634/theoncologist.2019-0207. [PMID:31182656]

[42] M Abdel-All, AG Thrift, M Riddell, KRT Thankappan, GK Mini, and CK Chow. Evaluation of a training program of hypertension for accredited social health activists (ASHA) in rural India. BMC Health Serv Res. 2018;18:320 DOI: 10.1186/s12913-018-3140-8. [PMID:29720161]

[43] C Carlos, R Capistrano, CF Tobora, MR Delos Reyes, S Lupisan, and A Corpuz. Hospital preparedness for Ebola virus disease: a training course in the Philippines. Western Pac Surveill Response J. 2015;6:33-43. DOI: 10.5365/wpsar.2014.5.4.008. [PMID:25960920]

[44] H Choi, S Kim, H Ko, Y Kim, and CG Park. Development and preliminary evaluation of culturally specific web-based intervention for parents of adolescents. J Psychiatr Ment Health Nurs. 2016;23:489-501. DOI: 10.1111/jpm.12327. [PMID:27500792]

[45] P Gao, H Xiang, S Liu, Y Liu, S Dong, and F Liu. Evaluation of a consulting training course for international development assistance for health. BMC Med Educ. 2018;18:234 DOI: 10.1186/s12909-018-1339-3. [PMID:30305087]

[46] B Gyawali, SR Mishra, D Neupane, A Vaidya, A Sandbæk, and P Kallestrup. Diabetes management training for female community health volunteers in Western Nepal: an implementation experience. BMC Public Health. 2018;18:641 DOI: 10.1186/s12889-018-5562-y. [PMID:29783961]

[47] JM Kim, WK Kwang, HO Chul, HJ Woo, HK Sung, and HP Dae. Assessment of village health worker training program in Tuguegarao, Philippine. J Prev Med Public Health. 2009;42:377-85. DOI: 10.3961/jpmph.2009.42.6.377. [PMID:20009484]

[48] J Kang, EB Yang, YJ Chang, JY Choi, HJ Jho, and SJ Koh. Evaluation of the National Train-the-Trainer Program for Hospice and Palliative Care in Korea. Asian Pac J Cancer Prev. 2015;16:501-6. DOI: 10.7314/APJCP.2015.16.2.501. [PMID:25684478]

[49] YS Kusuma, D Burman, R Kumari, AS Lamkang, and BV Babu. Impact of health education based intervention on community's awareness of dengue and its prevention in Delhi, India. Glob Pealth Promt. 2019;26:50-9. DOI: 10.1177/1757975916686912. [PMID:28349734]

[50] R Limato, R Ahmed, A Magdalena, S Nasir, and F Kotvojs. Use of most significant change (MSC) technique to evaluate health promotion training of maternal community health workers in Cianjur district, Indonesia. Eval Program Plann. 2018;66:102-10. DOI: 10.1016/j.evalprogplan.2017.10.011. [PMID:29091785]

[51] EMM Monoto and NKM Alwi. Breastfeeding Videos as a New Training Tool for Malaysian Breastfeeding Peer Counsellor Program in Empowering Breastfeeding Communities. Int Med J Malays. 2018;17:59-63.

[52] P Negandhi, H Negandhi, K Sharma, S Wild, and S Zodpey. Enhancing public health practice through a capacity-building educational programme: an evaluation. Hum Resour Health. 2015;13:31 DOI: 10.1186/s12960-015-0024-4. [PMID:25967579]

[53] A Nugroho, RWS Coulter, V Erasmus, PA Laksmono, TS Mihari, and JH Richardus. Evaluation of a training aimed at building capacity for outreaching to men who have sex with men and transgender women in Indonesia. Health Educ Res. 2019;34:223-33. DOI: 10.1093/her/cyy046. [PMID:30508187]

[54] Z Shen, C Jiang, and L Chen. Evaluation of a train-the-trainer program for stable coronary artery disease management in community settings: A pilot study. Patient Educ Couns. 2018;101:256-65. DOI: 10.1016/j.pec.2017.07.025. [PMID:28774653]

[55] DP Slakey, R Reily, I Davidson, and JR Korndorffer. Evaluating a surgeon led training program: Targeting kidney disease in Vietnam. Int J Surg Open. 2016;4:18-22. DOI: 10.1016/j.ijso.2016.06.001

[56] K Sranacharoenpong, RM Hanning, PP Sirichakwal, and U Chittchang. Process and outcome evaluation of a diabetes prevention education program for community healthcare workers in Thailand. Educ Health (Abingdon). 2009;22:335 [PMID:20029769]

[57] Y Wang, LD Xiao, S Ullah, GP He, and A De Bellis. Evaluation of a nurse-led dementia education and knowledge translation programme in primary care: A cluster randomized controlled trial. Nurse Educ Today. 2017;49:1-7. DOI: 10.1016/j.nedt.2016.10.016. [PMID:27855297]

[58] XF Xu, Y Wang, YY Wang, M Song, WG Xiao, and Y Bai. Role-playing is an effective instructional strategy for genetic counseling training: an investigation and comparative study. BMC Med Educ. 2016;16:235 DOI: 10.1186/s12909-016-0756-4. [PMID:27589999]

[59] BX Yang, TE Stone, and SA Davis. The effect of a community mental health training program for multidisciplinary staff. Arch Psychiatr Nurs. 2018;32:413-7. DOI: 10.1016/j.apnu.2017.12.007. [PMID:29784223]

[60] X Zhan, Z Zhang, F Sun, Q Liu, W Peng, and H Zhang. Effects of Improving Primary Health Care Workers’ Knowledge About Public Health Services in Rural China: A Comparative Study of Blended Learning and Pure E-Learning. J Med Internet Res. 2017;19:e116. DOI: 10.2196/jmir.6453. [PMID:28461286]

[61] CM Zhang, D Xiao, R West, S Michie, R Troughton, and P Hajek. Evaluation of 3-day smoking cessation training course for doctors from 38 cities in China. Chin Med J (Engl). 2012;125:1338-40. [PMID:22613611]

[62] Y Zhao, Y Hu, J Liang, and X Qian. A pilot study on the simulation-based training for Ethiopia skilled birth attendants. Nurse Educ Pract. 2019;34:130-8. DOI: 10.1016/j.nepr.2018.12.001. [PMID:30529835]

[63] BA Ziganshin, LM Yausheva, M Sadigh, AP Ziganshina, AA Pichugin, and AU Ziganshin. Training Young Russian Physicians in Uganda: A Unique Program for Introducing Global Health Education in Russia. Ann Glob Health. 2015;81:627-35. DOI: 10.1016/j.aogh.2015.10.007. [PMID:27036719]

[64] R Cianelli, C Wilkinson, E Mitchell, D Anglade, G Nicolas, and V Mitrani. Mental health training experiences among Haitian healthcare workers post-earthquake 2010. Int Nurs Rev. 2013;60:528-35. DOI: 10.1111/inr.12047. [PMID:24251943]

[65] BA Knettel, SE Slifko, AG Inman, and I Silova. Training community health workers: an evaluation of effectiveness, sustainable continuity, and cultural humility in an educational program in rural Haiti. Int J Health Promot Educ. 2017;55:177-88. DOI: 10.1080/14635240.2017.1284014

[66] YM Rivera, L Moreno, KJ Briant, H Velez, JC Jimenez, and J Torres. Developing Sustainable Cancer Education Programs: Training Public Health Students to Deliver Cancer 101 in Puerto Rico. J Cancer Educ. 2018;33:128-33. DOI: 10.1007/s13187-016-1076-4. [PMID:27424481]

[67] ML Gonzalez, M Melgar, M Homsi, A Shuler, F Antillon-Klussmann, and L Matheu. Measuring readiness for and satisfaction with a hand hygiene e-learning course among healthcare workers in a paediatric oncology centre in Guatemala City. Int J Infect Control. 2016;12:1-12. [PMID:29147140]

[68] KA McConnell, LK Krisher, M Lenssen, M Bunik, S Bunge Montes, and GJ Domek. Telehealth to Expand Community Health Nurse Education in Rural Guatemala: A Pilot Feasibility and Acceptability Evaluation. Front Public Health. 2017;5:60 DOI: 10.3389/fpubh.2017.00060. [PMID:28405582]

[69] J Molokwu, E Penaranda, S Flores, and NK Shokar. Evaluation of the Effect of a Promotora-led Educational Intervention on Cervical Cancer and Human Papillomavirus Knowledge Among Predominantly Hispanic Primary Care Patients on the US-Mexico Border. J Cancer Educ. 2016;31:742-8. DOI: 10.1007/s13187-015-0938-5. [PMID:26507743]

[70] R Abdelhai, S Yassin, MF Ahmad, and UG Fors. An e-learning reproductive health module to support improved student learning and interaction: a prospective interventional study at a medical school in Egypt. BMC Med Educ. 2012;12:11 DOI: 10.1186/1472-6920-12-11. [PMID:22433670]

[71] K Gholipour, JS Tabrizi, M Farahbakhsh, S Iezadi, A Ghiasi, and H Jahanbin. Evaluation of the district health management fellowship training programme: a case study in Iran. BMJ Open. 2018;8:e020603. DOI: 10.1136/bmjopen-2017-020603. [PMID:29525773]

[72] M Omar, N Gerein, E Tarin, C Butcher, S Pearson, and G Heidari. Training evaluation: a case study of training Iranian health managers. Hum Resour Health. 2009;7:20 DOI: 10.1186/1478-4491-7-20. [PMID:19265528]

[73] S Siabani, T Driscoll, PM Davidson, and SR Leeder. Efficacy of a home-based educational strategy involving community health volunteers in improving self-care in patients with chronic heart failure in western Iran: A randomized controlled trial. Eur J Cardiovasc Nurs. 2016;15:363-71. DOI: 10.1177/1474515115585651. [PMID:25944829]

[74] P Fung and R Montague. A qualitative evaluation of leadership development workshops for mental health workers from four Pacific Island Countries. Australas Psychiatry. 2015;23:218-21. DOI: 10.1177/1039856215576417. [PMID:25783669]

[75] F Charlson, O Chang, I Kubuabola, J Schess, C Latu, and E Hunter. Implementation of the mental health Gap Action Programme (mhGAP) within the Fijian Healthcare System: a mixed-methods evaluation. Int J Ment Health Syst. 2019;13:43 DOI: 10.1186/s13033-019-0301-z. [PMID:31249611]

[76] M Khan, P Lamelas, H Musa, J Paty, T McCready, and R Nieuwlaat. Development, Testing, and Implementation of a Training Curriculum for Nonphysician Health Workers to Reduce Cardiovascular Disease. Glob Heart. 2018;13:93-100.e1. DOI: 10.1016/j.gheart.2017.11.002. [PMID:29331282]

[77] LMX Gomes, TLA Barbosa, EDS Vieira, LJT Vieira, KPAN Castro, and IA Pereira. Community healthcare workers’ perception of an educational intervention in the care of patients with sickle cell disease in Brazil. Mediterr J Hematol Infect Dis. 2015;7:e2015031. [PMID:25960859]

[78] LA Hull, S Arora, AC Amaya, A Wheelock, H Gaitán-Duarte, and C Vincent. Building global capacity for patient safety: a training program for surgical safety research in developing and transitional countries. Int J Surg. 2012;10:493-9. DOI: 10.1016/j.ijsu.2012.07.005. [PMID:22846618]

[79] A Joshi, MA Novaes, S Iyengar, JL Machiavelli, J Zhang, and R Vogler. Evaluation of a tele-education programme in Brazil. J Telemed Telecare. 2011;17:341-5. DOI: 10.1258/jtt.2011.101209. [PMID:21933894]

[80] EB Monier, DV Arajo, AEF Oliveira, DCL Baesse, JRO Pinho, and GVS Brasil. Student Evaluation of Distance Learning for Healthcare Professionals. Telemed J E Health. 2019;25:485-91. DOI: 10.1089/tmj.2018.0065. [PMID:30020855]

[81] CA Pereira, CL Wen, and H Tavares. Alcohol abuse management in primary care: an e-learning course. Telemed J E Health. 2015;21:200-6. DOI: 10.1089/tmj.2014.0042. [PMID:25599269]

[82] MA Vieira, AA Gadelha, TS Moriyama, RA Bressan, and IA Bordin. Evaluating the effectiveness of a training program that builds teachers’ capability to identify and appropriately refer middle and high school students with mental health problems in Brazil: an exploratory study. BMC Public Health. 2014;14:210 DOI: 10.1186/1471-2458-14-210. [PMID:24580750]

[83] Pluye P, Robert E, Cargo M, Bartlett G, O’Cathain A, Griffiths F, et al. Proposal: A mixed methods appraisal tool for systematic mixed studies reviews. Montreal: 2011.

[84] J Huber, S Nepal, D Bauer, I Wessels, MR Fischer, and C Kiessling. Tools and instruments for needs assessment, monitoring and evaluation of health research capacity development activities at the individual and organizational level: a systematic review. Health Res Policy Syst. 2015;13:80 DOI: 10.1186/s12961-015-0070-3. [PMID:26691766]

[85] J Wang, Z Ma, H Fu, K Huang, H Yu, and J Wang. Building capacity for information and communication technology use in global health research and training in China: A qualitative study among Chinese health sciences faculty members. Health Res Policy Syst. 2017;15:59 DOI: 10.1186/s12961-017-0222-8. [PMID:28659155]

[86] WS Bolton, N Aruparayil, A Quyn, J Scott, A Wood, and I Bundu. Disseminating technology in global surgery. Br J Surg. 2019;106:e34-43. DOI: 10.1002/bjs.11036. [PMID:30620068]

[87] P Howitt, A Darzi, G-Z Yang, H Ashrafian, R Atun, and J Barlow. Technologies for global health. Lancet. 2012;380:507-35. DOI: 10.1016/S0140-6736(12)61127-1. [PMID:22857974]

[88] KAK McConnell. L. K.: Lenssen, M.: Bunik, M.: Bunge Montes, S.: Domek, G. J. Telehealth to Expand Community Health Nurse Education in Rural Guatemala: A Pilot Feasibility and Acceptability Evaluation. Public Health Front. 2017;5:1-8. DOI: 10.3389/fpubh.2017.00060

[89] S Van den Broucke. Strengthening public health capacity through a health promotion lens. Health Promot Int. 2017;32:763-7. DOI: 10.1093/heapro/dax064. [PMID:28992188]

[90] J Huber, D Bauer, M Hoelscher, J Kapungu, A Kroidl, and T Lennemann. Evaluation of health research capacity strengthening trainings on individual level: Validation of a questionnaire. J Eval Clin Pract. 2014;20:390-5. DOI: 10.1111/jep.12143. [PMID:24828323]

[91] JM Pentimonti, TA Zucker, LM Justice, Y Petscher, SB Piasta, and JN Kaderavek. A standardized tool for assessing the quality of classroom-based shared reading: Systematic Assessment of Book Reading (SABR). Early Child Res Q. 2012;27:512-28. DOI: 10.1016/j.ecresq.2011.12.007

Background

Methods

Results

Conclusion