What is CHNRI?

The CHNRI process is a consultative approach that entails identifying, compiling, and listing competing research questions, and then evaluating these research questions via a standardized set of criteria. The process engages experts from programmatic, research, donor, government, policy, and implementation backgrounds to participate in a six step process (see Box 1). CHNRI was originally developed in 2006 based on the recognized need to strengthen the global health evidence base, and on the need for a systematic approach to setting and making investment decisions on global child health and nutrition research priorities [4-6]. CHNRI has since become the most commonly used methodology for health research priority setting. The approach has been used for more than 50 health applications[7-9], including those related to integrated community case management (iCCM) of childhood illness [10], emerging interventions against childhood diarrhea [11], child protection in humanitarian settings [12], and pediatric and adolescent human immunodeficiency virus (HIV) [13]. There are many advantages to adopting this method to generate research priorities, including engaging a diverse group of relevant stakeholders; relying on a well-defined and systematic yet flexible process; and using a democratic, transparent approach to establishing a research agenda.

Defining the private health sector

The private health sector is broad, and includes entities such as private health care providers, private insurance companies, and private actors across the health supply chain such as distributors and wholesalers of medical products. For the purposes of this CHNRI process, the private health sector is defined broadly to include for-profit providers, non-governmental organizations, social enterprises, marketing and franchising organizations, and faith-based organizations that deliver preventive and curative health services for children, particularly those under five. This includes both clinical and non-clinical private providers. Clinical providers are defined in this exercise to include those working in a clinic or other health facility setting. In contrast, non-clinical providers are those who work outside of a formal health facility setting, even if they have received some level of medical training. Examples of both clinical and non-clinical health providers are provided below.

Scope of the private sector CHNRI

This CHNRI focused on private health sector strategies related to the management of sick child care, with an emphasis on children under age 5. Recognizing that private health sector service delivery models may not fully align with integrated management of childhood illness (IMCI) protocols, the scope of this private sector CHNRI includes but also extends beyond the World Health Organization (WHO)-defined case management protocol for IMCI that is typically used in the public sector. We therefore encouraged participants in this CHNRI to interpret IMCI broadly, and to consider how IMCI or other integrated case management approaches could be adapted to private health sector models.

This article summarizes the results of our private health sector CHNRI and presents a prioritized research agenda for closing evidence gaps related to the management of sick child care through the private health sector. We expect that advancing this research will enhance the quality, efficiency, and sustainability of child health care services delivered by the private health sector, which will in turn contribute to meaningful global reductions in child morbidity and mortality.

1. Identify and invite experts to participate in the process

We proactively sought to engage a diverse group of individuals with expertise in the private health sector and child health to generate research questions. Using purposive and snowball sampling, these experts were drawn from the core research team’s global professional networks, referrals from individuals in these networks, members of the Private Sector Engagement and Implementation Science subgroups of the Child Health Task Force, and participants in the 2018 Global Symposium on Health Systems Research.

In November 2018, we invited 129 technical experts to participate in the CHNRI exercise via email invitation. Nearly 90 global and country experts, representing academia, implementing partners, donors, bilateral organizations, and national and subnational Ministry of Health representatives, agreed to participate. Figure 1 and Figure 2 below illustrate the distribution of all CHNRI participants by geographic region and type of institutional affiliation, respectively.

Figure 1.  Distribution of CHNRI participants by geographic region. CHNRI − Child Health and Nutrition Research Initiative

Figure 2.  Distribution of CHNRI Participants. by type of institutional affiliation. CHNRI − Child Health and Nutrition Research Initiative.

2. Determine criteria against which participants will evaluate all questions

Next, the core research team determined the evaluation criteria that would be used to evaluate all proposed research questions submitted by respondents from among this group of global and country experts. We established four evaluation categories for proposed research questions: (i) answerability, (ii) research feasibility, (iii) sustainability/ equity, and (iv) importance/potential impact.

These evaluation criteria were adapted from those used during a previous CHNRI exercise conducted to establish a prioritized research agenda for iCCM [10]. Each of the four evaluation categories included multiple criteria, for a total of eleven evaluation criteria across the four evaluation categories (Table 1). The CHNRI expert group received these evaluation criteria and categories prior to being asked to submit their proposed research questions.

3. Ask experts to submit their ideas for priority research questions

We asked the 89 experts who agreed to participate in this CHNRI exercise in Step 1 to generate their own priority research questions. After receiving the evaluation criteria for proposed research questions that were established in Step 2, experts had one month to submit their ideas for priority research questions (from December 2018-January 2019). Over the course of this month, the core research team sent this expert group three email reminders to respond. Thirty-eight experts (43 percent) responded to this request, generating a total of nearly 150 ideas for priority research questions.

4. Consolidate and refine research questions to reduce duplication

Many of the research questions we received reflected similarities. This suggested some alignment of research priorities across expert respondents, and that the submitted questions had reached theoretical saturation. Under Step 4, we then consolidated and organized all submitted research question ideas, without altering the content of these ideas, to reduce redundancies and facilitate scoring and analysis. This consolidation exercise yielded a final list of 50 candidate research questions.

5. Send prioritization survey to experts, asking them to evaluate submitted research questions

In April 2019, we again contacted the 89 experts by email and asked them to evaluate the abovementioned list of 50 candidate research questions against the 11 evaluation criteria established under Step 2 of this CHNRI exercise (Table 1). Respondents were asked to share their feedback anonymously through an approximately 45-minute, online survey created using SurveyGizmo. In line with previous CHNRI applications [10,13], survey questions were listed in random order to reduce the possibility that question order would affect respondent scoring, and to mitigate differential respondent fatigue across questions. Respondents were asked to score each candidate research question using the CHNRI evaluation criteria they had received earlier, and according to a Likert scale ranging from 1 to 5 (with 1 = “No, strongly disagree,” and 5 = “Yes, strongly agree”). Respondents could also indicate if they were unable to evaluate a research question against a certain evaluation criterion by selecting the choice, “Do not know”. Respondents were given one month to complete the survey and received three email reminders from the core research team during this period.

6. Analyze results

We exported all survey responses from SurveyGizmo into an Excel spreadsheet. All 11 CHNRI evaluation criteria were determined to be of equal importance and were hence weighted equally, such that the sustainability/equity and importance/potential impact categories, of which included four evaluation criteria, had four times the weight of the research feasibility category, which included only one evaluation criterion, for example. The Research Priority Score (RPS), or average score across scorers and criteria, indicates the “collective optimism” among the scorers that a research question satisfies all 11 evaluation criteria [11]. The RPS was calculated separately for each proposed research question, summing the results of each of the 11 evaluation criteria questions, q, such that:

The numeric score for each criteria varied from 1 (strongly agree) to 5 (strongly disagree). Respondents who did not provide an answer for a particular evaluation criterion or who answered “do not know” were excluded from both the numerator and denominator.

We calculated the RPS separately for each proposed research question. Please see Figure S1 in the Online Supplementary Document for a more detailed version of the RPS calculation.

The evaluation criterion score for each of the eleven evaluation criteria was also calculated for each research question. This score is very similar to the RPS, but rather than an overall research question rating, indicates the collective optimism among the scorers that a research question satisfies a particular evaluation criterion. The evaluation criterion score was calculated for each research question and criterion as:

As noted above, respondents who did not provide an answer for a particular evaluation criterion or who answered “do not know” were excluded. Please see Figure S2 in the Online Supplementary Document for additional detail.

The Average Expert Agreement (AEA), the degree to which scorers were in consensus regarding the scores they provided, was also calculated. More specifically, the AEA describes the proportion of scorers who provided the mode, or most frequent Likert Scale score. For example, an AEA of 50 means that, on average, half of scorers agreed with one another and provided the same scores across all 11 evaluation criteria.

The AEA for each research question was calculated as:

where q was the particular evaluation criterion that a research question was evaluated against.

Internet Protocol (IP) addresses at the time the respondent completed the online evaluation survey were used as a proxy measure for stratifying results and scores according to high-income country (HIC) and LMIC respondents and responses. Country classifications by income level were based on the 2019-2020 World Bank country classifications [14]. We calculated evaluation criteria scores, RPS, and AEAs according to these HIC and LMIC categories.

A Spearman’s rank correlation test was used to determine the correlation of the ranking of research questions between HICs and LMICs, as research priority ranking is an ordinal and non-parametric outcome. A Spearman’s Rho (ρ) or correlation ranges from -1 to 1, with 1 indicating a high, positive correlation between two groups and a -1 indicating a high negative correlation between two groups. A coefficient of 0 indicates no correlation.

Statistical data analysis was conducted in SAS 9.4 (IBM Inc, Armonk, NY, USA). We conducted two versions of this analysis: one version with completed surveys only and one version with completed surveys and four partially completed surveys that were at least 30% complete. The results of both types of analyses were very similar, so we chose to present only the analysis with complete surveys.

Ethics statement

The Boston University Medical Campus Institutional Review Board reviewed this study and determined that it was not human subjects research [15].

Stratified analysis by HIC and LMIC respondents

The evaluation results were very similar across respondents in HICs and LMICs. There was a strong and statistically significant correlation between the scores for HIC and LMIC respondents (Spearman’s P = 0.71, P < 0.0001), indicating that respondents from these varying locations largely prioritized similar research questions related to child health management in the private sector. Table 3 and Table 4, respectively, show the top 10 research priorities as evaluated by HIC and LMIC participants. Questions on which both high-income and low-and middle-income country respondents agreed were among the top ten research priorities are highlighted in bold. For example, experts located in HICs and LMICs agreed that the following research questions were of utmost importance: what are the key drivers of appropriate and inappropriate antimalarial and antibiotic prescription for children in private-for-profit sources of care by type of provider (#6 among HIC respondents and #8 among LMIC respondents); can supportive supervision lead to improved quality of care in the private sector (#5 among HIC respondents and #1 among LMIC respondents); and what is the effectiveness of training private medicine vendors to recognize, manage, and/or refer sick young infants (#3 among HIC respondents and #10 among LMIC respondents). The complete list of research priorities as evaluated by HIC and LMIC participants is presented in Tables S2and S3 in the Online Supplementary Document, respectively.

Strengths and limitations

The CHNRI process can be a powerful tool to identify new research questions and to use input from a wide variety of stakeholders to define which questions should be prioritized. We used a similar approach to prior CHNRI exercises with some new twists. These included implementation through a web-based survey, randomization of research questions, an equal weighting system per question rather than category, and stratified analysis by participant location. In contrast to earlier CHNRI research question scoring activities, which were done in highly detailed Excel spreadsheets, we used a more user-friendly web-based approach. This strategy should be considered for future CHNRI research question priority activities. Randomization of research questions may help by reducing ‘question fatigue’ which can arise during the prolonged process of evaluating questions. Given the importance of each question, it seems more logical to give equal weight to the question rather than the overall category in which the question lies. Finally, stratifying the responses by participant location provides some insight into geographic differences, although this may be somewhat artificial since some respondents may be based in a specific location but actually represent a completely different part of the world.

While participation in this exercise included experts both from LMICs and HICs, we did face challenges with non-responsiveness to the CHNRI survey. The overall response rate for completed surveys was 55% and, as noted above, we included only fully completed surveys in our analysis. While lower than hoped, this response rate was similar to the response rate reported in the 2014 iCCM CHNRI [10]. Further, our sample size of 49 complete surveys aligns with previous CHNRI research demonstrating that collective opinion among experts stabilizes with a sample size of 45 to 55 experts [25]. There were also 62 partially completed surveys, many with just a question or two answered, and four of which were at least 30 percent complete. The high number of incomplete surveys may have been due in part to the ability of respondents to save, exit, and re-enter the online form multiple times, a design feature intended to encourage respondents to complete the entire survey if they could not do so in one sitting. Time to complete the survey may have also been a factor, as some respondents noted taking 1.5 to two hours to complete the online survey. Noting that different CHNRI exercises often adapt the number of evaluation criteria or even introduce new criteria to suit the needs of each exercise [7], future CHNRI exercises could include fewer evaluation criteria to reduce survey completion time and potential respondent fatigue However, doing so should be considered against the risk of reducing evaluation rigor.

Although we aimed to ‘crowd-source’ as many diverse experts with specialized knowledge in child health and private sector approaches, our targeting efforts did not reach all professionals working in this technical area. Future CHNRI endeavors could use a more extensive snowball sampling approach in which all CHNRI participants − not just the core research team, in the case of this CHNRI exercise − is asked to identify several additional participants. This kind of broader snowball sampling approach could further enhance the inclusiveness of the exercise, particularly among experts in LMICs. We also assumed that respondents were from a LMIC or HIC based on their IP address at the time of survey completion. While this was likely a reasonable assumption, it may not have been true for all experts – some of whom may have just been traveling to other regions at the time. To address this limitation, future CHNRI surveys could query respondents on their place of residence as a more direct (vs proxy) measure of their geographic designation as LMIC or HIC.

Several research questions may have been interpreted by survey respondents as overlapping in scope, which may have contributed to respondent fatigue, acquiescence, or confirmation bias. Additionally, as with any CHNRI exercise, there may have been other potential research investment options or ‘good ideas’ that were not proposed by our expert group and hence not included in the final list of research questions from which to choose (eg,no research questions were proposed related to private sector management of childhood malnutrition). Using the broader snowball sampling approach mentioned above to solicit ideas from a larger, more diverse group of participants may help mitigate this limitation for future CHNRI exercises. Finally, another limitation inherent to the CHNRI approach is that we do not have knowledge on how participants arrived at their submissions for priority research questions. The method relies on the expertise of the participants and does not ask participants to explain how or why they have selected their particular research questions. Future CHNRI iterations could ask participants to briefly explain the process they used to formulate their proposed research questions.