The CHNRI method for setting health research priorities

The Child Health and Nutrition Research Initiative (CHNRI) started as an initiative of the Global Forum for Health Research in Geneva, Switzerland [4]. One of its main aims was to develop a tool that could assist decision–making and priority setting in health research investments to improve child health and nutrition. Their method also sought to achieve an acceptable balance between fundamental research, translational research and implementation research in order to maximize the potential of health research in reducing both disease burden and the inequities among the world's children [5].

The CHNRI method was developed between 2005 and 2007 through 12 consecutive meetings of a trans–disciplinary panel of 15 experts, supported with funding from the World Bank. The experts worked together to address a number of key challenges related to the multi–dimensional problem of setting priorities in health research investments [5–7]. The method aimed to carefully define the context for health research priority setting. The components of the context were: (i) the health issue on which the research is focused; (ii) the affected population that would benefit from the investments in health research; (iii) the timeframe within which the impact of supported research was expected (eg, short, medium or long term); (iv) the style of investment (eg, risk aversive or risk–seeking); and (v) the expected returns from investment (eg, burden reduction, patents, or various forms of public recognition) [6–8].

The method also introduced a systematic approach to listing many competing research questions. It identified four fundamental instruments of health research – “the four D’s” – research to achieve (i) description (through epidemiological research), (ii) discovery (through basic, ie, fundamental research), (iii) development (through translational research) and (iv) delivery (through health policy and systems research, which includes operations and implementation research). Moreover, it addressed the difference in depth and breadth of suggested research questions by categorizing them in broad research avenues, more focused research options (which correspond to a 5–year research program), and very specific research ideas/questions (which correspond to a typical research article). Finally, the method introduced a transparent set of criteria that could discriminate between many competing research options. CHNRI’s “standard” set of criteria followed a simple conceptual framework that demonstrated how the process of health research generates new knowledge. The five suggested criteria were (i) answerability, (ii) effectiveness, (iii) deliverability, (iv) the potential for a substantial reduction of disease burden and (v) the impact on equity [6–8].

The typical CHNRI process involves a small management team that reaches out to a large number of researchers (but also policy–makers and program managers, depending on focus of the exercise) who contribute hundreds of research ideas [9,10]. Once a list of a manageable number of research ideas/questions (usually up to 200) is consolidated by removing overlapping ideas and integrating related ideas, a number of researchers (from 20 to up to several hundreds, depending on the context) are invited to score all proposed research questions against each priority–setting criterion [7,10]. Their input measures “collective optimism” on a scale 0–100. In the final step, external stakeholders are invited to set different thresholds and weights for each of the priority–setting criteria, giving some criteria greater importance over the others, so that the overall score also includes the value system of a wider community [2]. The final output of the CHNRI process is a list that ranks up to 200 research ideas/questions by their scores against several transparent priority–setting criteria [7]. This serves to reveal strengths and weaknesses of all submitted research questions to the research community, judged by a subset of this community using several key criteria for prioritization [8].

The examples of implementation

We conducted a review of the literature to identify the first 50 examples of the application of the CHNRI method in chronological order, to study the evolution of the uptake of the method. There are presently more than 50 examples of application, with further CHNRI exercises being conducted or planned, but not all of them have reached their final stage of peer–reviewed publication. Therefore, to acknowledge a milestone in method's implementation, we decided to focus on the first 50 publications that have been reviewed and published. We searched Google Scholar, PubMed and so–called “grey literature” (usually defined as papers produced by organizations outside of the traditional publishing and distribution channels) using the search term “CHNRI” or “Child Health and Nutrition Research Initiative”. The first 50 CHNRI priority–setting exercises, published between 2007 and 2016 (the full list with details of each study is available in Table S1 in Online Supplementary Document(Online Supplementary Document) ), reached out to nearly 5000 researchers, policy makers and program officers, seeking their participation in the generation of research ideas/questions and the scoring of those questions according to the proposed criteria. The initial response rate across all exercises was above 60%, with more than 3000 experts submitting research ideas. They submitted about 10 000 ideas (more than 3 per expert). The redundancy rate in submitted questions was slightly above 50%, indicating a relatively high rate of duplicate ideas. Eventually, 4282 ideas were scored (an average of 86 per exercise) by 2403 participating scorers (an average of 48 per exercise). Most of the papers were published in journals including PLoS Medicine (20%), BMC Public Health (14%) and Lancet (12%). Among the six exercises published in the The Lancet journal, three were published as stand–alone exercises and three were a part of policy recommendation papers or “calls for action” within the Lancet series (see Table S1 in Online Supplementary Document(Online Supplementary Document) ).

Clarity over the context of prioritization and the criteria used for prioritization is one of the key conceptual advances of the CHNRI method. Given the history of the development of the CHNRI method and its initial focus on the reduction of child mortality, it is not surprising that the majority of the exercises have addressed child mortality (either all–cause or specific causes) (52%) ( Table 1 ). The use of the CHNRI method was then extended to questions related to childhood morbidity and improved development (4%). In a logical progression of the method's application to address the key global health issues, it was applied to questions of maternal, perinatal and sexual health (8%), followed by several major infectious diseases, such as tuberculosis and zoonoses (6%). Then, the method started to find its application in areas outside of its initial focus – such as mental health (16%), all–cause mortality, morbidity and disability in adults (8%) and dementia (2%). Most exercises were focused on low– and middle–income countries (50%). Further 32% of CHNRI exercises were global in scope, but there were also 14% of exercises conducted at the national level, and 2% at a sub–national level ( Table 1 ). This shows that application of the CHNRI method is beginning to expand to health issues beyond the initial focus on child health, and to national and sub–national levels, where there is also a lot of need for prioritization of health research. This is further reflected in 56% of exercises being focused on children (including newborns), 16% on adolescent and young adults, and 28% on adults or all age groups ( Table 1 ).

In terms of the adopted time frame until the expected impact of research, the large majority of the exercises (74%) used a “standard” time frame of 10 years, originally suggested in the guidelines for implementation of the CHNRI method. A sizable minority of the exercises deviated from the recommended timeframe to suit the contexts to which the exercises were conducted; 20% of the exercises had shorter timeframes, while 6% had longer time frames ( Table 1 ). The evolution of the originally proposed CHNRI method through its implementations is particularly apparent when the criteria used for prioritization are analyzed across the 50 exercises. The originally proposed 5 criteria were used only in one–third of the exercises, while they were modified in two–thirds. Modification included changes in the number of criteria used, and the changes in the criteria themselves. Although 56% of all exercises used 5 criteria, as originally suggested, 12% reduced their number to only four or three, while 32% expanded the number of criteria applied – up to 13 in one exercise. Interestingly, although the five “standard” criteria were used most frequently, as expected (from 86% for equity to 70% for effectiveness), it is clear that the groups conducting the CHNRI processes felt a need to replace them and/or introduce further criteria in their exercises, or even reduce their number. The most frequently added criteria were feasibility (in 22% of all exercises), acceptability (22%), low cost (22%), sustainability (22%) and relevance (12%). This shows the flexibility of the CHNRI process in allowing the use of different priority–setting criteria. Adjustments of the process to the needs of each specific exercise should be strongly encouraged ( Table 2 ).

The main messages from the conducted exercises

As a whole, the 50 CHNRI exercises generated several very broad messages relevant for health research policy. First, if the health issue that was the focus of the prioritization exercise was not well understood in terms of its burden in the population, or the risk factors that contributed to the issue, or the interventions that could be effective in controlling and mitigating the issue, then descriptive (epidemiological) research was identified as the leading research priority as a rule. This showed that generating the knowledge on the burden of the health issue and its “architecture” (in terms of contributing factors and effective interventions) was usually identified as the leading research priority, wherever such knowledge was unavailable.

Given that most contemporary health issues have a reasonably well–defined burden in the population and risk factors, and that effective interventions to reduce or control the burden do exist but are not being implemented, it is not surprising that research on delivery, including health policy and systems, along with operations and/or implementation research frequently dominated the exercise, particularly in low– and middle–income countries [11]. An additional important factor that explains why delivery research was frequently identified as a research priority is the relatively short time frame within which the impact was expected in most exercises (eg, 10 years) and greater urgency to reduce child mortality among the under–privileged populations of the world. Had the health issue been less devastating (eg, mild chronic diseases), and the specified time frame longer (eg, 20–30 years), it is very likely that research priorities would have shifted toward development research and discovery research [11].

Still, there were many examples where “development” (translational) research questions and “discovery” (basic, ie, fundamental) research questions made it close to the top of the list of priorities. Translational research questions were scored highly wherever there were pre–existing and effective interventions which required some clearly defined and straight–forward modification so as to enable their scale–up in low– and middle–income settings (eg, vaccines stable at high external temperatures). Research questions that required discovery (fundamental) research were prioritised in the exercises where the time frame was longer than 10 years and where hardly any effective interventions were available to reduce or control the health issue (eg, the effect of exercise on dementia and Alzheimer disease [12]). This begs the questions: 1) what time horizon(s) grant agencies adopt and how these differ across agencies; and 2) whether this is explicit or implicit and how this is decided – as the time frame of research questions clearly influences research prioritisation.

The key advantages of the CHNRI method

We believe that the popularity of the CHNRI method in setting health research priorities can be attributed to several key advances that it proposed. These advances addressed common concerns that persisted following the previous exercises. First, the CHNRI method is systematic, because it offered an acceptable framework for handling an endless spectrum of research questions, which provided equal opportunity to questions from different categories of health research.

Second, it is also transparent, because it clearly defines the context and priority–setting criteria and provides a replicable approach. All stages of the process and all input can be easily documented and stored in the form of a numerical data set upon which the priorities can be set.

Third, the CHNRI process is democratic. It relies on a “crowd–sourcing” approach to both submission of research questions and scoring of the proposed questions. In this way, no single participant in the exercise can have a decisive (or undue) influence on the final ranks. The scores reflect the collective opinion of the sample of researchers and other experts from the research community, with each individual input contributing only a minor fraction to the overall scores. The central idea of the crowd–sourcing principle is that a diverse collection of independently–deciding individuals will be likely to make certain types of decisions and predictions better than any experts in the great majority of cases [13].

Fourth, the CHNRI process is inclusive, fostering “ownership” of the results by ensuring the various groups invest in the process. This means that an appropriate role is given to donors, researchers and other stakeholders, all of whom can have a substantial influence on the final list of priorities: donors, through defining the context and criteria [9]; researchers, through providing research questions and scoring them [10]; and other stakeholders, through being able to assign more importance (weight) to some criteria over the others [2].

Fifth, the CHNRI process is extremely flexible and adjustable to many different contexts and needs. It is very easy to modify it by adjusting the components of the context and adding additional useful priority–setting criteria, as demonstrated through these first 50 applications. Sixth, the CHNRI process is extremely simple, which we believe is one of the main reasons for its uptake by many groups globally that haven't been trained in the application of the method. It is enough to study any previously conducted exercise to be able to easily organize and conduct it within any other setting. Although quantitative in its outcomes, the CHNRI method is based on a simple, qualitative input (Yes/No), avoiding any complicated mathematical or statistical computation to obtain the results. Intuitive scores that range between 0–100% and measure collective optimism of a group of experts toward each component of each research question are understandable to users, replicable, amenable to agreement statistics, post–exercise validation and evaluation [14,15]. Seventh, the CHNRI method is reasonably inexpensive to conduct. Finally, the results of the CHNRI method are relatively easy to disseminate to the global audience, as the process for priority–setting is structured, objective, replicable and transparent.

The main points of concern to address in the future

There are several concerns that were expressed in relation to the CHNRI process and they will need careful addressing. First, there is a risk that the spectrum of research ideas submitted and evaluated in the CHNRI process is not comprehensive and that it is missing some particularly promising research questions. Second, the response rate of the invited researchers, policy–makers and program leaders typically ranges between 30–70%, which means that a significant response bias could be introduced at this step [9]. It should be explored whether those who responded to the invitation to participate in the CHNRI exercises differed significantly from those who declined [9]. Third, statistical simulations using data sets from the conducted CHNRI exercises established the minimum number of expert scorers required per exercise to achieve “stable” scores and ranks, above which further addition of experts is unlikely to change the results, and these thresholds should be respected [14,15]. Fourth, a series of experiments on quantitative properties of human collective knowledge and opinion was designed and conducted to demonstrate that collective predictions indeed out–perform individual predictions in the vast majority of cases, but there were still some individuals who managed to out–perform the group's prediction [14,15].

Another risk of bias comes from the process of compiling and combining research questions. Reducing several hundreds of research ideas/questions to a number that is feasible for scoring, such as 200 or less, is an important step. It requires knowledge of the subject matter and is therefore usually performed by a very small group of process managers. The way questions are phrased, or how broadly they are framed, may influence the responses and could introduce bias at this step.

Ultimately, it should be demonstrated that the publications based on the CHNRI process have at least some impact on health research funders and research communities. This could be achieved through analysis of bibliometric indicators, showing the impact of the CHNRI papers on the research community and comparing the intensity of research on identified priorities before and after each of the exercises was published. More importantly, a series of interviews with research policy makers at key funding institutions should be conducted to learn whether they are aware of the CHNRI method and if they have been using it themselves to set research priorities, or used the results of the conducted exercises in their decision–making.

Opportunities for further development and implementation

The CHNRI method for setting health research priorities was developed to support decision–making for investments in international child health research at a regional level (low– and middle–income countries). However, its advantages have helped its expansion beyond its initial boundaries. There are clearly many opportunities to implement the CHNRI method to address research priorities relevant to all other population health issues. Moreover, the ease of implementation and low cost should help its implementation at a global, national and sub–national level. The development of a massive open online course (MOOC) in CHNRI implementation may facilitate its wider adoption. Another welcome progress would be the development of a free web–based and mobile phone–based and fully automated CHNRI application platform, which would further simplify the exercise and the computation of scores and agreement statistics, based on widely available spreadsheet software.

Finally, the CHNRI method shows how the area of global health may be particularly receptive to solutions based on “the wisdom of crowds” and crowd–sourcing. The CHNRI exercise could be conducted to set priorities among further ideas for crowd–sourcing–based solutions in global health. The world–wide web, mobile phones and crowd–sourcing could potentially serve to generate a massive amount of useful information in real time and solve a diverse set of problems ranging from coordinating funding support, alerting the development of epidemics, identifying areas of medical supplies shortage, monitoring program implementation over large geographic areas, estimating disease burden, effects of risk factors and impact of implemented health interventions in real–time, and many others.