These concepts were introduced as a part of the initial CHNRI method description [4,5]. The multi–disciplinary working group that developed the CHNRI method recognised the need to find a practical way to involve a much larger group of stakeholders in the priority–setting process. An agreement was reached that, at least in principle, most members of the public would not be expected to generate research ideas or score them, because they do not possess the knowledge that would enable them to discriminate among the proposed research ideas. Instead, it was agreed that their contribution to the process and the final results of the exercise would be be in the assignment of “weights” to the criteria that reflect their collective preferences and beliefs. Over the years of CHNRI implementation, it has been shown that stakeholders originating from funding institutions or political organizations prefer the criterion of maximum potential for disease burden reduction, because their targets are usually set around this criterion; programme managers are typically more focused on the deliverability and sustainability criterion; stakeholders from the industry tend to prefer knowing the likelihood of effectiveness of resulting interventions; while members of the general public often emphasize equity and ethics as their preferred criteria [6].

In addition to placing more “weight” on some criteria than others, which could affect the final rankings of all research ideas as a result of stakeholders' input into the CHNRI process, the stakeholders can also disqualify some research ideas using the system of “thresholds”. This means they may agree a priori that a research idea will not be considered a priority unless it reaches a certain minimum score against a particular priority–setting criterion. This can be important in a specific context; eg, in the aforementioned example of South Africa, where equity was a very important concern for all stakeholders, they could have insisted that a research idea must have a minimum score of 80% on the “equity” criterion to qualify as a priority. In practice, this means that a research idea with scores 50–70% on all other criteria, but 90% on “equity”, could be considered a research priority. However, another idea with scores of 80–90% on all other criteria, but 60% on “equity” would be disqualified from the exercise – or at least delayed, until it addresses the recognized issues with equity. Common examples of the latter are the new, high technology–based interventions that would likely first be utilised by the wealthy. In this way, research ideas with lower overall scores could be seen as greater priorities if they pass all the pre–defined “thresholds” [3,4].

Although the interdisciplinary group that developed the CHNRI method considered this approach as practical and inclusive, the question remained of how best to select the stakeholders and ensure their representativeness to the entire community of interest. Possibly the best solution to this problem to date has been achieved by Kapiriri et al. [3] who aimed to develop a “global” group of stakeholders by conducting an internet–based survey of the affiliates to the “Global research priority setting network”, which had been assembled in the years prior to the development of the CHNRI method by the staff from the University of Toronto, Canada. Between March and May 2006 a large number of affiliates to the “Global research priority setting network” agreed to participate in a pilot on the condition of anonymity. They agreed to provide stakeholder input to five forthcoming exercises that aimed to set research priorities to address the five major causes of global child mortality. Respondents included a very diverse mix of researchers, policymakers and health practitioners with an interest in priority setting in health care from high–, middle– and low–income countries. Participants were given a simple version of the questionnaire, and were asked to rank the five “standard” CHNRI criteria from 1st to 5th in the order of their perceived importance of the criteria. They were also asked to set a threshold for each of the five criteria. The respondents placed the greatest weight (1.75) on potential for disease burden reduction, while the weights for the remaining four criteria were similar to each other, and ranged between 0.86 to 0.96. The highest threshold was placed on the criterion of answerability in an ethical way (0.54), while the lowest was placed on potential for disease burden reduction (0.39).

We identified 50 research prioritization exercises using the CHNRI method that were published between 2007 and 2016. Of the 50 exercises, 38 (76%) did not seek inputs from stakeholders and 12 (24%) involved stakeholders as their larger reference group. This already shows how it may be remarkably difficult in most cases to identify and involve an appropriate group of stakeholders that would be representative of the wider community of interest – whether this is a global, regional, national or local population. It seems that, in the absence of simple solutions, most authors who conducted the CHNRI exercises preferred not to include stakeholders in the process, rather than including an ill–defined and non–representative group and then having to adjust the final ranks based on their input. By not including input from stakeholders, the CHNRI exercises simply remained “unfinished” to an extent, though weights and thresholds could still be applied post–hoc should an appropriate group of stakeholders be identified at some later stage – unless the context changes substantially in the meanwhile.

Among the 12 CHNRI exercises that involved stakeholders and took their input into account, 5 were papers that belonged to the series of exercises related to addressing research priorities for the five major causes of child mortality globally – eg, pneumonia, diarrhoea, neonatal infections, preterm birth/low birth weight, and birth asphyxia [7–11]. All of these papers were co–ordinated by the World Health Organization (WHO) and they used the weights and thresholds defined above by Kapiriri et al. [3]. However, the remaining seven exercises made their own individual attempts, using guidelines for implementation of the CHNRI method, to identify appropriate stakeholders within their own contexts and involve them in the process. This section explores the experiences and results from these seven studies. Table 1 summarizes the approaches to involving stakeholders in these seven exercises.

Two exercises were carried out at the global level. They were focused on mental health research and acute malnutrition in infants less than six months, respectively [12,13]. The remaining five exercises were conducted at the national level and focused on research in child health in South Africa [14], zoonotic disease in India [15], health policy and maternal and child health in China [16,17], and Prevention of Mother–to–Child Transmission of HIV (PMTCT) in Malawi, Nigeria and Zimbabwe [18]. Given that the large majority (over 80%) of the 50 CHNRI exercises were focused on either the global context, or on all low– and middle–income countries (LMIC), the high representation of national–level exercises among those CHNRI studies that used stakeholders input (5/12) is likely a reflection of the fact that it is much easier to involve stakeholders at the national or sub–national level than it is on a regional or global level.

In all exercises, the stakeholders involved were first given an induction course about the CHNRI process. Then, an opportunity for asking and sharing questions and concerns with respect to the CHNRI process was provided. In five of the seven exercises, stakeholders were asked to rank the relative importance of the pre–defined criteria from most important one (“1”) to the least important (“5”), while considering the context of the research prioritization. The average score was calculated for each criterion and was then used to calculate the relative weights by dividing the average expected score of 3.0 (ie, the average expected rank if all criteria were valued the same) by the mean assigned rank. For example, a mean assigned rank for “answerability” criterion of 2.47 translates a relative weight of 1.21 (ie, 3.00/2.47 = 1.21). In this way, “answerability” will receive 21% greater weight than if all the criteria were weighted equally.

The concept of thresholds was very rarely used. Even when it was applied, it was clear that it wasn't properly explained to participating stakeholders. This is not surprising, because the thresholds really refer to a measure of “collective optimism” of the scorers, rather than a real computation of likelihood or probability that is rooted in any real–world parameters. It is very difficult to estimate what this measure of “collective optimism” could amount to for different criteria. This is why such attempts to set thresholds typically resulted in them being set at 25%–30%, much too low to have any discriminatory power and disqualify many research ideas, so that almost all research ideas passed all the thresholds.

In the remaining two exercises, the nature of stakeholder involvement was modified radically from that which was originally envisaged in the CHNRI exercises with reasonable justification [13,18]. Instead of using the group of stakeholders only to adjust the ranks that were derived from an expert–driven scoring process, the authors involved a broad range of stakeholders in the generation of research ideas [18] and/or scoring the research ideas [13,18]. We will now reflect on these experiences in a critical way, identify some lessons learnt and make recommendations for future exercises.

In the 7 studies that tried to develop a larger reference group of stakeholders that would be appropriate to their respective contexts, the number of stakeholders involved was disappointingly small: it ranged from 20 to 70. Although attempts were clearly made to ensure diversity of the stakeholders involved, such small sample sizes can hardly be considered sufficiently inclusive of many different groups of stakeholders and their representativeness. Although good representativeness of stakeholders can be ensured without necessarily requiring a very large number of participants – such as, eg, in many examples of national parliaments in democratic societies, who represent all the people of the nation through a relatively small number of their elected members – we still feel that bigger numbers would ensure more legitimacy to the process, or more relevance of the outcomes to the context of the exercise.

.  Photo: Meeting with a group of stakeholders at the maternity health clinic in Ghana (Courtesy of Dr Alice Graham, personal collection)

It would be difficult to consider the examples in the reviewed exercises as truly representative of the wider communities, let alone the nation or the world. This shows that despite the authors’ best intentions to fully adhere to the guidelines and complete the CHNRI process, they didn't really manage to find a satisfactory solution to involving large and diverse group of stakeholders. In these papers, the profile of stakeholders often included researchers, who would have been better reserved for the scoring process. Other stakeholders included clinicians, government officials, and representatives of academia and professional organizations, which again are rare in the society and hardly representative of the wider community. The examples of the profiles of persons who we would expect included in the larger reference group are also laypersons, frontline health workers and direct beneficiaries of health services, such as patients who contracted disease of concern. We encourage the authors of the future CHNRI exercises to try to get as much feedback as possible from those groups, because they have their own specialised knowledge (including lived experience), which would not be captured by other participating groups in the process. They also have “stake”, or interest, in the outcome of the exercise.

The small sample sizes and differences in approaches to ensure diversity and representativeness of the stakeholders led to large variations in stakeholders' input [12–18]. In the global exercise, the greatest relative importance was assigned to effectiveness, and the lowest to answerability, though these results should not be generalized. Stakeholders at the national level varied in their preferences, alternately supporting the criteria equity, deliverability (with affordability and sustainability), or the maximum potential for disease burden reduction ( Table 1 ). Clearly, small sample sizes used in these exercises limit the generalizability of such preferences even within their local context, let alone more broadly.

It is also important to note that in all exercises that applied the “weights”, this procedure didn't really have dramatic effects on the final rankings of the research ideas. Although a research idea might move a few places up or down the list following the weighting procedure, these shifts did not profoundly affect the non–weighted ranking order that was determined by the researchers and experts. Perhaps this is one of the additional reasons why so many groups conducting the CHNRI exercise did not place sufficient importance on involving stakeholders. From the exercises that involved stakeholders, one might conclude that the process of expert scoring is sufficient and the outcome of the exercise will not be greatly altered by the involvement of stakeholders. We believe that such a view is premature and would like to see more examples of the involvement of the stakeholders in the CHNRI process before such judgements could be made.

In two exercises that actively involved stakeholders, their involvement wasn't limited to weights or thresholds, but rather they were also involved in research idea generation and scoring [13,18]. In the exercise on PMTCT in three African countries [18], about 40–70 people took part in respective countries, and all participants contributed to all stages of the CHNRI process. This included academics/researchers, district health workers and implementing partners such as UN agencies, people living with HIV/AIDS, frontline health workers and policy makers. The authors’ justification for including these diverse groups in all stages of the CHNRI process was to avoid discriminating within this diverse range of groups, but to truly engage the groups according to their technical expertise and to enhance inclusiveness and participation in similar priority–setting exercises across the nation. Eventually, the stakeholders' weighting of the scores was not even applied, possibly due to an assumption that it was no longer needed. This example represented a rather interesting deviation from the original CHNRI conceptual framework, but we can see a rationale for this modification, which makes it an illuminating exception.

The other exercise, on the management of acute malnutrition in infants in low– and middle–income countries, involved stakeholders only in the scoring process [13]. The stakeholder group included participants at meetings and symposia related to the topic area ( Table 1 ). In this exercise, the core group of researchers (“management team”) developed the list of research questions based on the review of the literature in this field that preceded the CHNRI exercise as the preparatory step. The final list of questions was then circulated for scoring to both researchers invited to the CHNRI process and also the conference participants, who were considered stakeholders. Equal weighting was given to all criteria. The management team justified this on the grounds that malnutrition was a new area of research in infants younger than 6 months and they therefore believed that unweighted estimates would be more suitable and interpretable by their intended policy–maker audience. However, the authors stated that the lack of weighting of criteria might have resulted in limited reflection of the values in the broader community. In this case, we can conclude that the borderline between the invited researchers and the “stakeholders” (who were likely to include unrelated researchers and any other people of similar profile who could be expected to attend an international conference in this topic), was blurred and not really clear. It is likely that this deviation from the suggested approach didn't really invalidate the conceptual framework, because all the scorers would still be expected to possess knowledge on the topic of interest. It would perhaps be more appropriate not to call the second group “stakeholders”, but rather an additional, “convenience” sample of scorers that increased the number of scorers considerably.

So far, there hasn't really been a good example of stakeholder involvement as originally envisaged by the CHNRI across the first 50 implementations, apart from perhaps the Kapiriri's priority–setting network involvement that was used in 5 child mortality papers [3,7–11]. This is certainly a shortcoming of all the previously conducted processes. This finding may also reinforce the initial concern that involving stakeholders in research priority setting processes is very challenging and that the solutions proposed in the original CHNRI method were quite difficult to implement as envisaged.

This is not to suggest that the results of previous CHNRI exercises are not useful, and the thresholds and weights can be applied later, if a good solution to obtain them can be found within the time scale during which the context described to scorers would still remain largely unchanged. The efforts conducted to date to perform the CHNRI exercises were not wasted and their results can be used. However, it must be acknowledged that most CHNRI exercises to date are, in fact, incomplete at least with respect to the original vision for them. To bridge this gap better definition is needed of who are the stakeholders at different levels (ie, global, regional, national and local) and how best to represent them.

For global exercises, we'll inevitably need a very large and inclusive crowd–sourcing exercise of many stakeholder representatives, who would place weights and thresholds on all 25 priority–setting criteria that were used to date across all 50 CHNRI exercises (5 “standard” and 20 new). The sample of stakeholders will need to be truly large, because we may later need several sub–samples that could provide us with region–specific stakeholders, or allow selecting specific groups of stakeholders and leaving others out of the exercise. In this way, the large “global” sample of stakeholders would also serve as a base for the regional samples of stakeholders. A major concern relating to this suggested approach would be how to avoid a strong urban bias in low–income settings and be inclusive of undeveloped and/or rural areas. In terms of national–level or local–level exercises, it is likely that highly targeted samples that aimed to include 500–1000 stakeholders would already be sufficient and representative of national or local context. The “targeting” component of the sampling strategy would define the profile of the stakeholders that would be most appropriate to the exercise, and then a person could be found in the community to fit each such profile.

How could these large sample sizes be achieved technically? How could we engage thousands of people globally, or hundreds nationally? With further attention to the development of the area of “crowd–sourcing” in the age of the internet and social networks (such as Facebook, Twitter, etc.), we should be able to do lot more in the future with respect to truly engaging the stakeholders in the process of setting priorities in health research investments at different levels of the human population.