The CHNRI method was used to marry opinions of both experts in global health and crowdsourcing to generate and systematically rank potential uses of crowdsourcing in global health. The CHNRI method uses the principle of collective opinion, or wisdom of the crowds, to score ideas against pre-selected criteria which in turn enables donors, policy makers, and programmers to view the strengths, weaknesses and relative ranking of each idea. Over 50 CHNRI exercises have been conducted, the majority of which have been in the areas of maternal and child health, including research priority setting exercises embedded in high-profile series intended to inform policy, such as the Lancet Newborn Series and Lancet Diarrhoea and Pneumonia Series and others exploring universal health coverage, early child development, and quality of care for maternal and child health [15-20].

Experts in both global health and crowdsourcing were invited by email to participate through a combination of snowballing, literature searches and participation in previous CHNRI exercises. Those who did not respond to our first invitation were not followed-up. Those who agreed to participate were provided ‘seed questions’ and explanations of the context of crowdsourcing and global health. 25% of participants were from Asia and 13% from Africa; 34% of participants resided in LMICs. The participants represented a range of viewpoints, including academic, research, programmatic, non-governmental, and donor organisations. A list of participants who scored ideas can be found in Table S1 in Online Supplementary Document(Online Supplementary Document) .

94 participants submitted 361 suggestions for potential ideas of applying crowdsourcing in global health. 239 ideas remained once duplicates were removed and similar ideas combined. Ideas were then scored against 7 criteria, developed specifically for this exercise: (i) technological possibility and cost/time of development; (ii) feasibility in a low- or middle-income country; (iii) issues surrounding use; (iv) scale; (v) impact; (vi) equity; and, (vii) cost, time and innovation. The criteria were meant to address issues of technology development and use in LMICs, equity, impact and innovation over current or traditional interventions or data monitoring activities, and respect for local practices and cultural beliefs. Each criterion has three sub-questions and can be found in Box 1 .

Each expert was assigned to score 3 of 7 criteria against all ideas in order to minimize scorer fatigue. Criteria were assigned in accordance with each expert’s background. In all, 53 experts completed scoring. Table 1 displays the number of scorers assigned to each criterion, the number of scorers who scored each criterion and the average number of ideas scored within each criterion (as some scorers left blanks). Relative ranking of ideas was calculated using the average across criteria; within-criterion scores and average expert agreement (AEA) were also calculated. Finally, ideas were categorised into themes. The Box S1 Online Supplementary Document(Online Supplementary Document) contains further information on the calculations.

239 ideas for potential uses of crowdsourcing were systematically scored against 7 comprehensive criteria by 53 experts. Scores ranged from 80.39 to 42.01 and AEA ranged from 0.73 to 0.48. Scores within each criterion ranged as follows: (i) technical possibility and cost/time of development, 0.98 to 0.45; (ii) feasibility, 0.91 to 0.32; (iii) issues surrounding use, 0.98 to 0.51; (iv) scale, 0.79 to 0.26; (v) impact, 0.86 to 0.29; (vi) equity, 0.74 to 0.14; (vii) cost, time and innovation, 0.85 to 0.31. The top 15 ideas, their scores within each criterion, overall scores, AEA, and themes can be found in Table 2 . The full list of ideas, their categorisations into themes is available in Table S2 in Online Supplementary Document(Online Supplementary Document) .

Themes identified include epidemic response, problem solving, data generation, research, education and novel discoveries. Overwhelmingly, the ideas fell into the themes of problem solving (n = 112) and data generation (n = 91). Ten ideas related to epidemic responses, including the top-ranked idea. Eight ideas related to research and to education, seven to diagnostics, and three to novel discoveries.

Using large crowds of health workers to rapidly share information about a disease outbreak, ensuring a more global response, ranked first. Its score was higher than all others by 3 points, and also had the highest overall agreement (AEA = 0.73). This idea was the only to score in the top 25th percentile across all seven criteria. Four other ideas in the top 15 also related to containing communicable diseases; another related to the theme of epidemic responses, one to research, another to access to health and the final to data generation. The criteria that drove the high scores for these ideas were technological possibility and cost/time of development and issues surrounding use, which were either the first or second highest scores for all five of the ideas. Three of the five ideas had their lowest scores in the equity criterion.

Experts felt that crowdsourcing would be a feasible platform for vital registration. The registration could be developed through involvement of community or religious leaders or CHWs, or over SMS in the case of hard-to-reach areas or during conflict or natural disaster. These ideas fell into the theme of data generation and scored highest in issues surrounding use and technical possibility and cost/time of development, but they scored low on equity.

Ideas involving improving maternal and child health were also identified in the top 15. These fell into the categories of problem solving and data generation. They scored high in technical possibility and cost/time of development and feasibility. Despite ranking 7th overall, one idea had a score of only 0.49 in equity.

One idea involving novel discoveries ranked highly, seeking to find a novel way to measure nutritionally relevant issues. This idea’s high score was driven by the issues surrounding the use criterion, where it scored 0.92, and it had fairly unremarkable scores in other criteria.

A sub-group analysis was conducted on ideas relevant to humanitarian or conflict settings, and the top 15 ideas in these settings are displayed in Table 3 . As these ideas are taken from the same list of general ideas, many overlap with the overall top 15. Six of the ideas were categorised as epidemic responses, six as data generation, two as education and one as problem-solving. Many of the ideas in this category focused on disease spread. Ideas in this domain, for the most part, ranked highly in issues surrounding use and technological possibility and cost/time of development and scored lowest in equity, feasibility, and impact, respectively. The range of AEA for the top 15 conflict ideas was 0.73 to 0.58.

The bottom 15 ideas can be found in Table 4 . The lowest ranked idea (RPS = 0.42, AEA = 0.58) was to ask families where they are seeking care and how much they have spent; this idea was categorised as data generation. It scored the lowest in scale (0.26) and equity (0.28).

The lowest 15 ideas scored low across all criteria. Most notably, an idea based on an already successful crowdsourcing application (U-Report in Uganda) ranked in the bottom 15, at #232, had an RPS of 0.49, and scored lowest in equity (0.14). This score was the lowest score received by any idea for any criterion. The idea asks that people submit an SMS to report poor behaviour in the health sector, such as bribes, counter fees, and absenteeism. A similar idea asking to counter corruption in the health sector more generally scored much higher at #102; this idea scored higher in equity as well (0.51).

This was the first application of the CHNRI method, which is in itself a crowdsourcing-based exercise, to explore potential uses of crowdsourcing in global health and in conflict settings. This exercise brought together expertise from experts in global health and in crowdsourcing, using their collective opinions to systematically and transparently generate and score ideas against tailored criteria.

The AEA in our exercise was fairly low, ranging from 0.73 to 0.48. This could be a result of combining different expertise, from the global health and crowdsourcing communities, or be reflective of crowdsourcing being a new field, or that other exercises have generated ‘research priorities,’ which there may be more consensus on.

The vast majority of ideas fell into the categories of problem solving or data generation (n = 112 and n = 92, respectively). As crowdsourcing is often used to fill a knowledge gap, complete a task or solve a problem, this is not surprising. However, as other categories included novel discoveries, research and diagnostics, which current crowdsourcing companies, competitions, and pilots are aimed at, it is surprising that there were so few ideas proposed in these areas. Indeed, major drug companies currently use crowdsourcing competitions to produce novel discoveries as a means to cheaply conduct research on new interventions [21]. In previous CHNRI exercises, novel discoveries have often scored low due to short timeframes [15]; however, our exercise did not stipulate any timeframe. Not only did two of three novel discoveries score low, but also only three were suggested. This should be investigated in further exercises.

Even in many of the top-ranked ideas, the equity criterion scored low. Upon further examination, despite our best efforts, the wording of the sub-questions for equity may have not been worded to suit the proposed ideas. For example, while an idea might be considered equitable overall, it may not specifically result in improving equity for quality of care and access to care and distribution of disease. To have an impact on all three of these characteristics would be excellent, but would require quite a comprehensive intervention.

The top-scored idea, to use crowdsourcing to share information regarding disease outbreaks and epidemics, could be related to the timing of the exercise. The experts were invited to submit their ideas shortly after the Ebola crisis, where the slow response was catastrophic and widely criticised [22,23]. During the scoring phase, early reports of Zika were surfacing. However, crowdsourcing is an ideal method to spread information, especially regarding early disease reporting in hard-to-reach or remote areas. Various crowdsourcing platforms have already been successfully used in disaster and humanitarian response settings, such as Ushahidi, Frontline SMS and Geochat [24]. Ushahidi is an open-source crowdsourcing application originally developed to respond to election violence in Kenya. It collects geotagged reports via SMS, voice, and email. The ideas that ranked highly in the conflict-relevant list ( Table 2 ) are, naturally, particularly amenable to being addressed by Ushahidi and similar platforms, which have already been used in similar situations.

Interestingly, although ‘wisdom of the crowd’ type of crowdsourcing was predominant in a recent literature review of crowdsourcing in health [5], especially crowdsourcing in diagnostics, the experts overall did not score these ideas as the most promising uses. Additionally, many of the uses of crowdsourcing identified in the literature review specifically combined crowdsourcing with machine learning, fine tuning algorithms to merge results from humans and computers to create the best predictive result; these appeared to be among the most successful instances of crowdsourcing. Only one idea, #230, suggested to use crowdsourcing to train algorithms, and although it does not specifically mention machine learning, this appears to be implicit. Interestingly, this idea scored quite low in technological possibility and cost/time of development (0.50). This may be indicative to challenges of bringing two different fields together.

Within the conflict/humanitarian related results, #11 asks whether crowd sourcing could be used to de-escalate tensions or diffuse conflicts in a trouble spot. This idea highlights the potential to use social media to mobilise crowds to achieve a goal. Interestingly, since the exercise’s completion, a movement called the “Injustice Boycott” has formed in the United States in response to the United States election, alongside Black Lives Matter and the Dakota Access Pipeline protests. The Boycott is e-mail and social media based, has mobilised over hundreds of thousands of people and produced real results, including pressuring the city of Seattle to divest from Wells Fargo [which supported the Dakota Access Pipeline), pressuring the Governor of New York to raise the mandatory minimum age for children to be tried as adults, to compel the Golden State Warriors to fire an employee known for overt racism and raising half a million USD for Standing Rock [25]. The Injustice Boycott is an example of how social media can be used to mobilise people to influence government and organisations to produce palatable results.

While this exercise was not primarily focused on the Sustainable Development Goals (SDGs), we noticed that many of the ideas generated relate to a wide range of the SDGs. In the top 15, ideas relate predominantly to the SDG 3.3, addressing epidemics and eliminating communicable diseases. Other SDG targets found in the top 15 include SDG 16.9 (providing legal identity for all, including birth registrations), SDGs 3.1 and 3.2 (reducing maternal and child mortality), SDG 4.3 (increasing the number of youth and adults with relevant skills and training for employment), SDG 3.8 (achieving universal health care and increasing assess to quality health care, essential medicines and vaccines) and, finally, SDGs 2.1 and 2.2, which focus on ending hunger and malnutrition (with the latter focusing on vulnerable populations and periods of strategic growth). In the conflict-specific list, using crowdsourcing to address SDG 1.5 (building resilience of the poor and vulnerable, reduce exposure to climate-, economic-, social-, environmental-related shocks and other disasters), SDGs 5.2 and 5.3 (eliminating violence against women and girls, and eliminating harmful practices such as early and forced marriage and female genital mutilation), SDG 11.5 (significantly reduce the number of deaths and people affected and substantially decrease the direct economic losses caused by disasters), SDG 13.1 (strengthen resilience and adaptive capacity to climate-related hazards and natural disasters), SGD 16.1 (significantly reduce all forms of violence everywhere) and SDG 16.2 (end abuse, exploitation, trafficking and all forms of violence and torture against children) were also prioritised. Many ideas addressed more than one SDGs concurrently. With this post-hoc analysis, there was more diversity in the SDG targets addressed by the highest ranked ideas, whereas the ideas that were ranked lower either did not address an SDG or often related to SDG 3.8 (achieving UHC, including quality of care) and usually appealed to the improvement of the quality of care aspect of the target. It would be interesting to conduct another CHNRI exercise to specifically explore uses of crowdsourcing for SDG targets.

The Millennium Development Goals (MDGs) were not achieved by many countries, and at the launch of the SDGs, no country was meeting all the health-related SDGs [26]. Many argue that the SDGs are more ambitious, and now the world is facing greater challenges as populations are becoming rapidly displaced. Indeed, one in 113 people are displaced according to recent statistics, which is the highest since the UNHCR began keeping records [27]. We cannot think of health without thinking of risk factors that affect almost 1% of the world’s population. Innovative solutions, such as crowdsourcing, may help achieve these goals. Indeed, the ideas generated address many of the SDGs. It would be beneficial to conduct an additional exercise exploring how crowdsourcing could address SDGs, specifically.

With 90% of the world’s population carrying a mobile phone [29], the potential for involving laypeople in health decisions or health reporting through mHealth technology is enormous. While crowdsourcing has been used in global health in pilot studies for diagnostics and in platforms in humanitarian settings already, it is not being used to its full potential [30]. This exercise is the first systematic, transparent assessment of the potential of crowdsourcing in global health and conflict settings. Its results can help future researchers, funding agencies, donors and policy makers choose which would be most appropriate and promising for their goals. Crowdsourcing is low-cost, rapid, and can complement traditional data collection; these qualities make it especially advantageous in global health, conflict or humanitarian settings.