Diagnosis

Diagnosis was the most common usage of crowdsourcing in health. Crowdsourcing has been used multiple times for diagnosing malaria, specifically, and then for grading images in order to diagnose various conditions and diseases [24-28,30-32,88]. It has also been used to assist physicians in diagnosing conditions [34,35].

Three articles described the BioGames project, where laypersons were able to diagnose red blood cells (RBCs) infected with malaria [24-26]. Mavandadi et al. argue that rapid diagnostics for malaria are expensive, unreliable due to heat/stability issues and not trusted by health care workers (HCWs) in low- and middle-income countries and that gamification of malaria diagnosis could improve the management of malarial fevers, be a better use of funding and reduce drug stock-outs [24]. BioGames was available by Android and online. Gamers were given a tutorial, then in the game had a syringe to ‘kill’ infected cells and ‘collect’ healthy ones. The gamers reached an accuracy of 99% (95.1% sensitivity, 99.4% specificity), with the highest level of accuracy being with the largest crowd; however, the authors believe that with a larger crowd, a hybrid algorithm of machine learning and crowdsourcing would perform optimally. Ozcan argues using this method to create gold standard image libraries, for telepathology, point-of-care diagnostics in other conditions and also, to train HCWs in low-income settings [25]. Using the same platform, Feng et al. report on training an education module of the game. The authors used positively or negatively marked cells and had training and diagnostician versions so users could see progress, review misdiagnosed cells and view their accuracy in comparison to their peers. The authors found it was easiest to diagnose negative cells and proposed that in future, laypersons or machines could pre-screen negative cells and send questionable and positive cells to experts for diagnosis. The BioGames project had >2150 gamers participate from over 77 countries, providing over 1.5 million diagnoses [26]. A second malaria diagnostic project, MalariaSpot, utilised gamers over a website to also diagnose malaria in RBCs online. In the span of a month, this project had participants in over 95 countries, and over 12 000 games played [27]. Participants were asked to tag as many malaria parasites as possible in 1 minute. Like BioGames, the accuracy of MalariaSpot was also over 99%.

Crowdsourcing has also been used to diagnose conditions through grading images. Mitry et al. used the Amazon Mechanical Turk (AMT) platform, which is an online crowdsourcing platform that pays users micropayments for small tasks, to grade glaucoma images. The crowd performing the tasks on AMT are called “Turkers”. The authors obtained 2540 classifications of 127 disc images, with 83.22% sensitivity, but only 35%-43% specificity and suggested further micro-izing tasks to improve accuracy [28]. AMT was employed in another study to grade images to screen for diabetic retinopathy, with three phases of grading. The authors found that Turkers were able to sufficiently determine normal vs abnormal, but had trouble grading the severity of retinopathy; however, sensitivity for whether retinopathy was present was 100% at all stages [29]. Another study used a website to gather a crowd’s scores of estrogen receptor in breast cancer tumour tissue. A portion of the tissues was previously scored by a pathologist, allowing the authors to assign users a ‘user performance scale’ and a reliability/trust score for the crowd. The authors found the crowd’s ability was similar to that of a trained pathologists [30]. Gehl et al. used a non-internet based crowdsourcing study to explore skin self-examination (SSE) for atypical moles. The authors recruited 500 participants from a mall, administered a pre-test and post-test and provided each participant with basic SSE techniques. The analysis used a threshold of 19% of participants to identify the mole as abnormal for it to be considered as such. Using this threshold, the participants correctly identified 90% of the melanomas and 72% of the non-melanomas [31]. Xiang et al. attempted to use crowdsourcing to answer the need for a scarcity in medical imaging specialists outside major cities in China. The authors recruited 13 students in 2nd and 3rd year of medical school with a medical imaging major. The average accuracy for the individual was 39.54%; however, the authors were able to increase the accuracy using crowdsourcing algorithms to 56%, and found that machine learning algorithms performed higher. The authors noted that since all the participants were from the same department in the same medical school, the lack of diversity and experience may have contributed to the poor results [32].

Three articles described used crowdsourcing to either assist physicians in providing a diagnosis or to replace physicians in providing a diagnosis. Cheng et al. investigated the feasibility of three crowdsourcing platforms to provide diagnoses: volunteer platforms (ie, Yahoo! Answers), Amazon Mechanical Turk (AMT), and O’Desk. They assessed cases of easy, medium and hard levels of difficulty. The case studies of hardest level of difficulty were taken from CrowdMed.com, which is a website which enables patients with “mystery” diseases to provide their detailed medical information to a crowd of medical experts (including specialists and medical students) who will submit their answers, and the correct diagnostician receiving a reward. The easy and medium levels were taken from another paper, which posted medical questions on Facebook to see if Facebook friends were able to answer [33,89]. The authors’ attempts at posting on the volunteer sites failed, as they did not receive responses. On AMT, the Turkers were able to solve many of the easy cases but had trouble with the medium difficulty cases, although the authors stated that these may have been poorly described. None of the CrowdMed cases were answered correctly by AMT workers. O’Desk is a crowdsourcing platform that contracts employees, and the authors hired health care professionals. All the easy level questions were answered correctly by O’Desk, but each only answered one medium level case correctly. The original O’Desk workers declined to answer the CrowdMed cases, expressing uncertainty; the second contracted O’Desk workers were unable to answer correctly.

The DocCHIRP (Crowdsourcing Health Information Retrieval Protocol for Doctors) is a mobile application that helps clinician’s problem solving at the point-of-care. Sims and colleagues presented the experiences of clinicians using the application. 78% of clinicians using the application reported benefit on routine patient care, medical education and accurate referrals, as well as diagnosing unusual cases. Some concerns reported by clinicians included lowered productivity, due to responding to the application, and interference with ‘off the clock’ time, though the latter was only reported among non-users [34]. McComb and Bond also reported on an application that assists clinicians in making diagnoses. Their application, called CoDiagnose, has junior doctors upload case information and receive feedback from a crowd of expert clinicians and it features a built-in e-Learning component. The junior doctors’ diagnostic accuracy improved 14% with the use of the application; however, the authors reported a lack of enthusiasm on part of the expert clinicians about participating [35].

Surveillance

Another very common purpose to which crowdsourcing is used in health is surveillance, both in the context of research and in emergency situations for programming. A number of articles described pilots or projects that employed crowdsourcing for health surveillance [36-42]. As of 2013, 70% of the world's population carried a mobile phone [90], making surveillance through mHealth a promising avenue.

Freifeld and colleagues reviewed a number of crowdsourcing platforms that have been used for community surveillance and participatory epidemiology. Frontline SMS, also called Frontline SMS Medic, enables users to request needs via SMS. It has been used in Malawi, Burundi, Bangladesh and Honduras. Ushahidi is an open source crowdsourcing application that collects individual reports via web, SMS, and email. It can classify, translate and geotag results. Ushahidi was initially created in response to election violence in Kenya, but it has been used most famously in the aftermath of the Haitian earthquake. It has also been deployed in Uganda, Malawi, Afghanistan, and Zambia. Ushahidi also has a feature for collecting voice reports, which is essential for people who are not literate. Geochat is another crowdsourcing application. It aims to aid in faster and more coordinated responses to disease outbreaks and natural disasters. Team members use the application to communicate their location through SMS, email and web. This information synchronises on all users’ devices. The application has been launched in Thailand and Cambodia [36].

Asthmapolis is a GPS-enabled inhaler that is linked to a user’s phone and tracks asthma attacks. The application compiles the information from those using its inhalers and generates a risk map for environmental triggers [36]. Freifeld et al. also reported on two other crowdsourcing applications, HealthMap and OutbreaksNearMe, which mapped influenza in the United States using submissions by laypersons. Chunara et al. also reports on a similar tool, FluNearYou, which maps influenza outbreaks using submissions by laypersons and generates a map to identify outbreaks [37].

ClickClinica is an application that was developed to provide General Practitioners (GPs) and Medical Doctors (MDs) indexed guidelines for diseases. Currently, GPs and MDs use the application to diagnose diseases by asking questions about the condition of the patient. The submitted data are graded by the quality of the user information, ie, if the submitter has a medical ID or an institutional email. Over 1000 MDs across the world have been using the application, despite it not being marketed. The application can also be used to increase recruitment for clinical studies through GIS notification of nearby, relevant studies. The authors suggested that this application could be developed into real-time global surveillance based on usage [38].

Qureshi and colleagues described the Jaroka Tele-Health System, which employed Lady Health Workers in rural Pakistan to use SMS/MMS to register patients, report symptoms, acquire prescriptions and connect to specialists. The resulting data was then used to track disease spread and the authors have been able to use the visualised data via crude numbers or rates and identify clustering; the authors found higher rates of disease during times of migration and with internally displaced persons, which could be explained by poor sanitation and overpopulation. They also found increased rates of hypertension in women in these populations [39].

Lwin et al. designed a participatory epidemiology application, called Mo Buzz, in Sri Lanka to combat dengue. Mo Buzz has three components: (i) predictive surveillance; (ii) civic engagement; and, (iii) health communication. The predictive surveillance component uses a machine learning algorithm to determine weather, vector and human data in the form of hotspot maps for the public and for health officials. The civic engagement component involves people reporting breeding sites, symptoms, and bites; these are reflected on the hotspot map. Finally, this information is communicated to the public and to health officials [40].

Chunara et al. use AMT to survey the malaria prevalence in India. Turkers are asked questions regarding malaria symptoms, date of onset, the malaria status of their household members and their awareness of malaria in their communities. The study found that diagnosis peaked in August and this correlated with official reports [41].

A recent study using Global Burden of Disease (GBD) data examined creating a game to scan the GBD database for errors, as algorithms currently used are imperfect. While participants were staff from the Institute for Health Metrics and Evaluation at the University of Washington, and thus not laypersons with no background knowledge, the author found that using gamification increased the accuracy of results by 1.7 times [42]. As only 4% of infectious diseases have been comprehensively mapped, innovative solutions such as AMT and other crowdsourcing applications detailed above may be useful in providing infectious disease mapping and surveillance [91].

Nutrition

In the area of nutrition, articles employing crowdsourcing focused on food safety, food labelling, assessing how healthy the meals were, and identifying predictors of obesity [43-50]. Two articles used Yelp reviews to assess food safety in restaurants, one in New York City and the other in Seattle [43,44]. In New York City, the Department of Health and Mental Hygiene received data from Yelp and used computer algorithms to scan the data for probable food poisoning cases. These cases were then reviewed by a foodborne epidemiologist, and probable cases were requested for interview. Three outbreaks were discovered by the study [43]. The second article, which uses Yelp reviews for Seattle, tested whether Yelp reviews would be able to predict whether a restaurant would fail its health inspection. The authors found that the Yelp reviews were 82% accurate predictors for restaurants that would fail health inspections [44].

As food packaging is often complicated and difficult to understand for consumers, Dunford and colleagues created a traffic light application that is populated through crowdsourced submissions in order to enable people in Australia to make healthier choices about their diets [45]. Their application is called “FoodSwitch Australia.” Three other articles published efforts to ensure consumers were able to make healthy decisions about the food they eat. Noronha and colleagues, Turner-McGrievy and colleagues and Moorhead and colleagues developed applications to enable crowds to rate the ‘healthiness’ of food being eaten. Noronha et al.’s application, “Platemate” had participants take a photo before and after their meals and asked the crowd to estimate the calories and nutritional composition. They found the estimates of the crowd did not differ significantly from that of experts [46]. Turner-McGrievy et al.’s “EateryApp” had the crowd rating the healthiness of photos of food after 1.5 hours of training. The authors also compared their ratings to those of experts and found a strong correlation between the ratings (r = 0.88, P < 0.001) [47]. Moorhead also used photos to have a crowd estimate the calories in food, but also developed personalised messages for prevention and management of obesity. They piloted their application with a crowd of 12 non-experts and 12 experts. In both cases, the group estimates were more accurate than any individual estimate [48].

Finally, two articles reported on using crowdsourcing to develop predictors of obesity in a statistical model [49,50]. Both articles used reddit to recruit participants and had participants initially answering questions commonly known to be predictors of obesity. Then, they posed new questions, which the group as a whole would answer. Bevelander et al. found that some participants identified examples that were not well documented in literature and suggested that those could be possible new directions for future research. The study also found that only about 7% of participants posed new questions; the remaining participants answered questions only.

Public health and environment

Crowdsourcing has been used in public health for research in the areas of tobacco control, physical activity and built environment, environmental health, to shape messaging and for public health related contests [51-58].

Patel et al. created an application to measure the prevalence of smoking in vehicles. The application had a mechanical counter to track passing vehicles, thus providing the denominator, and the person using the application would record each instance of a person smoking in the vehicle. The authors found a prevalence of 2.9% and had 66 users [51]. Two articles used crowdsourcing for point-of-sale tobacco (POS) monitoring. Ilakkuvan and colleagues used AMT for image annotation. The authors were testing image annotation, rather than monitoring per se, and found that image annotation improved when Turkers were provided with microtasks and given the option to zoom in on photos [52]. The final article examining POS tobacco use physically deployed their crowd to the locations under surveillance. The crowd photographed the stores and answered questions. The authors found high agreement on what the stores sold, but poor agreement on promotions; however, authors noted this could be due to the crowdsourced visitors visiting the stores at different times than the trained experts, when the stores simply had different promotions [53].

Hipp and colleagues used webcams in the United States to capture changes after altering the built environment, for example, by adding a crosswalk or adding a bike lane. The authors then used AMT to annotate the webcam images for the pedestrian and cyclist traffic before and after the changes to the built environment. This was done in order to determine the impact the alteration has had on people’s behaviours. The authors found that AMT was a successful method for image annotation and that there were measurable changes after the built environment was altered [54]. Castell et al. used physical crowdsourcing to explore environmental health, and they hoped to have an impact on the health, too. The authors created CITI-SENSE and CITI-SENSE-MOB, two applications which used sensors that were attached to mobile phones that obtained individual-level data on air quality pollution that were GPS-tagged. These data provided users with a map of where pollution was the worst, so they could avoid it [55].

Turner and colleagues employed AMT to test multilingual promotional dental materials (in English and Spanish). They were able to receive feedback from almost 400 Turkers in less than 2 weeks and received especially valuable feedback from the Spanish-speaking Turkers regarding the cultural appropriateness and dialects of their messaging [56]. An additional public health messaging project that used crowdsourcing was CrowdoutAIDS, which was a large campaign by the UNAIDS Secretariat that had included youth, both online and offline, from 79 countries in shaping UNAIDS messaging and their priorities for sexual health [57].

Another way crowdsourcing has been used in public health is through contests to draw attention to important causes and promote public engagement. In Philadelphia, a large crowdsourced competition was deployed to bring attention to heart disease through mapping automated external defibrillators (AEDs) [19,20]. Through this contest, 1429 AEDs were located through 313 submissions. Data were validated through GPS, door-to-door and photo verifications. The authors were pleasantly surprised that, despite being a social-media based exercise, many older participants contributed. Another exercise that used a crowdsourcing contest to draw attention to a public health concern asked participants to design and develop videos to promote HIV testing in China. Seven eligible videos were submitted in an eight-week time period [58].

Education

Bow and colleagues reported using crowdsourcing with their pre-clinical medicine students at Johns Hopkins University to create flashcards to improve studying. The professors had questions and respective answers after lectures available on GoogleDrive, which was shared with the class. Students were able to add to questions, add new questions and add to answers. The questions were changed to flashcards using Java, to assist students as study aids. The students’ grades improved in comparison with students from the previous year [59]. Another study stated that crowdsourcing could help educators grade students’ assignments more fairly, but did not elaborate [92].

Genetics

In genetics, crowdsourcing has been used for challenges in genetic research, for matching genes to mutations, to identify novel hypotheses through crowdsourced data, and it was proposed as a solution for incidental findings in genomics studies [60-65].

Sage and Dream are two organisations that heavily use crowdsourcing in genetics research. Plenge et al. report on one of their challenges, which was to develop genetic predictors of a response to immunosuppressive therapy in rheumatoid arthritis, using genome-wide association study (GWAS) data. The challenge was team-based, collaborative and open to both public and private contributors. The team that produced the best predictive model would win the challenge [60]. Ewing and colleagues reported on another Dream challenge, which aimed to identify somatic mutations in cancer genomes. The data for this challenge was distributed via GoogleCloud, and the challenge employed a Leaderboard for a competitive aspect. There were 248 submissions by 21 teams in 157 days [61].

Dizeez is a human gene annotation game where players guess which gene causes which disease, out of four options. The game aims to identify gene-disease associations that are known but not present in structured annotation databases. Authors took the genes selected to be associated with a particular disease by many players for further investigation. The authors note that, unfortunately, when players suggest potentially novel associations, they are ‘punished’ by the game. However, the game was able to successfully identify gene-disease associations [62]. Another gene matching game, EntrezGene, used AMT to match genes from papers and abstracts to their EntrezGene identifier. Turkers were asked to judge whether the gene is associated with a mutation and 20% of the tasks were controls. In the authors’ report, there were problems with the study giving false information to the Turkers but after adjusting for this, the Turkers achieved 82.3% precision [63].

Kido and Swan reported on using crowdsourced data from MyQuantifiedSelf, which is a personal genomics company, in order to test their novel hypothesis, that some genetic profiles would exhibit a natural capacity for social intelligence. The authors combined citizen science as a form of crowdsourcing and the daily tracking of their “MyFinder” application in order to explore the role of genomics (OXTR gene mutations) on personalities. The authors found that their hypothesis did not appear to be valid, that the increase in the frequency of the G allele did not lead to increased optimism; however, an increase in the frequency of the A allele seemed to lead to decreased optimism. The authors stated that further analyses with larger sample sizes needed to be conducted in order to confirm their hypotheses [64].

Finally, as GWAS studies are becoming more prevalent and the probability of incidental findings becomes more likely, Krantz and Berg suggest crowdsourcing as a solution to managing incidental findings. The authors proposed a ‘binning system’ in genetic studies that employs crowdsourcing, such as through AMT, to separate incidental findings into bins based on their current risk to the individual. For example, clinically actionable results would be placed in “bin 1”, while results with a high clinical validity with no actionability would be in “bin 2” and those with no clinical significance would be in “bin 3.” The results in bin 2 would need to be re-scored as new advances in medicine are made [65].

Psychology

There were four articles published in psychology [66-69]. The first explored the viability of using crowdsourcing, specifically via AMT, for studying mental health issues. The authors assessed misrepresentation, inconsistencies in basic demographic information and clinical symptom reporting. The authors found that AMT workers' mental health mirrors that of the general population, other than social anxiety and satisfaction of life scales. AMT workers have an increased social anxiety, which mirrors other internet-based studies, and lower satisfaction with life scales [66]. A second study used AMT to diagnose major depressive disorder (MDD). It asked users for the Twitter account and data-mined their accounts for one-year prior in order to measure user engagement, egocentric social graph, linguistic style, depressive language use, and their mention of using antidepressants. The control was a standard user class. The authors found lower social activity, greater negative emotion, higher self-attentional focus, increased relational and medicinal concerns and heightened expression of religious thoughts among the MDD group [67].

The third study explored the potential for crowdsourcing to adequately respond to discussions in an autism support group by outsourcing the help questions to AMT and having the responses rated against the in-group answers. The AMT answers were rated as more helpful and AMT was seen as a quick way to provide direct and informal emotional support and to broaden perspectives of the autistic community [68]. The final study combined crowdsourcing and data mining, using AMT to generate alternative Life Quality Statements to enable comprehensive data mining for these statements from Twitter [69].

General medicine/other

Remaining examples of crowdsourcing were in areas of oncology, medical text, various aspects of drugs, including curation, severity of reactions and even black market prices, examples of physical crowdsourcing and other miscellaneous examples [70-76,78-87,93].

Love and colleagues used crowdsourcing to explore women’s questions regarding collateral damage from breast cancer treatment. Advocacy organisations collected responses to the questions posed. Many of the women who submitted questions complained of fatigue, memory loss, numbness, anxiety or depression [70]. Another study used crowdsourcing to survey the knowledge of the population about ovarian cancer, using breast cancer as a control. The authors used AMT as a platform and found that 56% of those surveyed reported no knowledge of ovarian cancer [71]. A third study employed a crowdsourcing game, ‘the Cure,’ to predict breast cancer survival in order to improve prognostic indicators of breast cancer. Approximately 60% of players were not knowledgeable about breast cancer. The authors reported that both the expert and the all (expert and non-expert combined) set ‘significantly enriched’ knowledge, but that the non-expert group alone did not. The responses of the expert group performed well in a Sage contest as well [72].

As a solution to challenges with health literacy and poor comprehension and adherence to text-based medical instructions, Yu and colleagues used AMT to test picture-based medical instructions using existing pictograms from the Internet. The results showed that semantic concepts were difficult to represent in pictures (ie, take with additional water or take in the AM). Turkers with higher levels of education performed better. Yu et al. suggested future studies should use tailor-designed pictograms and explore the interplay between education and responses as well as the role of culture and different countries in ability to comprehend the pictograms [73]. Seifert and colleagues describe conceptual framework for using AMT with the aim to extract facts from interdisciplinary scientific literature in order to help researchers keep track of overlapping topics across disciplines [74]. Another article described the use of gamification to enable both experts and laypersons to complete text extraction, term categorisation, relation extraction and relation categorisation in games called “Dr. Detective” and “Crowd Watson.” These games employed crowdsourcing through natural language processing (NLP) and relied on inter-annotator agreement [75]. A third example of NLP using crowdsourcing was Parry et al.’s SNOMED CT (Systemised Nomenclature of Medicine-Clinical Terms) which used semantic tagging, similar to Medical Subject Headings (MeSH) terms, in order to help clinicians code free-text documents. SNOMED CT used crowdsourcing, along with a learning algorithm, to imply membership in a particular ontology and the degree of ‘fuzziness’ the member believes the ontology to be; users then rate how related it is [76]. Crowdsourcing was also used to investigate errors in SNOWMED CT, and when compared to experts, their results were comparable [77].

Crowdsourcing has been used in drug research in various ways, including through NLP, ranking the severity of reactions, curating drug indications and to identify black market prices for drugs. Zhai et al. used Crowdflower, which is a crowdsourcing platform similar to AMT, in order to have a crowd perform NLP tasks that identify medication types, names and link these to their attributes. The authors found that there was high agreement between the crowd’s NLP and expert-generated NLP (0.87 for medication names, 0.73 for medication types and 0.90 for linking medication to attributes). There were no significant differences between crowdsourced NLPs and experts after developing a “trust” threshold in the analysis, where the Turker meets a threshold to become “trusted” [78]. Gottlieb and colleagues used AMT to rank the severity of almost 3000 adverse drug reactions (ADRs). The authors found that the ADRs ranked most severe were more correlated with death (r = 0.53) [79]. Khare et al. aimed to use AMT to create a database of unique drug indications, by providing microtasks to Turkers, asking them to differentiate between whether a drug is indicated for a particular condition or the condition is listed for another reason (ie, it is a risk factor, side effect, contraindication, etc.). The crowd achieved over 90% accuracy and over 96% accuracy in identifying drug indications through majority voting [93]. Finally, Dasgupta et al. described the use of crowdsourcing to identify black market prices for opioids through StreetRx. With StreetRx, people were able to anonymously post where, when and the prices that either they or someone they know purchased street drugs for. People were able to access the website without posting, thus drug users were able to see if they were buying their drugs at a fair price. Drug users were motivated to post on the website, as it could prevent them from overpaying for their drugs. The researchers compared the prices on StreetRx to prices reported by law enforcement officials and from a ‘dark web’ website, Silk Road. The researchers found no significant differences, other than for morphine [80].

Three articles reported on physical instances of crowdsourcing. Maki and Cohnstaedt reported on using crowdsourcing to collect physical samples of mosquitos from a mostly trained crowd. Through crowdsourcing, the authors were able to collect mosquitos in “geographically vital, hard-to-access locations” and achieved a 91% response rate [81]. McInerney and colleagues reported on a pilot to employ crowdsourcing and Bayesian modelling to deliver items in low- and middle-income countries. The authors described using cell towers during texts and calls to predict temporality, assessing the number of people to achieve geographic coverage and the feasibility to deliver to rural locations and briefly mention a pilot. Despite an in-depth description of the theory behind their crowdsourcing model and the plans for their pilot, the authors hardly discussed their pilot, except to say that it did not perform well and there were delays in all areas (urban and rural) [82]. Finally, Maier-Hein et al. suggested using crowdsourcing for minimally-invasive surgery that required establishing correspondences and was typically done by a medical expert. The authors found that crowdsourcing was comparable to medical experts and could be used to train algorithms [83].

Similar to the psychology studies which tested whether crowdsourcing could be used to sample a population for psychological studies, Behrend et al. explored whether crowdsourcing would be an appropriate population for organisational research, comparing the demographics of a crowd from AMT to a university population. The authors found that the crowdsourcing population was more diverse professionally, and constituted an attractive pool for organisational research [84].

Carlson described a clinical trial protocol that has been reviewed not only by peers, but by patients, using crowdsourcing. He asserted that the clinical trial protocol was faster, and that using the intellectual capacity of patients in addition to researchers would enable the trial to attract participants [85]. Another author advocated for including the general population in health research, but in shaping their research priorities. Villaroel gave an example of a research priority setting exercise that was done in India using AMT. In this exercise, there were significant differences in priorities of those who did and did not vote in the last election, leading the author to assert that elected officials may not have the entire populations’ interests at heart [86]. Finally, Meisel et al. suggested an innovative solution using crowdsourcing to combat overcharging at US hospitals. As US hospitals often charge different rates for insured and non-insured customers and can charge exorbitant rates for simple procedures, such as US$ 55 000 for an appendectomy, the authors suggested creating a database where consumers post their hospital bills. Membership in this database could be free once the user has posted one bill. This would help maintain accountability and transparency among the hospitals [87].