Study setting

The study was implemented in 10 non-contiguous urban communities of Freetown, Western Urban District, Sierra Leone (Table 1 and Figure 1). The population of under-5 children, pregnant women, and mothers of under-5 children in these communities was determined through a participatory community-based census conducted in 2012-13. The total population of the study area in 2008 according to the national census was 166 417. The communities in the study area are primarily unplanned, informal settlements characterized by overcrowding, low-quality housing, limited access to clean water, and lack of sanitation. Flooding is common in parts of the study area, causing fatalities every year. The entire city of Freetown is densely populated, with an estimated 1.1 million inhabitants within an area of 138 square miles [9].

Figure 1.  Map of Freetown showing intervention and comparison communities.

The PCBHIS intervention was carried out in five communities that made up the intervention area. The remaining five communities served as the comparison area where the PCBHIS intervention was not implemented. Concern Worldwide, an international non-governmental organization, worked with the Ministry of Health and Sanitation (MOHS), the District Health Management Team (DHMT), the Freetown City Council (FCC), and local communities to implement a child survival project (CSP), as discussed below, as well as the OR Study.

The 2016 Human Development Index ranked Sierra Leone 179^th out of 188 countries [10]. While health outcomes have improved somewhat since a 10-year civil war ended in 2002, the country continues to be challenged by shocks: cholera epidemics in 2012 and 2013 followed by an Ebola epidemic that spanned more than two years (2014-2016). The Western Area Urban District, which includes Freetown and the study area, was one of the worst affected areas during all of these public health crises.

National health indices remain among the worst in the world. According to recent estimates by UNICEF, the maternal mortality ratio is the highest in the world (1360 maternal deaths per 100 000 live births), and the under-5 mortality rate is the fourth highest in the world (114 under-5 deaths per 1000 live births [11].

Local health system structure

Government primary health care facilities available at the sub-district level are referred to as Peripheral Health Units (PHUs). The DHMT plans, coordinates and supervises services at PHUs as well as interventions in the community such as public health campaigns. The FCC also plays a key supporting role in the delivery of community health services.

Each PHU is supported by a Health Management Committee (HMC). The HMC serves as a liaison between the PHU and the community and supports community mobilization and outreach activities from the PHU. Each of the 39 wards within the Freetown municipal area has an elected Ward Development Committee (WDC), responsible for engaging community members on general development activities. The WDC chair sits as a Councilor on the FCC. Each community in the study area covers 1-2 wards. In cases where the community covers more than one ward, both WDCs participated in the study. Roles and methods for HMCs and WDCs to act as the liaison to the community have not been well-developed. While the Government of Sierra Leone highlights the need for community engagement in recently developed policy documents, such as the Basic Package of Essential Health Services (BPEHS) 2015-2020, it focuses on CHWs to fulfil this role. The BPEHS recognizes HMCs and WDCs as community actors but does not outline roles or the ways in which they should fit into the health system [12].

The national community health worker program

The MOHS finalised a national community health worker (CHW) policy in 2012 and began operationalising it in early 2014. With the Ebola outbreak also emerging in 2014, the rollout of the CHW policy and the extent of government ownership of the CHW program were impaired. During the period of the OR study, Concern Worldwide supported the introduction of a CHW program across the full study area in line with the 2012 CHW policy, tailored to the urban context and approved by the Western Area DHMT. These CHWs were involved only in health promotion, referral, and surveillance; they did not provide any curative treatments. Under the newly developed National CHW Policy 2016-2020, community structures are to have a role in reviewing community-level health data, but no mechanism for this review has yet been determined [13].

The Child Survival Project

The OR Study described below was embedded within a USAID-supported child survival project (CSP), known locally as Al Pikin Fo Liv (All Children Should Live), with co-funding from Irish Aid. The CSP was implemented in all of the 10 study communities listed in Table 1 from October 2011 to June 2017 (see also Figure 1).

1325 volunteer CHWs were trained by Concern Worldwide CSP staff and the Western Area DHMT using the Sierra Leone MOHS 2012 National CHW Program training materials. Introductory sessions on the CHW Program were held for community leaders by CSP staff and the Western Area DHMT. 606 male and 709 female community members were then selected by community leaders to be trained as CHWs as per National CHW Program selection criteria. About two-thirds of those trained were between 18 and 34 years of age, and 21% were 35-54 years old. Only 4% of CHWs were over 55 years of age. Almost 60% of those trained had completed some secondary school and 5% classified themselves as non-literate. The number of CHWs trained was calculated using a CSP- and community-led census and policy-mandated population-to-CHW ratios. 106 CHWs were selected as Peer Supervisors by CSP together with community leaders, based on their performance during initial CHW training. Peer Supervisors were given additional training by the CSP, and assigned 8-12 CHWs to supervise. These Peer Supervisors were attached to the specific area, or “zone” of the community where their supervisees operated. At least one HMC and one WDC member from the same zone provided oversight and assistance to the Peer Supervisor. Using the participatory census results, CHWs were assigned 25 households, generally in close proximity to their own homes, to visit monthly and disseminate health messages, check for danger signs of illness, and collect vital event and morbidity data using MOHS registers.

The CHW intervention was designed so that every household with a pregnant woman or under-5 child in the 10 study communities would receive a visit at least once per month. CHWs made home visits and collected household-level data for 13 months (May 2014 – June 2015) before the OR intervention (the PCBHIS) began in July 2015. The CHW intervention and OR activities then ran concurrently for a subsequent 20 months (July 2015 – March 2017). See Figure 2 for a timeline of the CSP and the OR Study. Appendix S1 Table S1 in Online Supplementary Document contains the objectives and activities of the CSP.

Figure 2.  Timeline of key contextual events and key activities of the child survival project, the operations research study, and data collection.

Study design

The OR Study was developed in 2011 and updated in 2012 and 2013 through a collaborative process between the Johns Hopkins University and Concern Worldwide. The objectives of the study were 1) to assess the extent to which the PCBHIS facilitated local community structures to use data to plan and implement actions for improving maternal, neonatal, and child health (MNCH) and 2) to assess the extent to which this contributed to improved community-level MNCH outcomes. The study was a cluster-randomized controlled trial, with communities randomly selected to either the intervention or comparison area. The OR Study intervention consisted of two activities in addition to the CSP activities:

Table 2 summarizes the activities that took place in the intervention and comparison areas.

The source of the data for the PCBHIS was the monthly CHW reports. The original study design, developed in 2011, called for CHWs to collect morbidity and mortality data only in the intervention area, using data collection materials designed by the study through its formative research. However, the MOHS Policy for Community Health Workers in 2012 required that morbidity and mortality data be collected by CHWs nationally and thus was collected across the entire study area, using standardized MOHS materials. As a result, the initially envisioned OR Study intervention had to be altered, and the OR Study was unable to use its own forms for registering morbidity and mortality in the intervention area communities. Numerous other challenges arose that required modification of the original study design. These are described in Table S1 and S2 Appendix S2 in Online Supplementary Document.

The principal operations research study intervention: Community Health Data Review Meetings

CHDR Meetings were designed to be held in each of the five intervention communities every two months during the period of OR implementation. The purpose of these meetings was to support the HMC, WDC, PHU staff, and CHW Peer Supervisors of that community to engage with health data that was collected by the CHWs in that specific intervention community, and determine actions in response. All HMC members, WDC members, and Peer Supervisors for the community and the PHU In-charge were invited to each meeting.

Each community had 5-12 CHW Peer Supervisors depending on the community population. All communities had one HMC consisting of 15 members. Photos of CHDR meetings are included in Appendix S3 in Online Supplementary Document.

Generally, data for the preceding 4-6 months were reviewed in each CHDR meeting. The HMC Chair for the community chaired the meeting. Sessions were facilitated by OR Study staff, but, over time, participants increasingly took the lead in reviewing the data and reporting the findings. OR Study staff analyzed CHW-collected data with the CSP staff attached to that community prior to the meeting and determined topics and data to present in CHDRs. OR Study staff prepared simple data sheets to be used by participants, and participants used them to draw and interpret bar charts in front of the group, as shown in in Appendix S3 in Online Supplementary Document, Figures S1 and S2. Records were kept of discussion topics.

Six months after the initiation of CHDR meetings, OR staff began to share information from verbal autopsies on deaths of children who died before reaching five years of age in the meetings. The discussion focused on the caretaker’s account of the events leading up to the death of the child and the efforts undertaken by the family to obtain treatment. A companion paper [14] describes the community-collaborative verbal autopsy process and the results obtained.

Following the review of data, CHDR participants developed action points as seen in Appendix S3 Figure S2 in Online Supplementary Document. Action points were documented during the meeting on flipchart paper which the HMC Chairman kept after the meeting. Action points from previous meetings were reviewed in subsequent meetings, and discussions held on the extent to which actions had been completed.

Indicators of the capacity of community committees to engage with the local health system: the Health Institution Capacity Assessment Process (HICAP)

The capacity of the community to engage with the local health system was assessed with the Health Institution Capacity Assessment Process (HICAP). This is a participatory, self-assessment tool used to assess local organizational capacity to perform their functions. The tool and the process for using it were developed by Concern Worldwide and tailored for use by the HMCs and WDCs in the study area (see Appendix S5 in Online Supplementary Document for a copy of this instrument). HMCs and WDCs used consensus decision-making to determine their structures’ score for 21 indicators. Each indicator has five pre-determined criteria and associated values. Six HICAP assessments were conducted with each of the ten HMCs and ten WDCs over a 37-month period, taking place at intervals of 6-9 months. Assessments were conducted through day-long workshops in which the HMCs and WDCs came together in each community. Each HMC and WDC determined its own score.

Following each assessment, indicator values were recorded and calculated as percentages, representing scores out of a maximum score of 5. For analysis purposes, indicator values determined in the three assessments that took place during the baseline period were averaged and considered as baseline values, and those scores determined from assessments following this period were considered as post-baseline values (see Figure 2 for timing of the baseline and post-baseline periods). Further analyses were then conducted to determine the difference in differences between the intervention and comparison areas from the baseline to the post-baseline periods.

Indicators of effective health system functioning: functionality of the CHW program as determined by rate of CHW reporting

Over a 34-month period, CHWs submitted monthly registers to their Peer Supervisor, and Peer Supervisors submitted monthly summary forms to the CSP and OR Study staffs. This information was summarized for each community and entered into an Excel database. Included were the number of CHWs and Peer Supervisors submitting reports and the number of households reached by CHWs, as reported by the CHWs in their own reports.

The 13 months preceding the initiation of CHDRs were considered as the baseline period, and baseline averages of each indicator were calculated for the intervention and comparison areas. The 21-month period following the initiation of CHDRs was considered the post-baseline period. As with baseline calculations, averages of each indicator were calculated for intervention and comparison areas during the post-baseline period. A difference-in- differences analysis was then conducted between the intervention and comparison areas from baseline to post-baseline.

Indicators of effective health system functioning: health system utilization and household behaviors as determined from household surveys

A baseline knowledge, practice and coverage (KPC) survey collected data from both intervention and comparison areas in May 2015 (the month prior to initiation of the OR intervention). See Appendix S4, Tables S3 and S4 Online Supplementary Document for indicators measured. An endline survey measured the same indicators following the end of the OR intervention in April 2017. The survey used a structured questionnaire with a core module directed at mothers of children 0-23 months of age as well as a sub-module directed at mothers of children 0-23 months who had had symptoms of diarrhea, pneumonia or malaria within the two weeks prior to the survey. At endline, an additional module was added on frequency and quality of CHW home visits. See Appendix S4, Table S1 in Online Supplementary Document for indicators. 0 clusters in the intervention and comparison areas each were selected at random using a probability proportional to size methodology based on population projections from the most recent Government of Sierra Leone census. Ten interviews were conducted in each cluster through a random selection of households and a random selection of the respondent within the household. The baseline KPC survey had 299 respondents in the intervention area and 300 in the comparison area. The endline KPC survey had 379 respondents in the intervention area and 413 in the comparison area. The sampling methodology provided a 95% confidence interval of ±8% or less for the prevalence of indicators measured in the population.

Data were collected using digital data collection devices and uploaded to an electronic database. A difference-in-differences (DID) analysis was carried out to assess the degree to which changes in the intervention area were different from those in the comparison area. Results from the KPC survey concerning CHW visitation, obtained only at the time of the endline survey, were compared between the intervention and comparison areas.

The statistical significance of differences in simple comparisons between baseline and endline values of indicators or between the intervention and comparison areas were calculated using WINPEPI version 11.65 (Brixton Health, London, UK). The statistical significance of the DID estimate was assessed using a z-test based on the variances of its four component proportions.

Ethical review

The OR Study protocol was approved by the Sierra Leone Ethics and Scientific Review Committee and was declared exempt from human subjects review by the Johns Hopkins University School of Public Health Ethical Review Board prior to study implementation. An Operations Research Steering Committee was established that consisted of Dr Perry as chairperson, national and local government representatives, UNICEF representatives, and Concern Worldwide representatives.

Community Health Data Review Meetings

Twenty-nine meetings were held over a 20-month period from July 2015 to April 2017. Each community in the intervention area was involved in five to seven meetings, and one community held an initial pilot meeting. The number of participants in meetings ranged from approximately 30 to 50 people. Generally, the same HMC members, WDC members, and Peer Supervisors attended each meeting. PHU In-Charges rarely attended meetings, but generally sent the same representative to each meeting. PHU staff attendance was not strong at the beginning of the process, but HMC members engaged PHU staff and attendance improved.

All meetings included a review and discussion of the number of CHWs reporting and number of Peer Supervisors reporting. Almost all meetings included a review of the number of households reached by CHWs during previous months, whether as an overall number compared to the target or as the average households reached per CHW compared to their monthly target of 25. Additional data that were reviewed and discussed, in approximate order of frequency, are shown in Table 3. All data presented were specific to the community except for the data from the two communities of Grey Bush and Kingtom since they were engaged in joint meetings.

Discussions of data on CHW and Peer Supervisor reporting rates and on coverage of the community by CHWs centered on:

As detailed in Table 3, the data presented at the CHDR meetings and during the ensuing discussions focused overwhelmingly on the reporting rates of CHWs and the coverage of homes with visits from CHWs. It became increasingly clear that there was substantial under-reporting of morbidity data and vital events, even among those CHWs who were submitting monthly reports. The low level of completeness and quality of morbidity and mortality data led the OR study staff to focus discussions on data quality and on actions to improve quality and completeness, rather than on a review of health trends. While the OR study design had intended for CHDRs to focus on health trends and actions to address these trends, the data quality issues needed to be resolved first. As the CHDR meeting activity proceeded, the OR Study staff gradually began presenting data at a more granular level, often through group work between the Peer Supervisor, HMC members and WDC members covering a specific zone of the community.

For example, this small group would discuss each CHW who was responsible for working in their specific area and agree on strategies for motivating those that were not covering all households or were not reporting. This adjustment reduced the abstractness of the information, increased accountability, and enabled participants to have more practical discussions about data quality. CHW motivation as well as quality and completeness of CHW monthly reports were the most frequent topics of discussion in response to CHW-gathered data. A frequently agreed action was for community leaders to hold group or individual meetings with low-performing CHWs, or between CHWs and their Peer Supervisor to resolve disputes which had arisen. CHWs and Peer Supervisors expressed that simple recognition by community leaders was a strong motivating factor.

One Peer Supervisor said:

Another Peer Supervisor said:

It is particularly interesting to note that the Peer Supervisor felt more able to review his own CHWs’ data during the CHDRs, and perhaps did not feel able to do this on his/her own. Project staff also expressed that the CHDRs assisted them in their own work by helping them to identify problems with the CHW data.

Presentations of verbal autopsy data generated active discussions about household level behaviors and health system capabilities. For example, participants sought clarity from PHU staff on clinic hours of operation and actions to take if no staff are found at the PHU. While community leaders may have been somewhat aware of some issues arising in CHDR meetings, the meetings gave them an opportunity to see them more concretely and to discuss possible solutions. Since community leaders face many competing priorities, including their own livelihoods, without this forum it is unlikely that these issues would have been discussed.

Community structure capacity to engage with the local health system

As described in the Methods section, we assessed the capacity of community-level committees to engage with the local health system through the use of the Health Institution Capacity Assessment Process (HICAP). Using the HICAP, data were collected during baseline and post-baseline periods on six aspects of the committees’ ability to fulfil their roles: (1) Support to CHW Peer Supervisors and CHWs; (2) Use of health information in planning; (3) Regular and systematic supervision of CHW Peer Supervisors; (4) Monthly and quarterly meetings in which supervision is an agenda item; (5) The process of reviewing and contributing to CHW activity plans; (6) Community perception of the committee. Results of the HICAP analysis for these six indicators showed that the difference in differences (DID) in HMC capacity between intervention and comparison areas were not statistically significant, as shown in Table 4. Table 5 indicates that the same analysis conducted on WDC scores showed a statistically significant difference for only one item: “Monthly and quarterly meetings in which CHW/Peer Supervisor activities is an agenda item” increased from 31.3% to 47.5% in the intervention area and decreased from 37.3% to 30.7% in the comparison area (P = 0.035 for the DID).

CHW reporting rates

Over the 34 months of the OR Study period CHWs submitted a total of 14 838-monthly reports and Peer Supervisors submitted 2409 reports (from both the intervention and comparison areas).We think it is reasonable to consider the submission of a monthly report as a proxy for CHW activity level, but we do not have any information on how active CHWs were that did not submit reports. As shown in Table 6 and Appendix S4, Table S2 in Online Supplementary Document, during the post-baseline period CHWs were significantly more active in the intervention area than in the comparison area. Monthly CHW reporting data shows that a greater percentage of intervention area CHWs and Peer Supervisors were active as compared those in the comparison area. Compared to baseline levels, there was a net increase of 13.5 percentage points (P = 0.003) in reporting rates for CHWs and an increase of 8.5 percentage points for Peer Supervisors (not statistically significant). In addition, there was a net gain of 14.2 percentage points (P = 0.000) in the percentage of targeted households visited in the intervention area relative to the comparison area. These three indicators, which served as proxies for CHW functionality, were the only indicators which were discussed with frequency in the CHDRs, as shown in Table 6.

CHW home visitation quality as perceived by household members

As part of the endline household survey, respondents in both the intervention and comparison areas were asked about their experiences regarding CHW home visits. As shown in Table 7 and Appendix 4. Table 2, these measures of CHW functionality were more favorable in the intervention area for 8 out of the 11 measures. The difference between the intervention and comparison area for 2 of these indicators was statistically significant, and that for a third measure approached statistical significance. None of the 3 differences that favored the comparison area were statistically significant. Baseline levels of these indicators were not measured, so DID analyses were not possible.

Changes in household health-related practices

As described in the methods section, a knowledge, practice, and coverage (KPC) survey was conducted at baseline (May 2015) and endline (April 2017). A DID analysis was carried out for household behavior indicators that could have been influenced by CHWs. The DID was more favorable in the intervention for 5 out of 7 household behavior indicators, and 2 out of the 5 were statistically significant. Neither of the differences favoring the comparison area were statistically significant (Table 5 and Appendix S4, Table S3 in Online Supplementary Document).

Changes in care-seeking behaviors

Six indicators of care-seeking behaviors promoted by CHWs were also measured in the KPC surveys described above. As Table S4 of Appendix S4 in Online Supplementary Document indicate, 4 out of these 6 DID measures are more favorable in the intervention area, 1 is statistically significant, and 1 approaches statistical significance. The 2 indicators for which the DID measures favoured the comparison area were not statistically significant.

For all 4 categories of measures of community-based health system functioning described above, 20 of the 27 differences favour the intervention area, 7 out of the 20 are statistically significant, and 2 approach statistical significance. None of the differences favouring the comparison area are statistically significant or approached statistical significance.

Limitations and challenges

This study experienced significant challenges and limitations. Delays in implementation caused by the delays in the government’s formulation of its national CHW policy and by cholera and Ebola epidemics caused serious operational challenges for the study. Most significantly, we were not able to address original research questions on the effect of the PCBHIS on under-five mortality. The low rate of report submission from CHWs along with underreporting of vital events even for those who submitted reports made it impossible to accurately monitor changes in mortality in the intervention area, as had been planned originally. The abbreviated implementation period and the presence of a well-run child survival project in the comparison area both made it unlikely that a mortality impact in the intervention area would have occurred even if mortality rates has been measured accurately.

We had also hoped to observe stronger improvements in coverage of household-level maternal and child health behaviors and practices in the intervention area relative to the comparison area than we did, as behavior change communication was a key role of CHWs. However, the limited functionality of the CHW program, the shortened time period of intervention implementation, and the high rates of in- and out-migration in these urban communities all contributed to less than optimal health data for review at the CHDR meetings and for less than optimal capacity of the community to promote strengthened community-based services. In complex systems, such as community and social systems, effects of interventions may fit longer timelines than afforded by OR designs [15,16].

Capacity building, community engagement and strengthening of community-based health services: other reported experiences

Existing evidence suggests that community engagement can support improved functioning of government health facilities [16], reduce incidence of key childhood illnesses [17] and improve the quality of health programming [18]. Yet the literature calls for more evidence of effectiveness of community engagement [19,20].

A PCBHIS built on community-based surveillance is one approach to strategically support and engage communities to collect, analyze and act on local health data. The PCBHIS developed for this study centered on household-level epidemiological, morbidity and/or mortality data gathered by CHWs. While using health data for decision making is not a new approach (USAID led a project called Data for Decision Making for the Health Sector beginning in 1991), these activities have usually focused on higher-level health officials using data collected by paid, trained research assistants directed by a vertical national program. The limited use of the data-for-decision making approach for community-based maternal and child survival programs that has been published in the peer-reviewed literature has not undergone the level of rigorous assessment that we present here. Few studies and interventions such as ours have focused on providing access to data gathered through a local CBHIS to the members of the community from which it was gathered. Community engagement can facilitate ownership of locally collected data [21], and it can have positive impacts on health and nutrition indicators [22]. However, we have not identified any evidence for this from slum communities in low-income countries.

Integral to the functionality of the CBHIS is the quality of the surveillance data. In our study, volunteer CHWs with other responsibilities provided this data. Various studies have found that CHWs can be used effectively to collect community-based surveillance data [23], and that this data provides an important complement to data coming from health facilities, which can be incomplete since it relies only on those coming to the facilities for care [24,25].

CHW programs often have challenges with quality and coverage, particularly in volunteer programs [26,27], and CHW performance has been specified as a key element of high quality CHW programming [28]. Several studies have found that community engagement and support can lead to better CHW performance and motivation [29,30]. However, documentation of approaches and effects of community engagement on CHW performance such as those we report here remain scarce.

Sierra Leone faces a unique opportunity, with the country now stabilized after the catastrophic Ebola outbreak, with the government and donors supportive of CHW programming and community engagement, with a stronger new national CHW program, and with newly finalized guidelines for Health Management Committees to help formalize these structures and standardize their operation. Establishing a functional PCBHIS linked to the work of CHWs and including a practical, low-cost intervention such as Community Health Data Review meetings may allow Sierra Leone, and other countries with similar contexts, to enable the community and health system to work more effectively together for improved health outcomes.

Nonetheless, valuable lessons can be extracted from the implementation of this intervention and its evaluation. While CHW visitation coverage during the post-baseline period was significantly higher in the intervention area (42.7% as compared to 24.2% in the comparison area), the lack of CHW visits in more than half of the households presents obvious limits for the overall effectiveness of the CHW program. However, this does not negate the positive influence of the PCBHIS intervention itself on the functionality of the CHW program.

For practitioners and policy makers seeking to address the difficult challenge of CHW motivation through non-financial means, the CHDR meetings present a practical solution, with the added benefit of meaningful community engagement. While it is yet to be seen if the PCBHIS in this context can be used to address and influence trends in serious morbidity and mortality, our findings demonstrate that communities are able to engage with data, determine actions, and motivate volunteer CHWs to become more active.