In 2017, ICF conducted final evaluations of each of the six RAcE project areas to determine whether the programme goal of increased coverage of diagnostic, treatment, and referral services for malaria, pneumonia, and diarrhea to decrease overall child mortality was reached. One of the main evaluation objectives was to estimate the reduction in childhood mortality, and number of under-five lives saved over the project period in the RAcE project areas.

We used the Lives Saved Tool (LiST) to estimate the impact of changes in coverage of treatment for the three illnesses targeted by iCCM – malaria, pneumonia, and diarrhea – while accounting for other changes in maternal and child health interventions in each RAcE project area.

LiST is computer-based software for modeling maternal and child mortality that runs through Spectrum. Spectrum is a suite of eleven integrated modules built and maintained by Avenir Health under guidance of a team at Johns Hopkins Bloomberg School of Public Health’s Institute for International Programs [6-8]. Included in the Spectrum suite of modules are DemProj (demography), AIM (AIDS impact model), FamPlan (family planning), and LiST (child survival). LiST estimates the impact of scaling up of maternal, newborn, and child health and nutrition intervention coverage. Specifically, LiST estimates all-cause child mortality and the number of child lives saved by intervention and by cause. To produce these estimates, LiST incorporates detailed demographic information through automated linkages with the DemProj and FamPlan modules; country-specific cause of death information for newborns, children under five, and mothers; health and nutrition status, including through linkage with the AIM module; coverage of key health interventions; and effectiveness measures for each intervention determined by scientific review [9-11]. LiST Version 5.55 was used for this study.

We created six LiST models, one for each RAcE project area. Each LiST model was built from the country’s DemProj data and national mortality profiles in Spectrum. The under-five cause of death information for each country, which includes the three iCCM focus illnesses, is shown in Table 2.

Using LiST’s subnational projection module, we adjusted each model to reflect the RAcE project area population, baseline subnational mortality rates (as available), and baseline intervention coverage estimates. This adjustment relies on a comparison between national (default data in Spectrum modules) and subnational data [12]. To accurately make this comparative adjustment, national data must be comparable to subnational data. If data were from the same source, then the data were comparable, and the subnational data point was input into the model. If subnational data were from a different data source than the source of national data in LiST, we adjusted these by creating a ratio between the subnational coverage rate and the national coverage rate, and then applying this ratio to the national coverage rate given in the subnational LiST module [12].

We used data from RAcE baseline and endline household surveys, Demographic and Health Surveys (DHS), WHO-UNICEF, and routine health service data as inputs for each LiST model.

Baseline years were set to 2013 for the DRC, Mozambique, Niger, and the two Nigeria models. For Malawi, the baseline year was set to 2010 in concurrence with the available baseline DHS. The subnational baseline files were used to create the full models in LiST.

Data for treatment of pneumonia with amoxicillin, treatment of fever with ACT within 48 hours, treatment of diarrhea with ORS, and treatment of diarrhea with zinc from the RAcE baseline and endline household surveys were input into each LiST model for baseline and endline data points, respectively. For Malawi, 2010 DHS data for treatment of pneumonia with amoxicillin, fever with ACT, and diarrhea with ORS and zinc were held constant to 2013 values, and 2013 RAcE baseline household survey data points were input for 2013. iCCM intervention coverage modeled is summarized in Table 3.

Baseline and endline data points for other maternal, newborn, and child health (MNCH) interventions input into the model were sub-national DHS data, where available. The lowest administrative level (eg, state, province, or district) of DHS data available comprising the project area was used. WHO and UNICEF estimates of immunization coverage were used in all countries except Malawi, for which DHS estimates were used. No large-scale national-level survey data for 2016 were available for the DRC, Mozambique, Niger, and Nigeria at the time of the RAcE final evaluation. Routine health service data for Mozambique project areas in 2016 were used for that model’s endline; if data were unavailable, values were held constant. For the DRC and Nigeria, we projected 2016 estimates for MNCH indicators by applying linear interpolation to continue the trend as measured by DHS for the time period preceding the 2013 baseline. For Niger, available routine health service data were used for 2016 and linear interpolation of DHS data was used to estimate other indicators values. Thus, the models assumed that the same trends measured for the preceding time period continued through 2016.

Values for all indicators were linearly interpolated from 2013 to 2016 in each model, except where otherwise noted. If projected values were over 100 percent, these were determined unrealistic and consequently baseline values were held constant. Due to the change in Malawi’s iCCM treatment policy, from presumptive treatment to confirmed treatment of malaria, the percentage of cases of fever treated with ACT within 48 hours measured at baseline was held constant in the model to avoid an artificial decrease in treatment coverage resulting in lives lost. A summary of data sources for the LiST models is provided in Table 4.The data, and their sources, used to create each project’s LiST model are detailed in Tables S1-S6 in the Online Supplementary Document.

Each LiST model yielded three outputs: 1) Under-five mortality rates for each year, to estimate the change in all-cause child mortality from baseline to endline (estimated impact); 2) Number of lives saved per year, among children under five years of age; and 3) Number of lives saved per year by intervention, including malaria treatment, pneumonia treatment, diarrhea treatment with zinc, and diarrhea treatment with ORS. The number of lives saved by malaria treatment, pneumonia treatment, and diarrhea treatment in each RAcE project area model were then adjusted proportionally to the percentage of cases treated by CHWs as measured in the respective RAcE endline household surveys to provide an estimate of the lives saved due to iCCM services in the project areas. Because the LiST model uses the coverage estimates for the baseline year to estimate lives saved for the model’s projection period, the estimated lives saved for the baseline year is, by default, zero.

Under-five mortality declined in all six RAcE sites over the course of the programme, with an average decline of 10 percent. An estimated 6200 under-five lives were saved by malaria, pneumonia, and diarrhea treatment in the DRC, Malawi, Niger, and Nigeria. Approximately 4940 (75 percent) of under-five lives were saved by treatment provided by CHWs. This total excludes Mozambique, where there were no estimated under-five lives saved. The total estimated lives saved – lives saved by pneumonia, diarrhea, and malaria treatment in – each project area are shown in Table 5. The proportion of estimated lives saved by iCCM treatment from CHWs in each RAcE project area are shown in Table 6. Estimates of lives saved in each project area by year and by treatment intervention are provided in Tables S7-S12 in the Online Supplementary Document.

In the DRC, there was an estimated 15.2 percent decrease in under-five mortality over the course of the project. An estimated total of 1820 deaths were prevented (lives saved) by pneumonia, diarrhea, and malaria treatment interventions in children under 5 years of age over the course of the project. Based on the coverage of treatment by CHWs, an estimated 1730 (95 percent) of these lives were saved due to treatment provided by CHWs.

In Malawi, there was an estimated 4.6 percent decrease in under-five mortality over the course of the project. An estimated 570 deaths were prevented by pneumonia and diarrhea treatment interventions in children under 5 years of age over the course of the project. There were no estimated deaths prevented by malaria treatment. Based on the coverage of treatment by CHWs, an estimated 220 (38 percent) of these lives were saved due to treatment provided by CHWs.

In Mozambique, there was an estimated 0.2 percent decrease in under-five mortality over the course of the project. An estimated 100 child lives were lost in RAcE project areas over the project period. This estimate accounts for significant numbers of lives saved by increases in some interventions and lives lost due to decreases in coverage of others. An estimated 250 lives were saved due to the increase in diarrhea treatment coverage of zinc, and 20 lives due to the increase in ORS treatment coverage. An estimated 910 lives were lost due to a decrease in treatment coverage of pneumonia with oral antibiotics, and 1680 due to a decrease in treatment coverage of malaria with ACTs.

In Niger, there was an estimated 12.6 percent decrease in under-five mortality over the course of the project. An estimated 1130 deaths were prevented by pneumonia, diarrhea, and malaria treatment interventions in children under five years of age over the course of the project. Based on the coverage of treatment by CHWs, an estimated 970 (86 percent) of these lives were saved by treatments provided by CHWs.

In Abia State, Nigeria there was an estimated 12.1 percent decrease in under-five mortality over the course of the project. An estimated 1400 deaths were prevented by pneumonia, diarrhea, and malaria treatment interventions in children under five years of age over the course of the project. Based on the coverage of treatment by CHWs, an estimated 970 (69 percent) of these lives were saved by treatments provided by CHWs.

In Niger State, Nigeria there was an estimated 14.5 percent decrease in under-five mortality over the course of the project. An estimated 1280 deaths were prevented by pneumonia, diarrhea, and malaria treatment interventions in children under five years of age over the course of the project. Based on the coverage of treatment by CHWs, an estimated 1100 (83 percent) of these lives were saved by treatments provided by CHWs.

Our results reflect the observed coverage changes measured in each project area and, with LiST, show estimated mortality reductions in a way that can be adjusted to explain the impact of each iCCM project amidst other ongoing health trends [13]. We found an average estimated decline in under-five mortality of 9.9 percent across the six project areas, ranging from a 0.2 percent decline in Mozambique to a 15.2 percent decline in the DRC. An estimated 1020 child lives were saved from pneumonia treatment; an estimated 2470 child lives saved from diarrhea treatment; and an estimated 380 child lives saved from malaria treatment across the six project areas. Our findings show that when adjusted to reflect the proportion of treatment coverage provided by CHWs vis-à-vis treatment coverage by all providers, a substantial number of estimated child lives were saved by CHW-provided treatment. In the DRC, approximately 95 percent of the child lives were saved by CHW-provided treatment. In Niger and Niger State, Nigeria CHW-provided treatment accounted for over 80 percent of the child lives saved, and in Abia State nearly 70 percent. In Malawi, less than 50 percent of child lives were saved by CHW-provided treatment; the percentage in Mozambique is unclear because it could not be accounted for in the model.

Previous studies have used LiST to retrospectively evaluate changes in mortality and deaths averted by high-impact child health interventions [14-17]. LiST has also been used to demonstrate decreases in mortality by NGO-facilitated projects that used community-based intervention packages to contribute to improved coverage of child health interventions [18], as well as to demonstrate the potential impact of expanding coverage of such interventions through scaling up CHW-delivered interventions at the population level [19]. An evaluation of iCCM in Burkina Faso also used LiST to model mortality changes, and, similar to our study, used program data to inform the interpretations of both changes in mortality and changes in coverage measured through household surveys [17]. Two evaluations of the Catalytic Initiative in Malawi demonstrated the impact of iCCM, one using LiST. The first used population-based survey data with results showing a significant decrease in child mortality over the period of implementation, and that the majority of deaths averted were due to interventions that could be delivered at the community level by CHWs [14]. In contrast to our analysis, this study did not examine the proportion of these interventions actually delivered by CHWs. The second drew conclusions about a decrease in mortality using national level changes in mortality estimates from DHS and Multiple Indicator Cluster Survey (MICS), but did not find evidence that these increases in intervention coverage were due to iCCM [15]. Similar to our findings, contextual information suggested that, for a number of reasons, only a small proportion of treatment of malaria, pneumonia, and diarrhea is provided by CHWs in Malawi [15]. A similar evaluation of iCCM in Uganda that utilized household survey data in LiST found that iCCM increased treatment coverage for diarrhea and fever, mitigated effects of amoxicillin stockouts for pneumonia treatment, and saved lives [16]. While that study supports the potential impact of iCCM, it did not account for changes in other MNCH interventions during the study period, as our analysis did.

Substantial increases in the iCCM intervention coverage were measured from baseline to endline in the DRC, Niger, and Nigeria project areas, with high proportions of cases treated by CHWs. Small, non-significant increases were measured in diarrhea and pneumonia treatment in Malawi project areas. Holding the percentage of cases of fever treated with ACT within 48 hours measured at baseline constant in the Malawi model, to account for the change in iCCM treatment policy and to avoid an artificial decrease in treatment coverage resulting in lives lost, resulted in the model showing no estimated impact from malaria treatment. The Malawi model likely underestimated the number of child lives saved due to iCCM intervention coverage, because it does not account for lives saved by the improvement in malaria case management following the introduction of RDTs.

With the exception of Mozambique, overall coverage of pneumonia, malaria, and diarrhea treatment increased in each RAcE project area, including measurable increases in treatments provided both at the facility level and by CHWs in communities. Unlike other RAcE project sites, there were measured decreases in overall malaria and pneumonia treatment coverage – treatment by any provider – in Mozambique. Diarrhea treatment with ORS did not significantly change, while diarrhea treatment with zinc increased significantly. Due to the decreases in pneumonia and malaria treatment coverage, the LiST model estimated that lives were lost. However, the source of treatment among pneumonia cases that received treatment shifted over time – with a larger proportion being treated by CHWs at endline; the model obscured this effect of pneumonia treatment provided by CHWs. There was not a statistically significant increase in the proportion of malaria cases treated by CHWs, likely because of widespread stockouts of malaria kits. Thus, there were likely not a significant number of lives saved due to CHW-provided malaria treatment in Mozambique. The largest decreases in mortality and numbers of lives saved (estimated impact) were in project areas in which iCCM was introduced through RAcE (the DRC, Niger, and Nigeria). Given that, in these contexts, RAcE operated in an environment with few other child health interventions that could have influenced the estimated decreases in under-five mortality, it is likely that iCCM contributed substantially to the estimated mortality decreases. In areas in which RAcE was supporting the continuation and expansion of more a mature iCCM strategy (Malawi and Mozambique), there was less estimated impact. In Mozambique, widespread stockouts due to weaknesses in the health system limited the provision of life saving treatment and consequently, we observed negligible impact on mortality reduction.

The LiST model is a valuable tool for estimating the impact of maternal and child health intervention coverage on child mortality, though not without limitations [13,20]. Direct measurement of child mortality is a time and resource intensive effort, and thus typically done every 5 to 10 years at the national level and highest subnational administrative unit by large-scale surveys such as DHS and MICS. This is a common constraint of sub-national public health programmes, and why models such as LiST can be a useful proxy for estimating impact. Child mortality was not directly measured before or after RAcE project implementation in any country, and valid mortality data were not available for the project sites before or after RAcE.

The results of the RAcE LiST models suggest that iCCM is a strategy that can save lives and measurably decrease child mortality in settings where access to health facility services is low and adequate resources, including trained and supported staff and supplies, are provided. The overall increases in child health intervention coverage in RAcE project areas demonstrate the added value of iCCM as an extension of the health system and as an equity-focused strategy.

In the context of this evidence and limited funding for child health, we advocate for continued investment in integrated child health programming. For greatest impact, iCCM should be implemented as an extension of the health system. It is not a strategy to replace health facility services, but rather to extend select, critical life-saving interventions directly to hard-to-reach communities. A successful iCCM programme should be part of a larger effort to strengthen health systems and facility services. The case of Mozambique emphasizes this point in the context of the adage ‘no drugs, no service.’ Without adequate commodities in hand, CHWs are unable to provide life-saving treatment to their communities, and risk losing community trust of their work and the programme.