Verbal autopsy cause of death assignment

Verbal autopsy (VA) expert algorithms are combinations of illness signs and symptoms judged by verbal autopsy researchers to be predictive of particular causes of death. The algorithms validated in the current study were based on those developed by researchers for prior VA validation studies, further consultation with additional verbal autopsy experts, and a literature review to identify illness signs and symptoms commonly associated with particular neonatal and child illnesses. The sources and algorithms themselves are provided in a recent publication [21]. We used the expert algorithms to estimate cause of death given each individual’s PHMRC VA questionnaire responses. While the PHMRC questionnaire includes close–ended questions on illness signs and symptoms, an open–ended narrative response and recording of data from medical records and death certificates available in the home, the expert algorithms are based only on the responses to close–ended questions on illness signs and symptoms.

Because the algorithms determine all contributing causes, in the event that more than one cause was identified the primary cause was chosen according to a pre–specified hierarchy. We determined the primary causes of neonatal death utilizing the same algorithms across five hierarchies for neonatal deaths that are currently in use: Arifeen et al. [22], Baqui et al. [23], Kalter et al. [21], Lawn et al. [24], and Liu et al. [25]; and the primary causes of child death (1–59 months of age) utilizing the hierarchies for this age group described by Arifeen et al. [22], Kalter et al. [21], and Liu et al. [25]. Other things being equal, estimating more causes at once will yield lower accuracy than estimating fewer causes [26]. Therefore, for neonatal deaths, we also examined a compromise hierarchy that included four cause categories in common across all five neonatal hierarchies ( Table 1 ).

Reference standard cause of death

We used the reference standard causes of death from the PHMRC study to approximate the cause of death distribution in community settings, where verbal autopsy is most relevant. Because the PHMRC study was hospital– as opposed to community based, and the cause distribution in the community and hospital may differ, we resampled from the study deaths to represent a variety of cause distributions.

We approximated three specific mortality settings with the PHMRC data: (1) communities with high under five mortality where malaria is endemic, (2) communities with high under five mortality where malaria is not endemic, and (3) communities with moderate under five mortality. We used the Child Health Epidemiology Reference Group (CHERG) definition of high under–five mortality (more than 35 deaths per 1000 live births) [4], took moderate mortality as 20 to 35 deaths per 1000, and defined malaria endemicity as greater than 5 percent of under–five deaths due to malaria. In addition to these three specific scenarios of interest, we also considered a fourth general scenario, where all cause-specific mortality fractions were randomly varied between 5% and 40%.

Estimated cause proportions of death for all countries in the world, including those where most deaths occur outside the formal health sector, are available from the WHO [4]. We used these estimated causes of neonatal and child mortality as a guide in choosing cause distributions in our scenarios of interest. To generate a possible set of verbal autopsies to represent a given death distribution in a particular mortality scenario, we selected one country at random among all those appropriate, and resampled the PHMRC questionnaire data to correspond approximately to that cause of death distribution. For neonates, we included deaths due to prematurity, birth asphyxia, congenital malformations, meningitis, pneumonia and sepsis; and for children we used deaths from HIV, diarrhea, measles, meningitis/encephalitis, malaria, pneumonia, injuries, other infectious causes, and non–infectious causes. Some causes of interest for Liu et al. [4] do not occur in the PHMRC study data, requiring that we use relative proportions of causes reported by the PHMRC, while unreported causes were not considered. For example, the tetanus mortality fraction for neonatal deaths as reported by WHO is as high as 8%, but there are no neonatal deaths due to tetanus in the PHMRC data.

The PHMRC data include neonatal deaths due to co–morbid preterm delivery, birth asphyxia and/or sepsis; and child deaths due to co–morbid pneumonia and diarrhea. For deaths with co–morbid reference standard causes of death, we used the ICD–10 rules to assign a single underlying cause of death [27]. In accordance with the rule that the mode of perinatal death, including prematurity, should not be classified as the main disease or condition unless it was the only condition known, we assigned deaths due to co–morbid preterm/birth asphyxia to birth asphyxia, preterm/sepsis to sepsis, and preterm/sepsis/birth asphyxia proportionately to sepsis and birth asphyxia. Deaths from conditions directly due to prematurity, such as Respiratory Distress Syndrome, were classified as being due to preterm delivery. For child deaths, we proportionately reallocated co–morbid pneumonia/diarrhea deaths to pneumonia or diarrhea. Using these verbal autopsies for harmonized causes of death, we repeated our selection of cause of death distribution and resampling 1000 times for each of the four scenarios. Table 2 summarizes our harmonization of the verbal autopsy algorithms and reference standard causes of death.

Accuracy of VA cause of death determination

After resampling the reference standard cause of death data for neonates and children according to the four mortality scenarios as described above, we then, separately for neonatal and child deaths and for each hierarchy in each scenario, used the expert algorithms to estimate cause of death in the resampled reference standard cause of death data given each individual’s VA questionnaire responses. We used four metrics to examine the validity of the VA cause of death estimates, two at the population level and two at the level of individual cause assignment. Cause-specific mortality fraction (CSMF) accuracy, as defined by Murray et al. [28], is an overall summary of the estimated and reference standard cause distributions with larger values indicating VA CSMF measurements closer to the reference standard. CSMF accuracy is the sum of absolute errors by cause, scaled by the extent of possible error given the smallest cause fraction, and subtracted from one. It is generally interpretable as percent accuracy. To assess the validity of VA estimates of particular causes of death we examined the absolute difference between VA and reference standard CSMFs for these causes.

The last two metrics estimate the accuracy of VA cause of death assignment at the level of individual deaths. Cohen’s kappa is a general measure of agreement between estimated and reference standard causes [29]. Large values of kappa indicate more agreement, where in general values less than zero indicate no agreement, values between 0 and 0.2 are rated as minimal agreement, 0.2 to 0.4 as fair, 0.4 to 0.6 as moderate, 0.6 to 0.8 as substantial, and 0.8 to 1 approach exact agreement [30]. Chance corrected concordance (CCC) is another measure of agreement between VA and reference standard causes at the individual level. This statistic is closely related to Cohen’s kappa and average sensitivity across causes or categories [28]. Similar to kappa, large values indicate more agreement. The CCC scale is from 1/(1–N) to 1, for the number of causes N, while the scale for Cohen’s kappa is from –1 to 1. We used these two metrics only to generate overall summaries of VA accuracy for all causes together.

Ethics statement

The study data are publically accessible and include no personal identifiers. Therefore, no ethical review of the study protocol or informed consent was necessary.

Neonates

Table 3 shows summary results for the expert algorithm cause of death assignments for all causes together from four mortality scenarios and three measures of accuracy. By the CSMF measure, the Baqui and Lawn hierarchies performed best in the moderate and general mortality scenarios, and the compromise hierarchy did best in both high mortality scenarios. These three hierarchies all did their best in the high mortality scenarios, whereas the Kalter and Liu hierarchies did their best in the general scenario, in which their performance nearly equaled that of the Lawn hierarchy. All the hierarchies did their worst, or nearly so, in the moderate mortality scenario. Figure 1 also summarizes CSMF accuracy for neonatal deaths in these scenarios.

Figure 1.  Cause-specific mortality fraction accuracy for six neonatal expert algorithm hierarchies in the resampled Population Health Metrics Research Consortium data, for 1000 simulated cause distributions from four neonatal mortality scenarios, and four neonatal causes (birth asphyxia, congenital malformation, prematurity, sepsis/pneumonia or sepsis/pneumonia/meningitis). Boxes represent interquartile ranges, with a line at the median. Whiskers represent 95% confidence intervals for the median values, and outliers are shown by dots.

The Baqui and Lawn hierarchies performed best by the Cohen’s kappa measure, followed closely by the compromise hierarchy. Generally, for all algorithms, the Cohen’s kappa was between 0.1 and 0.4, indicating minimal to fair agreement between VA estimated and reference standard causes. The CCC statistic also indicates that expert algorithms in the Baqui and Lawn hierarchies provide estimates that are closer to the reference standard causes than either the Kalter or Liu hierarchies, but overall the CCC statistics for all the hierarchies are between 0 and 0.45, indicating small to moderate agreement with the reference standard causes.

The median and range of absolute differences between estimated and reference standard CSMFs are shown in Table 4 for each neonatal cause of death, along with the proportion of deaths that were not classified by each hierarchy. Figure 2 shows the simulated reference standard and estimated CSMF in the general mortality scenario. This difference is identical across all hierarchies for the percent of deaths due to congenital malformations, because this cause is the first in each hierarchy. The Baqui and Lawn hierarchies perform best for birth asphyxia, and Baqui is best for sepsis/pneumonia. The compromise hierarchy is best for prematurity, and the Lawn and compromise hierarchies are jointly best for sepsis/meningitis/pneumonia.

Figure 2.  Cause-specific mortality fractions for six neonatal expert algorithm hierarchies in the resampled Population Health Metrics Research Consortium data, for four neonatal causes in the general neonatal mortality scenario, for 1000 simulated cause distributions.

Children

Table 5 shows summary results for the expert algorithm cause assignment of four causes of child deaths in three hierarchies and four mortality scenarios. We used the same three measures of accuracy as for neonatal deaths at the population and individual levels. At the population level, summarized by CSMF accuracy, the Kalter hierarchy performs best in each scenario. This population level comparison is also shown in Figure 3 .

Figure 3.  Cause-specific mortality fraction accuracy for three expert algorithm hierarchies in the resampled Population Health Metrics Research Consortium data, for 1000 simulated cause distributions from four neonatal mortality scenarios, and four causes of child death (pneumonia/diarrhea, measles, other infectious causes, and injury). Boxes represent interquartile ranges, with a line at the median. Whiskers represent 95% confidence intervals for the median values, and outliers are shown by dots.

The hierarchies are not as strongly differentiated at the individual level for child deaths. There is also some counter indication at the individual level between Cohen’s kappa and the CCC statistic as to which hierarchy is best in each mortality scenario. By Cohen’s kappa, the three hierarchies are very similar in the high mortality with malaria and the general mortality scenarios. Also by Cohen’s kappa, the Liu and Arifeen hierarchies are similar in the high mortality without malaria and moderate mortality scenarios, while the Kalter hierarchy has somewhat lower agreement. The Cohen’s kappa for these three hierarchies generally range from slight (less than 0.2) to fair agreement (0.2–0.4).

By the CCC statistic, Arifeen’s hierarchy has the largest median across the mortality scenarios, although the advantage is small, especially for the high mortality without malaria and moderate mortality scenarios. Overall CCC statistics range from 0.06 to 0.55, indicating small to moderate agreement by the standards for interpreting Cohen’s kappa, and somewhat higher agreement than for neonates.

Figure 4 shows the simulated reference standard and estimated CSMF in the general mortality scenario for six causes of child deaths. The median and range of absolute differences between estimated and reference standard CSMFs across these simulated instances of the PHMRC data for each cause of child death are shown in Table 6 . This difference is identical across all hierarchies for the percent of deaths due to injuries, because this cause occupies the same place in the respective hierarchies. The Kalter hierarchy is best for pneumonia/diarrhea, meningitis/encephalitis, and AIDS. The Arifeen hierarchy is best for other infectious causes, while the Liu hierarchy is generally best for malaria. The Liu hierarchy is especially accurate when malaria is below 0.10 CSMF, while the Kalter hierarchy tends to be more accurate as malaria increases, as shown in Figure 4 . Table 6 shows the median absolute difference in CSMF, which may mask differences depending on the reference standard CSMF.

Figure 4.  Cause-specific mortality fractions for three expert algorithm hierarchies in the resampled Population Health Metrics Research Consortium data, for six child causes in the general mortality scenario, for 1000 simulated cause distributions.

The median absolute differences between estimated CSMF and reference CSMF by cause are also shown for the two high mortality scenarios in Table 6 , both with and without malaria. The relative accuracy of the hierarchies by cause was similar to their performance in the general mortality scenario, except that in the high mortality with malaria scenario the Liu hierarchy did best for meningitis/encephalitis and the Kalter hierarchy worked best for malaria. The median absolute difference for pneumonia and diarrhea in the scenario for high mortality with malaria was 0.35, 0.14, and 0.24 for the Arifeen, Kalter, and Liu hierarchies respectively. These same median absolute differences in the high mortality scenario without malaria were 0.22, 0.09, and 0.19, indicating an improvement in estimated CSMF for pneumonia and diarrhea when the CSMF for deaths due to malaria was low. In addition, the median difference in the pneumonia CSMFs in the Kalter hierarchy was 0.12 in the high mortality scenario with malaria, and 0.10 in the high mortality scenario without malaria. These results reflect improved estimates for pneumonia, as expected given that high malaria burden may complicate other diagnoses, especially for pneumonia [31].

The software for the best performing neonatal and child algorithms and hierarchies, along with the PHMRC questionnaire needed to collect the input data, are available online [20].