Paperonium

← Back to the plain-language case

Ebola review calls case-fatality change non-significant while reporting OR 1.6, 95% CI excluding 1, p<0.001

BMJ Glob Health (2020) Β· doi:10.1136/bmjgh-2019-001955 Β· PMC7061886

Summary

This review of 34 Ebola virus disease outbreaks (1976–2019) states in its Abstract, Results and Abstract Conclusions that the case fatality rate (CFR) did not change significantly over four decades. It supports that statement with "OR in 2019 vs 1976=1.6 (95% CI 1.5 to 1.8), p<0.001". That interval excludes 1 and the P value is below 0.001, so the statistic contradicts the claim it is cited for. The Results text gives a different interval (1.5 to 1.6) for the same OR. The OR also points the wrong way for the paper's own CFRs (71% falling to 62% gives an OR of about 0.67, not 1.6). The Table 3 CFRs for three of the four periods and the overall CFR of "66% (62 to 71)" cannot be reproduced from the outbreak counts in Table 1. Those counts give a pooled CFR that falls from 71.3% to 41.5% (crude OR 0.29, 95% CI 0.24 to 0.34) and an overall CFR of 43.1%. The CFR trend is the review's only quantitative inference, and the Abstract Conclusions use it to support the claim that control challenges persist. As published, the reported statistics do not support the sentence they accompany. Whether the real trend is significant depends on an analysis the paper does not describe: a patient-level comparison finds a large decline, while analyses that allow for variation between outbreaks find no clear trend.

What the paper reports

Abstract, Methods:

Analyses of trends in case fatality were performed by calculating ORs between times.

Abstract, Results:

In the past four decades, a total of 34 EVD outbreaks affecting 34 356 cases and causing 14 823 deaths were reported in 11 countries in Sub-Saharan Africa. The overall case fatality rate (95% CI) was 66% (62 to 71) and did not change substantially over time (OR in 2019 vs 1976=1.6 (95% CI 1.5 to 1.8), p<0.001).

Abstract, Conclusions:

Sub-Saharan Africa continues to face considerable challenges in EVD control, whereby there are no significant changes in case fatality rates observed during the past four decades.

Methods, "Data extraction and analysis":

Additional analyses of trends in case fatality rates were performed by calculating ORs between periods using MedCalc statistical software (https://www.medcalc.org/calc/odds_ratio.php).

Results, "Context of EVD", fourth paragraph:

Case fatality rates varied from 71% during outbreaks in 1976–1986 to 62% during the later years. However, there was no statistically significant change over time (OR in 2019 vs 1976: 1.6 (95% CI 1.5 to 1.6)). The overall trend from 1976 to 2019 outbreaks shows a linear rate of change over time (table 3, figure 3).

Results, "Context of EVD", third paragraph:

During the past four decades, approximately 14 853 people have died from EVD outbreaks.

Table 3, "OR for case fatality rate of Ebola virus disease according to period" (columns: 1976–1986 | 1987–1997 | 1998–2008 | 2009–2019):

Death (n) | 454 | 348 | 733 | 13 288

Case fatality rate (%) | 71 | 67 | 64 | 62

OR (95% CI) | 1.6 (1.4 to 1.8) | 1.7 (1.5 to 1.9) | 1.4 (1.3 to 1.6) | 1.6 (1.5 to 1.6)

P value | <0.001 | <0.001 | <0.001 | <0.001

Table 1 lists each outbreak's year, country, virus, cases, deaths and CFR. Its total row:

Total | | | 34 356 | 14 823 |

Table 2, sex rows (columns: case | death | CFR (%)):

Gender | Male | 15 803 | 6373 | 46

Gender | Female | 18 553 | 8449 | 54

Table 3 does not state the comparison group or the direction of its ORs, and the paper does not say whether the analysis treated individual patients or outbreaks as the unit.

What the paper's own numbers show

Method. I entered the 34 outbreaks of Table 1 (year, cases, deaths) and assigned each multi-year outbreak (for example "2000–2001") to its first year. Grouping them into the four Table 3 periods reproduces Table 3's death counts exactly (454, 348, 733, 13 288), which confirms the grouping. For each period I computed the pooled CFR (deaths Γ· cases). I then computed crude odds ratios of death with Woolf 95% CIs, the CFR odds ratio implied by Table 3's own CFRs, and a chi-square test of equal CFR across periods. For each reported OR I also derived the z statistic its CI implies. These patient-level calculations treat every case as independent, as a crude OR from a 2Γ—2 table does. The cases are concentrated in a few outbreaks: the 2014 outbreaks in Liberia and Sierra Leone alone account for 72.2% of all cases. So I also ran three outbreak-level analyses, which the paper does not report: a Spearman correlation of outbreak CFR with year, a quasi-binomial logistic model, and a DerSimonian–Laird random-effects meta-regression. Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's numbers
Does the cited OR show "no change"? Abstract, Results 1.6 (95% CI 1.5 to 1.8), p<0.001; "did not change substantially" CI excludes 1; implied z = 10.1. By its own statistics the change is significant
Same OR in the body Results text 1.6 (95% CI 1.5 to 1.6); "no statistically significant change" CI excludes 1 (implied z β‰ˆ 28); also a different CI from the Abstract's for the same quantity
Table 3 ORs Table 3 1.6, 1.7, 1.4, 1.6; all CIs above 1; all P < 0.001 every CI excludes 1; none is a "no change" result
Direction of the OR, using Table 3's CFRs Table 3; Results 62% vs 71%, reported as "OR in 2019 vs 1976" = 1.6 (0.62/0.38) Γ· (0.71/0.29) = 0.67 (0.56 to 0.79), p = 4.1 Γ— 10⁻⁢; the inverse is 1.50 (1.26 to 1.78). A fall in CFR gives an OR below 1
Deaths per period Table 3 vs Table 1 454, 348, 733, 13 288 454, 348, 733, 13 288 (identical)
CFR, 1976–1986 Table 3 71% 454/637 = 71.3%
CFR, 1987–1997 Table 3 67% 348/467 = 74.5%
CFR, 1998–2008 Table 3 64% 733/1199 = 61.1%
CFR, 2009–2019 Table 3; Results 62% 13 288/32 053 = 41.5%
OR of death, 2009–2019 vs 1976–1986 from Table 1 β€” 0.285 (0.240 to 0.339), p β‰ˆ 9 Γ— 10⁻⁴⁢; the inverse is 3.50
Equal CFR across the four periods from Table 1 "no statistically significant change" χ² = 588, df = 3, p β‰ˆ 4 Γ— 10⁻¹²⁷ (patient level)
Overall CFR Abstract "66% (62 to 71)", labelled a 95% CI pooled 14 823/34 356 = 43.1% (binomial 95% CI 42.6 to 43.7); unweighted mean of the 34 outbreak CFRs 61.8%. 66.0% is the mean of the four Table 3 CFRs and 62–71 is their range
Total deaths Table 1; Abstract vs Results text 14 823 vs "approximately 14 853" Table 1 rows sum to 14 823; the text differs by 30
CFR by sex Table 2 male 46, female 54 male 6373/15 803 = 40.3%, female 8449/18 553 = 45.5%. 46.0% and 54.0% are each sex's share of all cases
Outbreak-level trend (not in the paper) from Table 1 β€” Spearman ρ = βˆ’0.16 (p = 0.35); quasi-binomial OR 2009–2019 vs 1976–1986 0.29 (0.04 to 2.09), p = 0.21, dispersion 122; random-effects meta-regression OR per decade 0.81 (0.62 to 1.05), p = 0.11

The table supports three findings.

  1. The reported inference contradicts the reported wording. Every OR the paper prints (Abstract, Results, the four Table 3 columns) has a CI that excludes 1 and a P value below 0.001. The sentences that cite them say there was no significant change.
  2. The reported OR does not match the reported CFRs. For CFRs of 71% and then 62%, the OR of death for the later period is 0.67. Its inverse, 1.50 (1.26 to 1.78 with the Table 1 period sizes), is close to the reported 1.6 but describes the earlier period against the later. Neither the Abstract's CI (1.5 to 1.8) nor the Results' CI (1.5 to 1.6) is reproduced.
  3. Table 3's CFRs, apart from the first period, do not follow from Table 1. Table 3's death counts match Table 1 exactly. Its CFRs for 1987–1997, 1998–2008 and 2009–2019 differ from deaths Γ· cases by βˆ’7.5, +2.9 and +20.5 percentage points. The largest gap is in the period that holds 93% of all cases (32 053 of 34 356). They do not match the unweighted means of the outbreak CFRs either (76.5%, 53.5%, 64.9%, 58.9%). The Abstract's "66% (62 to 71)" is reproduced exactly by the mean and range of the four Table 3 CFRs, not by any pooled estimate or confidence interval.

Figure

Reported and recomputed odds ratios of death for the later versus the earliest period, on one log axis

Figure 1. Odds ratios of death with 95% CIs on a log scale. The thin vertical line is OR = 1 (no change). Orange: every OR the paper reports for the CFR trend (Abstract, Results text, and the four columns of Table 3). All lie to the right of 1 with intervals that exclude it, and all carry P < 0.001, yet the paper says there was no significant change. Blue diamonds are recomputed from the paper's own numbers. The upper one is the OR implied by Table 3's CFRs (62% vs 71%). The lower one is the crude OR for 2009–2019 vs 1976–1986 from the Table 1 counts. Both lie to the left of 1, the opposite direction from the reported 1.6. The hollow blue square is the same Table 1 comparison with a CI widened for variation between outbreaks (quasi-binomial logistic model, dispersion 122). It is not an analysis the paper reports. It shows that the data are compatible with no change once outbreaks rather than patients are treated as the unit.

Period case fatality rates as reported and as computed from Table 1

Figure 2. Case fatality rate for each Table 3 period and overall, on an axis from 0 to 100%. Orange circles: Table 3's CFRs and the Abstract's overall 66%. Blue diamonds: deaths Γ· cases from the outbreak rows of Table 1, grouped so that the deaths equal Table 3's. The first period agrees (71% vs 71.3%). The others differ, most of all in 2009–2019 (62% vs 41.5%) and overall (66% vs 43.1%).

Why it matters

The CFR trend is the only quantitative inference in the review. It is stated in the Abstract Results, the Abstract Conclusions and the Results text, and the Abstract Conclusions present it as evidence for the review's framing ("continues to face considerable challenges in EVD control, whereby there are no significant changes in case fatality rates"). A reader who takes the Abstract at face value gets a claim (no significant change), a statistic that contradicts it (CI excluding 1, p<0.001), and an overall CFR (66%) that is about 23 percentage points above the pooled figure implied by the paper's own outbreak table (43.1%).

The paper's numbers do not settle what the correct conclusion is, and this write-up does not claim to. Two readings are defensible, and neither matches the paper.

Either way, the published statistics cannot support the published sentence, and the overall CFR and three of the four period CFRs are not what the paper's own table implies. The review's qualitative synthesis (epidemiological, health-system and sociocultural challenges) and its recommendations (one-health surveillance, community engagement, research on diagnostics, drugs and vaccines, health-system resilience) do not depend on these numbers.

Explanations considered

Status

Suggested correction

  1. Replace "did not change substantially over time" (Abstract), "there was no statistically significant change over time" (Results) and "there are no significant changes in case fatality rates" (Abstract Conclusions) with wording that matches whatever analysis is reported. The current ORs, with CIs excluding 1 and P < 0.001, indicate a significant difference.
  2. State the comparison and direction of every OR in Table 3 (which period against which reference), and the unit of analysis. Report a single, consistent CI for the "2019 vs 1976" OR; the Abstract gives 1.5 to 1.8 and the Results 1.5 to 1.6.
  3. Recompute the Table 3 CFRs as deaths Γ· cases from Table 1 (71.3%, 74.5%, 61.1%, 41.5%) and add the case counts to the table, or explain how the printed 67%, 64% and 62% were derived.
  4. Replace the overall "66% (62 to 71)" with the pooled CFR from Table 1 (43.1%) and a valid interval, or label it as the mean and range of the period CFRs if that is what it is.
  5. Because CFR varies widely between outbreaks and one region contributes most of the cases, consider a trend analysis that accounts for clustering by outbreak (for example random-effects or quasi-binomial models). Base the conclusion on its estimate and interval.
  6. Minor: correct "approximately 14 853" deaths to 14 823 (Table 1 total), and change the Table 2 "CFR (%)" entries for sex (46, 54) to 40.3 and 45.5, or relabel the column as the share of cases. The Table 2 age groups sum to 33 456 cases, 900 fewer than the 34 356 total, and the table does not say how the remaining cases are classified.