Paperonium

← Back to the plain-language case

Abstract's verified-vaccination percentages and effect do not match its own counts or the Results

Nat Med (2023) · doi:10.1038/s41591-023-02670-4 · PMC10719106

Summary

The abstract of this cluster-randomized trial in rural Ghana reports that 36.6% (355 of 1,058) of verified participants in the cash-incentive arms had at least one COVID-19 vaccine dose, compared with 30.3% (439 of 1,544) in the placebo arm, a difference of 6.3% (95% CI 2.4, 10.2; P = 0.001). The counts printed in that sentence give 33.6% and 28.4%. Those are the rates the Results text and Table 2 report for the same verified sample, where the difference is 5.2% (95% CI 1.5, 8.7; P = 0.005). The abstract's 36.6%, 6.3% and 2.4–10.2 match, figure for figure, a spillover estimate that the Results give for a different subgroup. Verified vaccination is the trial's only registry-confirmed outcome, so the abstract overstates the size and precision of its headline objective result, though the effect's direction and statistical significance are unchanged.

What the paper reports

Abstract:

We also verified vaccination status of participants: in the cash treatment arm, 36.6% (355 of 1,058) of verified participants had at least one dose of the COVID-19 vaccine compared to 30.3% (439 of 1,544) for those in the placebo—a difference of 6.3% (95% CI: 2.4, 10.2; P = 0.001).

Results, "Combined low and high cash treatment effect", third paragraph:

In the cash treatment arm (both low and high cash), we verified that 33.6% of individuals had at least one dose of the COVID-19 vaccine as of April 2023 compared to 28.4% of those in the placebo arm (difference from placebo: 5.2%; 95% CI: 1.5, 8.7; P = 0.005).

In relative terms, this represents about an 18% increase over the non-treated placebo vaccination rates.

Results, "Verified vaccination status":

For our six-district sample, we verified that 29.1% of the 3,075 participants received a COVID-19 vaccination

We verified that 28.4% of individuals in the placebo arm were vaccinated (at least one dose)

Table 2, phase IV panel ("Phase IV: verified vaccination status of treated participants (30 April 2022)"; columns: total sample | placebo | health message | low cash | high cash):

Number of participants | 3075 | 1544 | 473 | 548 | 510

Actual vaccination (%) | 29.1 (45.4) | 28.4 (45.1) | 21.6 (41.2) | 40.7 (49.2) | 25.9 (43.8)

Results, "Spillover", second paragraph, which describes Extended Data Table 5 (participants who saw the placebo video, grouped by the treatment assigned to their village):

Participants in placebo-treated village clusters had an average vaccination rate of 26.8%.

Individuals receiving the placebo video in low cash treatment villages had a verified vaccination rate of 38.3% (difference from placebo: 11.5%; 95% CI: 5.2, 17.8; P < 0.001).

Placebo participants in the combined cash treatment also had a verified vaccination rate, 36.6%, that is significantly larger than the placebo treatment village participants (difference from placebo: 6.3%; 95% CI: 2.4, 10.2; P = 0.002).

What the paper's own numbers show

Method. I divided the abstract's numerators by its denominators. I then rebuilt each arm's vaccinated count from the phase IV panel of Table 2 (n × printed %) and checked that the cash-arm count and size equal the abstract's 355 and 1,058. I computed the cash-minus-placebo difference with a naive two-proportion Wald 95% interval and normal-approximation P value. This interval ignores village clustering and serves only to check the printed arithmetic. Finally I set the abstract's figures beside the spillover paragraph's figures. Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's numbers
Cash-arm size Abstract; Table 2 phase IV 1,058; low cash 548, high cash 510 548 + 510 = 1,058, the same sample
Cash-arm vaccinated Abstract; Table 2 phase IV 355; 40.7% of 548, 25.9% of 510 223.0 + 132.1 → 223 + 132 = 355, the same count
Placebo size and vaccinated Abstract; Table 2 phase IV 439 of 1,544; 28.4% of 1,544 1,544 × 28.4% = 438.5 (438 or 439 after rounding), consistent
Total verified sample Table 2 phase IV 3,075; 29.1% 895/3,075 = 29.1% from the rebuilt counts, consistent
Cash-arm rate Abstract 36.6% (355 of 1,058) 355/1,058 = 33.55% → 33.6%
Cash-arm rate Results, combined cash effect 33.6% 33.6%, agrees
Placebo rate Abstract 30.3% (439 of 1,544) 439/1,544 = 28.43% → 28.4%
Placebo rate Results; Table 2 28.4% 28.4%, agrees
Counts the abstract's percentages would need Abstract 36.6% of 1,058; 30.3% of 1,544 387 (not 355); 468 (not 439)
Cash minus placebo Abstract 6.3% (95% CI 2.4, 10.2; P = 0.001) 5.12 pp, naive 95% CI (1.5, 8.7), P = 0.0057
Cash minus placebo Results, combined cash effect 5.2% (95% CI 1.5, 8.7; P = 0.005) 5.12 pp (33.6 − 28.4 = 5.2 from the rounded rates); same interval
Relative increase Results "about 18%" 5.12/28.43 = 18.0%; the abstract's figures imply 6.3/30.3 = 20.8%
Placebo-video participants in cash villages Results, Spillover 36.6%; difference 6.3 (95% CI 2.4, 10.2; P = 0.002) Identical to the abstract's 36.6%, 6.3 and 2.4–10.2; the P values differ (0.002 vs 0.001)
Spillover reference rate Results, Spillover placebo villages 26.8% 38.3 − 11.5 = 26.8 agrees for the low-cash line; 36.6 − 6.3 = 30.3, not 26.8 (36.6 − 26.8 = 9.8)

The abstract's counts and denominators agree with every other part of the paper. Its percentages, difference, interval and P value do not agree with those counts. The abstract's placebo rate, 30.3%, is exactly 36.6 − 6.3, the spillover paragraph's rate minus its difference.

Figure

Abstract versus Results/Table 2 for verified vaccination, with values recomputed from the abstract's counts

Figure 1. The left panel gives the abstract's verified-vaccination figures and the right panel the Results text and Table 2 (phase IV) figures, both as printed (orange). Both panels describe the same 1,058 cash-arm and 1,544 placebo participants. The blue strip shows the rates and difference recomputed from the abstract's own counts, 355/1,058 and 439/1,544. These agree with the Results and Table 2, not with the abstract. The note under the strip records where the abstract's 36.6%, 6.3% and 2.4–10.2 appear in the paper: in the spillover paragraph, for a different subgroup.

Cash-minus-placebo difference in verified vaccination with 95% CIs, as printed and as recomputed

Figure 2. Cash-minus-placebo difference in verified vaccination (percentage points, 95% CI) on one axis, with zero as a thin reference line. Orange marks the values printed in the abstract and in the Results. Blue marks the value recomputed from the abstract's counts with a naive, unclustered two-proportion interval. Gray marks the spillover-subgroup estimate printed in the Results. The recomputed value and interval match the Results. The abstract's estimate and interval are the same as the spillover subgroup's.

Why it matters

A side note for a statistician: the Results interval (1.5, 8.7) is the same as a naive interval that ignores clustering, although the Results report an intracluster correlation of 0.64 for verified vaccination. This write-up compares like with like and makes no claim about which interval is appropriate. The inconsistency described here does not depend on that question.

Explanations considered

Status

As of 2026-10-23:

Suggested correction

In the abstract, change the verified-vaccination sentence so that it matches its counts and the Results, for example: "in the cash treatment arm, 33.6% (355 of 1,058) of verified participants had at least one dose of the COVID-19 vaccine compared to 28.4% (439 of 1,544) for those in the placebo—a difference of 5.2% (95% CI: 1.5, 8.7; P = 0.005)." The authors may also wish to:

  1. Check the spillover paragraph's combined-cash figures (36.6%; difference 6.3%; 95% CI 2.4, 10.2; P = 0.002) against Extended Data Table 5 and the replication data, since 36.6 − 6.3 does not equal the stated reference rate of 26.8%.
  2. Change "as of April 2023" in the combined-cash sentence to the 30 April 2022 cut-off used in Table 2, if that is the intended date.