Paperonium

← Back to the plain-language case

Subgroup effects called significantly different although their 95% CIs overlap, contrary to the paper's criterion

JAMA Network Open (2024) Β· doi:10.1001/jamanetworkopen.2023.56619 Β· PMC10891466

Summary

The paper asks whether food insecurity modifies the association between SNAP receipt and nonadherence to antihypertensive medication. Its Methods say two subgroup estimates count as significantly different only if their 95% CIs do not overlap, and its Results then call the food-insecure estimate (βˆ’13.6 percentage points [pp]; 95% CI βˆ’25.0 to βˆ’2.3) and the food-secure estimate (5.0 pp; 95% CI βˆ’15.8 to 25.7) "statistically significantly different". Those intervals overlap from βˆ’15.8 to βˆ’2.3, and a Wald test on the same intervals gives a difference of 18.6 pp (95% CI βˆ’5.1 to 42.3; p = 0.12). The paper answers yes to its own effect-modification question, but neither its stated criterion nor a conventional test supports that answer. The food-insecure estimate on its own is not affected.

What the paper reports

The research question (Abstract, Key Points, Question):

"Is receipt of Supplemental Nutrition Assistance Program (SNAP) benefits associated with nonadherence to antihypertensive medications and, if so, does this association vary by food insecurity status?"

The criterion for effect modification (Methods, Statistical Analysis, second paragraph):

"The potential effect modification of food insecurity was tested by evaluating for statistical differences in the magnitude and direction of the PATEs for the food-secure and food-insecure subgroups; PATEs were considered statistically significantly different if their respective 95% CIs did not overlap."

The two subgroup estimates (Results, "Association Between SNAP Receipt and Nonadherence to Antihypertensive Medications"):

"Similarly, the rate of nonadherence to antihypertensive medication was similar (385 [56.7% weighted] vs 2671 [55.2% weighted]; PATE, 5.0 [95% CI, -15.8 to 25.7]) between SNAP recipients and nonrecipients among the 5354 patients who did not report experiencing food insecurity."

"In contrast, among the subgroup of 1338 antihypertensive medication users who experienced food insecurity, the rate of nonadherence to antihypertensive medications was 8.17 percentage points lower among SNAP recipients compared with nonrecipients (302 [56.0% weighted] vs 491 [60.0% weighted]; PATE, -13.6 [95% CI, -25.0 to -2.3])."

The conclusion drawn from them (Results, same paragraph, the sentence that follows):

"The observed PATE estimates among the food-secure vs food-insecure subgroups were statistically significantly different, suggesting that food insecurity modified the association between SNAP and nonadherence to antihypertensive medication therapy."

Table 2 gives the same values: food insecure, 302 (56.0) vs 491 (60.0), PATE βˆ’13.6 (βˆ’25.0 to βˆ’2.3); food secure, 385 (56.7) vs 2671 (55.2), PATE 5.0 (βˆ’15.8 to 25.7); overall, 687 (56.4) vs 3162 (55.8), PATE 1.7 (βˆ’21.2 to 24.6).

The finding is carried into the Discussion (first paragraph):

"Among a US sample of patients with hypertension identified from a linked, nationally representative dataset, we observed that SNAP had a differential association with nonadherence to antihypertensive medications between food-secure and food-insecure subgroups."

and into the Strengths (Discussion, Strengths and Limitations):

"Second, this study presented data suggesting that the potential consequences of SNAP receipt on adherence to antihypertensive medications are greatest among the food-insecure population. Thus, we have identified the food-insecure hypertension population as a target for food assistance intervention to address adherence to antihypertensive medications, especially when food assistance resources are limited."

(Quotes are from the PubMed Central archive, which prints minus signs as hyphens; the publisher's page prints the same values with true minus signs.)

What the paper's own numbers show

Method. All inputs come from Table 2 and the Results text. (1) The paper's criterion is applied as written: do the two reported 95% CIs overlap? (2) Each standard error (SE) is recovered from its CI as (upper βˆ’ lower)/(2 Γ— 1.96). Both CIs are symmetric about their point estimates, so this is valid. (3) The two subgroups are disjoint sets of participants, and the Methods describe a stratified analysis with subgroup-specific propensity scores. The difference is therefore tested with the usual Wald test for two independent estimates, z = (5.0 βˆ’ (βˆ’13.6)) / √(SE₁² + SEβ‚‚Β²). (4) To check whether a different confidence level would rescue the claim, the confidence level at which the two intervals just stop overlapping is also computed. Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed
Food-insecure PATE (95% CI) Table 2; Results; Abstract βˆ’13.6 (βˆ’25.0 to βˆ’2.3) Symmetric (half-widths 11.4 / 11.3); SE 5.79 pp; single-group p = 0.019
Food-secure PATE (95% CI) Table 2; Results 5.0 (βˆ’15.8 to 25.7) Symmetric (half-widths 20.8 / 20.7); SE 10.59 pp
Do the two 95% CIs overlap? (paper's criterion) Methods vs Results "statistically significantly different" Yes, they overlap, from βˆ’15.8 to βˆ’2.3 (13.5 pp). The criterion is not met
Food-insecure point estimate inside the food-secure 95% CI? Table 2 β€” Yes (βˆ’13.6 lies within βˆ’15.8 to 25.7)
Difference, food secure βˆ’ food insecure derived from Table 2 "statistically significantly different" 18.6 pp, SE 12.07, 95% CI βˆ’5.1 to 42.3 (includes 0)
Wald test of the difference derived from Table 2 no test statistic given z = 1.54; two-sided p = 0.123 (one-sided p = 0.062)
Confidence level at which the two CIs stop overlapping derived β€” Only below 74.4%. The 90%, 85.1%, 83.4%, 80% and 75% intervals all overlap
Confidence level at which non-overlap would match p < 0.05 for these SEs derived β€” 85.1%. The intervals overlap at that level too
Correlation between the subgroup estimates needed for p = 0.05 derived β€” 0.45
Nonadherent counts Table 2; Results 687; 3162; 3849 302 + 385 = 687; 491 + 2671 = 3162; 687 + 3162 = 3849 (consistent)
Subgroup sizes Results 5354 food secure; 1338 food insecure 6692 βˆ’ 1338 = 5354; 540 + 798 = 1338 (consistent)
Food-insecure gap described in the text Results "8.17 percentage points lower" Weighted crude gap 56.0 βˆ’ 60.0 = βˆ’4.0 pp; unweighted 55.9% βˆ’ 61.5% = βˆ’5.6 pp; adjusted PATE βˆ’13.6 pp. 8.17 matches none of them (separate, minor)
Model behind the PATEs Methods and Abstract vs Table 2 footnote b Methods and Abstract: probit; footnote b: "from a logit model" Internal inconsistency in the description (minor; does not change the CIs)

The counts and sample sizes are mutually consistent, so the problem is not in the reported data. It lies only in the inference drawn from the two intervals.

Figure

Reported subgroup estimates with overlapping 95% CIs, and the recomputed difference between them

Figure 1. All values are on one percentage-point axis (SNAP recipients minus nonrecipients; negative means less nonadherence with SNAP). Orange: the two subgroup PATEs and 95% CIs as reported in Table 2. The shaded band marks where the two intervals overlap (βˆ’15.8 to βˆ’2.3, 13.5 pp); under the paper's stated rule, any overlap means the estimates are not significantly different. Blue: the difference between subgroups (food secure βˆ’ food insecure), recomputed from the same table, 18.6 pp (95% CI βˆ’5.1 to 42.3; z = 1.54; p = 0.12). The thin vertical line labelled "null (0)" marks no effect for the subgroup rows and no difference for the bottom row; the interval for the difference includes it.

Upper bound of the food-insecure CI and lower bound of the food-secure CI as functions of the confidence level

Figure 2. Would a different confidence level change the conclusion? Blue curves: the food-insecure upper bound (solid) and the food-secure lower bound (dashed), recomputed from the SEs implied by Table 2 for confidence levels from 50% to 99%. Orange points: the reported 95% bounds (βˆ’2.3 and βˆ’15.8). The intervals overlap (shaded) at every level above 74.4%. That includes 85.1%, the level at which non-overlap of these two particular intervals would correspond to a two-sided Wald p < 0.05 (dotted line).

Why it matters

Effect modification by food insecurity is the paper's stated question. It is the second half of the Key Points question, it is the Objective ("To evaluate whether receipt of SNAP benefits can modify the consequences of food insecurity on nonadherence to antihypertensive medications."), and it is the purpose of the prespecified stratified analysis. The claim that the association differs by food-insecurity status underpins:

On the paper's own numbers, the data are compatible with no difference between the subgroups: the difference has a 95% CI of βˆ’5.1 to 42.3 pp. A result that is significant in one subgroup and not in the other is not evidence that the subgroups differ, and the paper's non-overlap rule, conservative as it is, was the guard it set against that inference. What survives is the food-insecure estimate on its own (βˆ’13.6 pp, CI excluding 0, p β‰ˆ 0.02 before any adjustment for multiple subgroups). The literal wording "associated ... among the food-insecure subgroup but not among their food-secure counterparts" therefore remains accurate as a description of the two subgroup estimates. The statement that food insecurity modified the association, the "differential association", and the targeting recommendations built on it are not supported. The hedged overall conclusion ("may be less likely ... especially if") is weakened rather than reversed.

Explanations considered

One further observation does not change the conclusion. The overall-sample interval (1.7, βˆ’21.2 to 24.6; half-width 22.9) is about twice as wide as the food-insecure interval (half-width 11.4), although it covers five times as many participants. Without data or code this cannot be examined further; the analysis above uses the subgroup intervals exactly as published.

Status

As of 2026-10-23:

Suggested correction

  1. Replace "The observed PATE estimates among the food-secure vs food-insecure subgroups were statistically significantly different, suggesting that food insecurity modified the association …" with a statement that matches the stated criterion, for example: "The 95% CIs for the food-secure and food-insecure subgroups overlapped; by our prespecified criterion, the PATEs were therefore not statistically significantly different, and we did not find statistical evidence that food insecurity modified the association."
  2. Preferably also report a direct estimate of the difference between subgroups (an interaction model, or a test of the difference between the two PATEs), with its CI and p value. From the published intervals this is approximately 18.6 pp (95% CI βˆ’5.1 to 42.3; p β‰ˆ 0.12).
  3. Revise the Key Points (Findings, Meaning), the Abstract Results and Conclusions, the opening of the Discussion, the Strengths paragraph and the Conclusions so that they report the food-insecure estimate without presenting a difference between subgroups as established.
  4. Correct "8.17 percentage points lower" to agree with Table 2 (adjusted PATE βˆ’13.6; weighted crude difference βˆ’4.0), and make footnote b of Table 2 ("from a logit model") agree with the probit model described in the Methods and the Abstract, or vice versa.