Paperonium

← Back to the plain-language case

Reported HbA1c intervention effect of 0.02% lies outside its own 95% confidence interval of 0.03%–0.14%

JAMA Netw Open (2023) Β· doi:10.1001/jamanetworkopen.2022.50375 Β· PMC9856961

Summary

This secondary analysis of a cluster randomized trial (Texas Sprouts, 16 elementary schools) reports that, compared with control schools, the intervention produced a "0.02% reduction in mean hemoglobin A1c (95% CI, 0.03%-0.14%; P = .005)". The 0.02 figure appears in the Key Points, and the full estimate with its CI and P value appears in the Abstract, the Results text and Table 3. In each of those three places the estimate 0.02 lies below the lower limit of its own 95% CI. The four reported quantities (estimate 0.02, SE 0.01, CI 0.03 to 0.14, P = .005) cannot all be correct. The CI and the P value agree with each other and point to an effect of about 0.085 percentage points. The estimate and SE agree with neither, and the unadjusted difference in group changes in Table 3 (βˆ’0.03 vs +0.06, so 0.09) agrees with the CI. HbA1c is the only outcome behind the paper's conclusion that the intervention "improved glucose control". The Limitations say the improvement "may not be clinically relevant", and then add that "even small reductions in HbA1c levels reflect clinical improvements in glucose control". The improvement the paper reports, 0.02, is about one quarter of the size the paper's other numbers imply.

What the paper reports

Key Points, "Findings":

In this secondary analysis of a cluster randomized clinical trial of 695 children, a gardening, nutrition, and cooking intervention resulted in reductions of 0.02% for hemoglobin A1c and 6.40 mg/dL for low-density lipoprotein cholesterol levels.

Abstract, "Results":

Compared with control schools, children from Texas Sprouts schools had a 0.02% reduction in mean hemoglobin A1c (95% CI, 0.03%-0.14%; P = .005)

Results, paragraph on intervention effects:

Compared with children in the control schools, children in the Texas Sprouts intervention had a 0.02% reduction in mean HbA1c (95% CI, 0.03%-0.14%; P = .005)

Table 3, HbA1c row. Columns: intervention children, preintervention mean (SE), change mean (SE) | control children, preintervention mean (SE), change mean (SE) | differences in changes, mean (SE) [95% CI] | P value, intervention effect. (The minus sign is a Unicode minus in the publisher's XML.)

Hemoglobin A1c, %c | 236 | 5.27 (0.01) | -0.03 (0.02) | 221 | 5.22 (0.01) | 0.06 (0.02) | 0.02 (0.01) [0.03 to 0.14] | .005

Table 3, footnotes b and c:

Generalized linear mixed models with the identity link for continuous variables were used to test differences in metabolic parameters between the intervention and the control estimates, with schools as clusters fixed for their intervention effect and children nested within schools.

Children at 5 intervention schools and 5 control schools were analyzed for hemoglobin A1c.

Methods, "Statistical Analysis":

Data were analyzed using SPSS statistical software version 28 (IBM) and a 2-sided type I error level of Ξ± = .05 was used as the threshold for statistical significance.

The conclusions that rest on this result. Abstract, "Conclusions and Relevance":

In this cluster RCT, Texas Sprouts improved glucose control and reduced low-density lipoprotein cholesterol in high-risk youth.

Discussion, first paragraph:

To our knowledge, Texas Sprouts was the first cluster RCT to show that a school-based gardening, nutrition, and cooking intervention can improve glucose control and lower LDL cholesterol in elementary schoolchildren.

Discussion, paragraph on body composition:

The current intervention resulted in reductions in LDL cholesterol and improvements in HbA1c, independent of changes in body composition.

Discussion, "Limitations":

Another limitation is that the improvement in HbA1c was rather small, albeit significant, and may not be clinically relevant. However, health care professionals believe that even small reductions in HbA1c levels reflect clinical improvements in glucose control.

Conclusions:

In conclusion, the present RCT showed that a school-based gardening, nutrition, and cooking intervention resulted in a small, albeit significant, improvement in glycemic control and a reduction in LDL cholesterol in predominately low-income and racially diverse elementary schoolchildren.

What the paper's own numbers show

Method. The paper states an identity-link linear mixed model, so each 95% CI should be symmetric about its estimate, with half-width equal to a fixed multiplier times the SE. To find that multiplier I divided the CI half-width by the SE in the other rows of Table 3. In the seven rows without other problems it is 2.138 to 2.206 (median 2.148, not 1.96), which matches a t reference with about 14 degrees of freedom. I therefore used t(14) for the HbA1c checks, and give normal-reference values as well. For each pair of reported quantities (estimate, SE, CI, P) I worked out what one implies for the other. As an independent check I took the unadjusted difference between the two group changes in Table 3, with an SE from the two group SEs. That SE ignores clustering, so it is only a rough check. Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's numbers
Estimate inside its own 95% CI? Abstract, Results, Table 3 0.02 with CI 0.03 to 0.14 No: 0.02 is 0.01 below the lower limit
Centre of the reported CI Table 3 (estimate) 0.02 (0.03 + 0.14) / 2 = 0.085
SE implied by the CI width Table 3 0.01 0.055 / 2.145 = 0.026 (t14); 0.028 (normal)
CI implied by estimate 0.02 and SE 0.01 Table 3 0.03 to 0.14 βˆ’0.001 to 0.041 (t14); 0.000 to 0.040 (normal)
P implied by estimate/SE (0.02/0.01 = 2.0) Abstract, Results, Table 3 .005 .065 (t14); .046 (normal)
P implied by the reported CI Abstract, Results, Table 3 .005 .0051 (t14); .0025 (normal); .004 to .007 for df 8 to 20
Estimate needed for P = .005 if SE were 0.01 Table 3 0.02 0.033 (t14); 0.028 (normal)
Unadjusted difference in changes, intervention βˆ’ control Table 3 change columns: βˆ’0.03 (0.02) vs 0.06 (0.02) 0.02 βˆ’0.09, i.e. a reduction of 0.09; SE 0.028, CI βˆ’0.145 to βˆ’0.035 (ignores clustering)
Sign of the difference column Table 3, rows with clear effects (e.g. LDL changes βˆ’8.43 vs βˆ’3.53, difference βˆ’6.40) HbA1c CI 0.03 to 0.14 (positive) In those rows the column reads intervention βˆ’ control, so a reduction should be negative: βˆ’0.14 to βˆ’0.03, which matches βˆ’0.145 to βˆ’0.035
Reported effect vs CI centre Key Points, Abstract 0.02 The CI centre is 4.25 times larger; the unadjusted difference is 4.5 times larger

Three things follow. First, the estimate cannot belong to the reported interval, so at least one of them is wrong. Second, the CI and the P value fit together: under the multiplier the rest of Table 3 implies, the CI gives P = .0051, against the reported .005. The estimate 0.02 and SE 0.01 fit neither of them, and they are also much smaller than the unadjusted difference. Third, the CI's sign does not follow the convention the table uses in its rows with clear effects. In those rows (fasting glucose, total, LDL and non-HDL cholesterol) the column reads intervention βˆ’ control, so an HbA1c reduction would be negative. With the sign restored, the HbA1c CI matches the unadjusted difference closely (βˆ’0.14 to βˆ’0.03 against βˆ’0.145 to βˆ’0.035). The most coherent reading of the paper's own numbers is an HbA1c reduction of about 0.085 to 0.09 percentage points, with the estimate and SE cells in error. The exact model estimate cannot be rebuilt from these rounded summaries, because elsewhere in Table 3 the model estimates differ from the unadjusted differences by up to about 2 SEs (for example, non-HDL cholesterol βˆ’6.13 against βˆ’3.83, about 2.2 SE). In fasting insulin and HDL the model estimate and the unadjusted difference differ in sign (+0.61 against βˆ’0.42; +0.43 against βˆ’0.31), which fits adjustment gaps of that size. So the sign convention is inferred from the rows with clear effects, not shown for every row.

Related observations in Table 3. These do not change the HbA1c finding, but they bear on how the table can be read:

Figure

Dot-and-interval plot on one axis of the HbA1c effect: the reported estimate 0.02 sits outside its reported CI 0.03 to 0.14; recomputed intervals from the reported estimate and SE, from the reported CI, and from the unadjusted group changes

Figure 1. HbA1c effect in Table 3, on one axis in percentage points, with positive values meaning lower HbA1c in intervention schools than in control schools, as the paper's text reads "0.02% reduction (95% CI, 0.03%-0.14%)". The top row (orange) is what the paper reports: the dot (0.02) sits outside the bar (0.03 to 0.14). The blue rows are recomputed from the paper's own numbers. The second row is the interval that the reported estimate and SE would give (βˆ’0.001 to 0.041, P = .065), which does not match the reported CI or P. The third row is the centre and SE implied by the reported CI (0.085, SE 0.026, P = .0051), which matches the reported P. The fourth row is the unadjusted difference in the two group changes in Table 3 (βˆ’0.03 vs +0.06), shown as a reduction of 0.09 with CI 0.035 to 0.145. Its SE ignores clustering. The vertical line marks no effect. Intervals use the t(14) multiplier that the other rows of Table 3 imply.

Position of each reported estimate within its own reported 95% CI, for every row of Table 3

Figure 2. Where each reported estimate sits within its own reported 95% CI, for all nine outcomes in Table 3, scaled so that the lower limit is 0, the centre 0.5 and the upper limit 1. Seven rows sit at the centre, as an identity-link model requires. Triglycerides sits off-centre (its lower limit, βˆ’15.42, is not symmetric about 1.52). HbA1c is the only estimate outside its interval.

Why it matters

Explanations considered

Status

Suggested correction

  1. Report the HbA1c intervention effect from the fitted model with an estimate, SE, 95% CI and P value that agree with one another. Correct the Key Points, Abstract, Results text and Table 3 together. Use the sign convention of the Table 3 rows with clear effects (intervention βˆ’ control, so a reduction is negative) in the table, and state the magnitude in the text.
  2. If, as the CI and P value suggest, the effect is about 0.08 to 0.09 percentage points, revisit the Limitations and Conclusions statements that the improvement was small and "may not be clinically relevant", which accompany the reported 0.02.
  3. State which test the "P value, intervention effect" column reports. For fasting glucose, HOMA-IR, total, LDL, HDL and non-HDL cholesterol the printed P values do not follow from the displayed differences and CIs (for example, fasting glucose has a CI of 1.60 to 6.84 but P = .13). If the P values are wrong, the statements that there were "no intervention effects" on glucose and other lipids need to be revisited too.
  4. Check the triglycerides interval (βˆ’15.42 to 6.47), which is not symmetric about its estimate 1.52, and remove the minus signs printed on SEs, as in "βˆ’6.40 (βˆ’1.20)".