Paperonium

← Back to the plain-language case

Abstract's 95% CI for the duration effect of tricyclics excludes its own point estimate

BMJ (2010) Β· doi:10.1136/bmj.c5222 Β· PMC2958257

Summary

The abstract reports that the benefit of tricyclic antidepressants for headache grew with treatment duration, with a metaregression slope of Ξ² = βˆ’0.11 and a 95% confidence interval of βˆ’0.63 to βˆ’0.15 (P < 0.0005). The point estimate βˆ’0.11 lies 0.04 above the upper limit of its own interval, which cannot happen for a 95% CI of any estimate. The interval's midpoint is βˆ’0.39, and its width implies a standard error under which βˆ’0.11 would have P β‰ˆ 0.37, not P < 0.0005. The Results give only subgroup slopes, both internally consistent: tension-type βˆ’0.16 (βˆ’0.23 to βˆ’0.10) and migraine βˆ’0.09 (βˆ’0.14 to βˆ’0.04). These support the direction and significance of the abstract's finding, so the error concerns the printed precision of one abstract result, not the conclusion's direction.

What the paper reports

The PMC-archived text prints minus signs as hyphens; the quotations below keep that form. Every quotation was checked verbatim against the PMC text with src/show.py.

Abstract, Results:

The effect of tricyclics increased with longer duration of treatment (Ξ²=-0.11, 95% confidence interval -0.63 to -0.15; P<0.0005).

Abstract, Conclusions:

The effectiveness of tricyclics seems to increase over time.

Results, "Tricyclics versus placebo", second paragraph:

For both headache types this beneficial effect increased over time (tension-type: Ξ²=-0.16, 95% confidence interval -0.23 to -0.10, P<0.0005, migraine: Ξ²=-0.09, -0.14 to -0.04, P=0.001, fig 3). Therefore for every additional week of treatment with a tricyclic, the number of headaches were reduced by 0.16 standard deviations.

Fig 3 legend:

Fig 3 Effect of tricyclic antidepressants compared with placebo over time. SMD=standardised mean difference

Results, "Assessment of bias and sensitivity analyses":

Metaregression found no effect of any other variables abstracted on summary outcomes, with the exception of study duration. Duration explained 32.4% of the heterogeneity for continuous outcomes and 27.3% of the variance for dichotomous outcomes.

Methods, "Statistical analysis":

We pooled data using the DerSimonian and Laird random effects model

Because of the large number of analyses, we used P=0.01 as our threshold for significance.

All analyses were done using Stata (v 11.2).

Discussion, "Strengths and weaknesses of the study":

This analysis should be viewed as exploratory rather than definitive.

The average duration of the trials in our study was 11 weeks and the longest only 26 weeks; consequently our timeline was constrained to four to 26 weeks.

What the paper's own numbers show

Method. A 95% confidence interval produced by metaregression (as in Stata) is estimate Β± 1.96 SE, so it must contain the estimate, and its half-width gives the SE. I checked whether each printed estimate lies inside its interval, took the midpoint and implied SE of each interval, and converted estimate Γ· SE into a two-sided P value to compare with the printed P. The paper reports no overall (pooled) duration slope outside the abstract. As a plausibility check on the abstract's βˆ’0.11, I pooled the two Results subgroup slopes by inverse variance (fixed effect). This is an approximation, not a re-run of the authors' metaregression, which used study-level data I do not have. Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's own numbers
Overall duration slope: is the point inside its CI? Abstract, Results Ξ² = βˆ’0.11 (βˆ’0.63 to βˆ’0.15) No. βˆ’0.11 is 0.04 above the upper limit βˆ’0.15
Midpoint of the abstract interval Abstract, Results β€” βˆ’0.39
SE implied by the abstract interval width (0.48) Abstract, Results β€” 0.1225
P for Ξ² = βˆ’0.11 with that SE Abstract, Results P < 0.0005 z = βˆ’0.90, P = 0.37
Interval symmetric about βˆ’0.11 keeping the printed upper limit Abstract, Results β€” lower limit βˆ’0.07, above the upper limit βˆ’0.15 (impossible)
Tension-type slope Results, "Tricyclics versus placebo", para 2; Fig 3 βˆ’0.16 (βˆ’0.23 to βˆ’0.10), P < 0.0005 inside CI; midpoint βˆ’0.165; SE 0.0332; z = βˆ’4.82; P = 1.4 Γ— 10βˆ’6. Consistent
Migraine slope Results, para 2; Fig 3 βˆ’0.09 (βˆ’0.14 to βˆ’0.04), P = 0.001 inside CI; midpoint βˆ’0.090; SE 0.0255; z = βˆ’3.53; P = 0.0004. Consistent (printed P slightly conservative)
Inverse-variance pool of the two subgroup slopes Derived from Results (abstract: βˆ’0.11) βˆ’0.116 (βˆ’0.156 to βˆ’0.076), z = βˆ’5.74, P = 9.6 Γ— 10βˆ’9; width 0.079, one-sixth of the abstract's 0.48
Heterogeneity between the two slopes Derived from Results β€” Cochran Q = 2.80 on 1 df, P = 0.09
One possible reading: a dropped zero, βˆ’0.063 to βˆ’0.15 (hypothesis) Abstract, Results β€” contains βˆ’0.11; midpoint βˆ’0.1065; SE 0.0222; z = βˆ’4.96; P β‰ˆ 7 Γ— 10βˆ’7, which matches P < 0.0005
Total change over the 11-week average trial at the printed lower limit Abstract; Discussion (11 weeks) βˆ’0.63 per week βˆ’6.93 SD, against an overall SMD of βˆ’0.96 for tricyclics vs placebo (Results)

Three separate checks point the same way. The printed point estimate is outside its interval. The printed interval width is inconsistent with the printed P value: if the SE were 0.12, as the width implies, βˆ’0.11 would be far from significant. And the printed interval is six times wider than the pooled interval of the two subgroups and wider than either subgroup interval, whereas an estimate that uses both subgroups would normally be more precise than either one alone. The point estimate and the P value, by contrast, agree with the subgroup results.

Figure

The abstract's printed 95% CI for the duration slope lies entirely below its own point estimate

Figure 1. Metaregression slopes of the standardised mean difference (SMD) per additional week of treatment, with 95% confidence intervals, on one linear axis; the vertical line at 0 marks no effect of duration. Orange: values printed in the paper, namely the abstract's overall slope βˆ’0.11 (βˆ’0.63 to βˆ’0.15) and the Results subgroup slopes for tension-type βˆ’0.16 (βˆ’0.23 to βˆ’0.10) and migraine βˆ’0.09 (βˆ’0.14 to βˆ’0.04). Blue: values recomputed from those printed numbers, namely the midpoint of the abstract's interval (βˆ’0.39, diamond) and the inverse-variance pool of the two subgroup slopes, βˆ’0.116 (βˆ’0.156 to βˆ’0.076). The abstract's point estimate sits outside its interval; the subgroup estimates sit inside theirs, and the pooled subgroup estimate lies close to the abstract's βˆ’0.11 with a much narrower interval.

Why it matters

Explanations considered

Status

Suggested correction

The authors may wish to check the overall duration metaregression output and replace, in the abstract,

(Ξ²=-0.11, 95% confidence interval -0.63 to -0.15; P<0.0005)

with the interval that the analysis actually produced for Ξ² = βˆ’0.11. If the intended interval is βˆ’0.15 to βˆ’0.063 (one possible reading, consistent with the point estimate and P < 0.0005), the corrected text would read "(Ξ²=βˆ’0.11, 95% confidence interval βˆ’0.15 to βˆ’0.063; P<0.0005)". It would also help readers if the overall slope and its interval were reported in the Results, next to the subgroup slopes, so that the abstract figure can be traced. The authors may also wish to check the abstract's interval for tension-type headache versus SSRIs (βˆ’2.63 to 0.02), which the Results give as βˆ’1.63 to 0.02.