Paperonium

← Back to the plain-language case

Main IGF-1 hospitalization odds ratio 0.62 is printed with a 95% CI that excludes it

eLife (2022) Β· doi:10.7554/eLife.79720 Β· PMC9576268

Summary

The paper's primary (random-effects IVW) estimate for genetically predicted IGF-1 and COVID-19 hospitalization is given as OR 0.62 (95% CI 0.25–0.51, p = 0.018) in the Abstract, the Results and Table 2. That interval cannot belong to that estimate, because its upper limit (0.51) is below the point estimate (0.62). On the log scale the printed interval is centred on 0.36, and it would imply p β‰ˆ 1.5 Γ— 10-8, not 0.018. The paper's own OR and p-value imply a 95% CI of about 0.42–0.92. The hospitalization forest plot in the paper's Figure 2 (the panel titled "A. MR analysis for IGF-1 and COVID-19 hospitalization"; "the forest plot" below) draws the IVW whisker at that interval, and the interval is close to the paper's MR-PRESSO (0.43–0.88) and BMI-adjusted IVW (0.41–0.90) intervals. The OR, the p-value and the stated conclusions are unaffected, but the precision printed for a headline result in the Abstract is wrong and overstates how far the data exclude small effects.

What the paper reports

Abstract, Methods part:

Random-effects inverse variance weighted (IVW) MR approach was used as the primary MR method and the weighted median, MR-Egger, and MR Pleiotropy RESidual Sum and Outlier (MR-PRESSO) test were conducted as sensitivity analyses.

Abstract, Results part, and repeated word for word in the first paragraph of Results:

For one standard deviation increase in genetically predicted IGF-1 levels, the odds ratio was 0.77 (95% confidence interval [CI], 0.61–0.97, p=0.027) for COVID-19 susceptibility, 0.62 (95% CI: 0.25–0.51, p=0.018) for COVID-19 hospitalization, and 0.85 (95% CI: 0.52–1.38, p=0.513) for COVID-19 severity.

Table 2 ("Sex hormones, SHBG, IGF-1, and COVID-19 outcomes in Mendelian randomization (MR) analyses"), IGF-1 IVW row. Columns, in each outcome block: SNPs | OR (95% CI) | p Effect | p Heterogeneity | p Intercept, for susceptibility, hospitalization and severity:

IGF-1 | IVW | 16 | 0.77 (0.61, 0.97) | 0.027 | 0.175 | – | 16 | 0.62 (0.25, 0.51) | 0.018 | 0.715 | – | 18 | 0.85 (0.52, 1.38) | 0.513 | 0.601 | -

Table 2, IGF-1 MR-PRESSO row (in all three MR-PRESSO cells the upper limit is printed first):

IGF-1 | MR-PRESSO | 16 | 0.77 (0.98, 0.61) | – | – | – | 16 | 0.62 (0.88, 0.43) | – | – | – | 18 | 0.85 (1.34, 0.54) | - | - | -

Results, first paragraph:

Associations of IGF-1 levels with COVID-19 susceptibility and hospitalization were not statistically significant after Bonferroni correction, albeit showing a nominal significance at p<0.05.

Results, second paragraph (Table 6, BMI-associated SNPs removed):

As shown in Table 6, after removing SNPs associated with BMI, we found similar associations of genetically predicted IGF-1 levels with the risk of COVID-19 susceptibility (OR = 0.76; 95% CI: 0.60–0.96, p=0.021), hospitalization (OR = 0.61; 95% CI: 0.41–0.90, p=0.014) …

Discussion, last paragraph (the Abstract's Conclusions open with the same sentence, without "In conclusion,"):

In conclusion, our study indicated that genetically predicted high IGF-1 levels were associated with decrease the risk of COVID-19 susceptibility and hospitalization, but these associations did not survive the Bonferroni correction of multiple testing.

What the paper's own numbers show

Method. IVW Mendelian-randomization estimates are log odds ratios with a normal (Wald) standard error, so the reported OR is exp(Ξ²), the 95% CI is exp(Ξ² Β± 1.96Β·SE) and p = 2Ξ¦(βˆ’|Ξ²|/SE). Two checks follow. (1) The geometric centre √(lower Γ— upper) of a printed CI must equal the OR, apart from rounding, and the CI's width fixes SE and hence p. (2) The OR and p fix SE = |ln OR| / Ξ¦-1(1 βˆ’ p/2), and hence the CI. I applied both checks to the hospitalization cell, to the other five IGF-1 IVW cells of Table 2 (main analysis) and Table 6 (BMI-associated SNPs removed), and to the neighbouring IGF-1 estimates. To see whether rounding could matter, I let OR range over [0.615, 0.625) and p over [0.0175, 0.0185). Finally, I read the vector coordinates of the forest plot (the panel "A. MR analysis for IGF-1 and COVID-19 hospitalization" in the paper's Figure 2, page 8 of the publisher's v2 PDF) and converted the drawn whiskers to ORs using the plot's own linear axis ticks (0, 1, 2, 3). Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's own numbers
IGF-1 β†’ hospitalization, IVW Abstract; Results para. 1; Table 2 OR 0.62 (95% CI 0.25–0.51), p = 0.018 0.62 lies outside 0.25–0.51
Centre of the printed interval same expected to equal 0.62 √(0.25 Γ— 0.51) = 0.357
p implied by the printed interval same 0.018 SE(log OR) 0.182; p = 1.5 Γ— 10-8
95% CI implied by OR 0.62 and p = 0.018 same 0.25–0.51 z = 2.366, SE(log OR) = 0.202, CI 0.417–0.921
Same, allowing for rounding of OR and p same β€” lower limit 0.410–0.424, upper limit 0.918–0.924; no rounding gives an upper limit below 0.515
IVW whisker drawn in the forest plot Forest plot (label "0.62(0.25,0.51)") β€” drawn from 0.420 to 0.920, box at 0.620
Calibration of that reading: MR-Egger bar Forest plot (label "0.72(0.37,1.38)") 0.72 (0.37, 1.38) drawn 0.370–1.380, box at 0.720
Weighted-median bar Forest plot (label "0.75(0.44,1.28)") 0.75 (0.44, 1.28) drawn 0.450–1.240, box at 0.750: matches Table 6's BMI-adjusted weighted median, 0.75 (0.45, 1.24), not the label or Table 2
IGF-1 β†’ hospitalization, MR-PRESSO Table 2 0.62 (0.88, 0.43) contains 0.62; centre 0.615
IGF-1 β†’ hospitalization, IVW without BMI SNPs Table 6; Results para. 2 0.61 (0.41–0.90), p = 0.014 CI from OR and p: 0.41–0.90; p from CI: 0.013
IGF-1 β†’ susceptibility, IVW Abstract; Table 2 0.77 (0.61–0.97), p = 0.027 CI from OR and p: 0.61–0.97; p from CI: 0.027
IGF-1 β†’ severity, IVW Abstract; Table 2 0.85 (0.52–1.38), p = 0.513 CI from OR and p: 0.52–1.38; p from CI: 0.505

The other five IGF-1 IVW cells of Tables 2 and 6 pass both checks to rounding precision: in each, the printed p lies inside the range of p-values implied by every rounding of the printed limits. So for these tables the Wald method fits how the paper's intervals were produced. Only the main-analysis hospitalization IVW cell fails, and it fails both checks by a wide margin. Its OR and p agree with each other and point to a CI of about 0.42–0.92. Three other items in the paper are consistent with that interval: the IVW whisker drawn in the forest plot (0.42–0.92, with the same axis reading reproducing the MR-Egger bar exactly), the MR-PRESSO interval for the same 16 SNPs (0.43–0.88), and the IVW interval with one BMI-associated SNP removed (0.41–0.90). The last two come from largely the same data as the IVW estimate, so they are not independent confirmations. No other cell in the paper prints the pair (0.25, 0.51).

Figure

The printed interval for the main hospitalization estimate lies entirely below its own point estimate; the interval implied by the paper's OR and p, and the IVW whisker drawn in its hospitalization forest plot, are 0.42–0.92

Figure 1. IGF-1 and COVID-19 hospitalization: odds ratio per 1-SD higher genetically predicted IGF-1, with 95% CI, on a log scale. The vertical line marks OR = 1. Orange: values printed in the paper. The top row is the primary IVW estimate as printed in the Abstract, the Results and Table 2, 0.62 (0.25–0.51); the short tick marks the interval's geometric centre, 0.36, which an interval for 0.62 would have at 0.62. The bottom two orange rows are the paper's MR-PRESSO interval for the same analysis (Table 2) and its IVW interval after removing BMI-associated SNPs (Table 6). Blue: the 95% CI recomputed from the printed OR 0.62 and p = 0.018, 0.42–0.92. Grey open square: the IVW whisker as actually drawn in the paper's hospitalization forest plot (Figure 2, panel "A. MR analysis for IGF-1 and COVID-19 hospitalization"), measured from the PDF's vector coordinates, 0.42–0.92. All numbers are from the paper or from recompute.py.txt.

For five of the six IGF-1 IVW cells the printed p matches the p implied by the printed interval; for main-analysis hospitalization they differ by six orders of magnitude

Figure 2. For each IGF-1 IVW estimate in Table 2 (main analysis) and Table 6 (BMI-associated SNPs removed), the printed two-sided p-value (orange dots) and the p-value implied by the printed 95% CI (blue open diamonds), on a log scale. The vertical line marks p = 0.05. In five cells the two agree to rounding (for example 0.027 printed and 0.027 implied for susceptibility). For main-analysis hospitalization the printed interval implies p = 1.5 Γ— 10-8, against a printed p of 0.018. Among the IGF-1 IVW cells of Tables 2 and 6, the error is confined to that one interval.

Why it matters

Explanations considered

Status

Suggested correction

In the Abstract, the Results (first paragraph) and Table 2 (IGF-1, IVW, hospitalization), the interval "0.25–0.51" / "(0.25, 0.51)" should be replaced by the interval from the original IVW output. From the printed OR and p-value that interval is about 0.42–0.92, which is also where the paper's forest plot draws the IVW whisker. The exact limits should come from the authors' analysis output, not from this back-calculation. The label "0.62(0.25,0.51)" in the forest plot should be corrected to match. The authors may also wish to check the forest plot's MR-PRESSO label "0.62(0.60,0.64)" against Table 2's "0.62 (0.88, 0.43)", its weighted-median bar (drawn at Table 6's 0.45–1.24) against its label, and the Table 3 IVW p-values for hospitalization and severity against their intervals. They could also print every interval in the paper lower limit first.