Paperonium

← Back to the plain-language case

Paired t-test p-values for one-year cortical thickness change do not match the reported t statistics

Hum Brain Mapp (2009) · doi:10.1002/hbm.20740 · PMC2903209

Summary

The paper's only real-MRI validation (Section IV.B) tests one-year change in mean cortical thickness in 9 patients with Alzheimer's disease (AD) and 13 controls. It reports paired t statistics of −1.200 and −1.495 in AD (df = 8) and −0.274 and +0.175 in controls (df = 12), with p = 0.001, 0.001, 0.009 and 0.001. For those t statistics the two-sided p-values are 0.264, 0.173, 0.789 and 0.864. Table I independently gives t values of about the same size and p-values of 0.19 to 0.84. With the correct p-values none of the four within-group changes is statistically significant. That undoes the paper's statement that both methods "detect a systematic change" in both groups. It also undoes footnote 9's statement that FreeSurfer's increase in controls "is statistically significant", which the paper uses in its comparison with FreeSurfer.

What the paper reports

Section IV.B (Real MRI Data), paragraph 3, sample:

Our dataset includes pairs of scans over a one-year interval from 22 subjects (total of 44 scans), of whom 9 had been diagnosed with Alzheimer’s disease at their first scan, and 13 were age-matched normal subjects.

Section IV.B (Real MRI Data), paragraph 5, the tests in question:

Both techniques are able to detect a systematic change in cortical thickness over time, both in the AD (Alzheimer’s disease) group and in controls. Our method found thickness declined in AD (tpaired = -1.200; p=0.001), and in controls (tpaired = -0.274; p=0.009); FreeSurfer found thickness declined in AD (tpaired = -1.495; p=0.001), and increased in controls (tpaired = +0.175; p=0.001).9

Footnote 9, attached to that sentence (in the PMC text it sits in the back matter):

This increase, which is anatomically not expected, is statistically significant. No increase is found with our proposed method.

Section IV.B, paragraph 6, the baseline/follow-up correlations (the next paragraph):

Our technique gives measures that correlated highly over the 1 year time-interval in AD (p=0.001) and in controls (p=0.009), and so did FreeSurfer (p=0.001 both for AD and for normal subjects).

Section IV.B, paragraph 9, the interpretation:

Both our method and FreeSurfer report a few subjects that show increases (less pronounced with our approach), while on average our proposed method shows decline for both populations (much more significant for the AD subjects, as expected). Even so, FreeSurfer shows a net increase for the control subjects, not very large but still statistically significant. A number of reasons might explain this, including registration issues and segmentation errors, which are minimized in our proposed technique.

Table I ("Statistical Results on Real MRI Data"; relative change in mean thickness over one year), rows for the two groups (columns: proposed algorithm | FreeSurfer; spacing normalised):

Mean change for AD subjects | -1.42 % | -1.15 % Mean change for normal subjects | -0.19 % | +0.20 % SD of change for AD subjects | 3.52 | 2.40 SD of change for normal subjects | 3.39 | 2.82

The Abstract mentions this dataset without giving numbers ("demonstrate its performance on both artificial volumes and real 3D brain MRI data from subjects with Alzheimer’s disease and healthy individuals"). The Conclusion says the authors "presented reasonable results for longitudinal MRI scans of Alzheimer’s disease and normal subjects".

What the paper's own numbers show

Method. The tests compare each subject's baseline and follow-up thickness, so df = n − 1: 8 for AD and 12 for controls. I computed the two-sided p-value for each reported t from the t distribution, along with the one-sided p-value and the |t| that the reported p would require. As an independent check I rebuilt each t from Table I as mean % change / (SD / √n), with a rounding envelope of ±0.005 on the mean and SD. I also used Table I's "all subjects" rows to confirm the group sizes. All values come from recompute.py.txt (scipy.stats).

Quantity Location Reported Recomputed
Proposed method, AD: paired t, df = 8 Sec IV.B para 5 t = −1.200; p = 0.001 two-sided p = 0.264 (one-sided 0.132); p = 0.001 needs |t| ≥ 5.04
FreeSurfer, AD: paired t, df = 8 Sec IV.B para 5 t = −1.495; p = 0.001 two-sided p = 0.173 (one-sided 0.087); p = 0.001 needs |t| ≥ 5.04
Proposed method, controls: paired t, df = 12 Sec IV.B para 5 t = −0.274; p = 0.009 two-sided p = 0.789 (one-sided 0.394); p = 0.009 needs |t| ≥ 3.11
FreeSurfer, controls: paired t, df = 12 Sec IV.B para 5; footnote 9 t = +0.175; p = 0.001; "statistically significant" two-sided p = 0.864 (one-sided 0.432); p = 0.001 needs |t| ≥ 4.32
Same four t, with unlimited df (normal approximation) — — p = 0.230, 0.135, 0.784, 0.861; no choice of df gives the reported p
t from Table I, proposed method, AD (−1.42 %, SD 3.52, n = 9) Table I (text: −1.200) t = −1.210 (rounding range −1.216 to −1.204); p = 0.261
t from Table I, FreeSurfer, AD (−1.15 %, SD 2.40, n = 9) Table I (text: −1.495) t = −1.438 (−1.447 to −1.428); p = 0.189
t from Table I, proposed method, controls (−0.19 %, SD 3.39, n = 13) Table I (text: −0.274) t = −0.202 (−0.208 to −0.196); p = 0.843
t from Table I, FreeSurfer, controls (+0.20 %, SD 2.82, n = 13) Table I (text: +0.175) t = +0.256 (+0.249 to +0.263); p = 0.803
Mean change, all subjects (weighted from group rows) Table I −0.69 % / −0.35 % −0.693 % / −0.352 %
SD of change, all subjects (pooled from group rows) Table I 3.42 / 2.68 3.416 / 2.683
Group separation, (mean normal − mean AD) / SD all Table I 0.36 / 0.50 0.360 / 0.504
p-values of the t-tests vs p-values of the baseline/follow-up correlations Sec IV.B paras 5 and 6 0.001, 0.009, 0.001, 0.001 (para 5); 0.001, 0.009, 0.001, 0.001 (para 6) identical, in the same order; at n = 9 a correlation of r = 0.898 gives two-sided p = 0.001

The Table I checks confirm n = 9 and n = 13. All four printed t values are close to, but slightly outside, the range implied by Table I's rounded means and SDs. They differ from the Table I values by 0.010 and 0.058 (AD) and by 0.072 and 0.081 (controls). The paper does not say whether the tests used percent change or absolute thickness, which could account for those differences. They do not affect the conclusion: every t value the paper's numbers support gives a two-sided p between 0.17 and 0.86.

A separate check on the next paragraph found a smaller inconsistency of the same kind. It does not affect the main finding:

Quantity Location Reported Recomputed
Agreement of change between methods, all subjects (r = 0.572, n = 22) Sec IV.B para 7 p = 0.0028 two-sided p = 0.0054 (one-sided 0.0027)
Agreement of change between methods, controls (r = 0.583, n = 13) Sec IV.B para 7 p = 0.0011 two-sided p = 0.0365 (one-sided 0.0183)
Agreement of change between methods, AD (r = 0.558, n = 9) Sec IV.B para 7 p = 0.0047 two-sided p = 0.1185 (one-sided 0.0592)

Figure

Reported and recomputed p-values for the four paired t-tests of one-year thickness change

Figure 1. The four paired t-tests of one-year change in mean cortical thickness in Section IV.B, on one log-scaled p-value axis. Orange dots are the p-values printed in the paper. Filled blue dots are the two-sided p-values for the printed t statistic and df (8 for 9 AD patients, 12 for 13 controls). Hollow blue rings are the two-sided p-values for t rebuilt from Table I's mean and SD of percent change. The thin vertical line marks p = 0.05. In every row the two recomputed values agree with each other (0.17 to 0.86) and lie well above the printed value, on the other side of the p = 0.05 line. All four changes are non-significant at the conventional level.

The t-test p-values in paragraph 5 and the correlation p-values in paragraph 6 are the same four numbers

Figure 2. The left panel shows the four paired t-tests from Section IV.B paragraph 5, with the printed t, the printed p (orange) and the two-sided p for that t (blue). The right panel shows the p-values that paragraph 6 gives for the baseline/follow-up thickness correlations in the same four method-by-group cells. The printed p-values in the two paragraphs are identical, cell for cell. That identity is a fact about the text. It is consistent with, but does not prove, the explanation that the correlation p-values were entered in place of the t-test p-values (see Explanations considered).

Why it matters

The paper is mainly a methods paper. Its central contribution is defining thickness as a minimum line integral over a gray-matter probability map, together with the algorithm, the synthetic-data validation and the run time, and none of that depends on these p-values. The method-agreement correlation for all 22 subjects (r = 0.572) is also unaffected in substance.

The longitudinal AD/control dataset is the paper's only test on real MRI, and these four tests are its main quantitative result. With the corrected values:

  1. "Both techniques are able to detect a systematic change in cortical thickness over time, both in the AD … group and in controls" is not supported. Neither method detects a statistically significant one-year change in either group (p = 0.17 to 0.86 with n = 9 and n = 13). The mean changes are in the expected direction for AD. With these sample sizes and SDs of change, they do not differ significantly from zero.
  2. The comparative argument against FreeSurfer loses its statistical basis. Footnote 9 and paragraph 9 describe FreeSurfer's mean increase in controls as "statistically significant" and attribute it to "registration issues and segmentation errors, which are minimized in our proposed technique." The increase has p = 0.864 (t = +0.175), or 0.803 from Table I, and is consistent with no change. Neither method's control-group mean differs significantly from zero, so on these numbers the paper cannot conclude that one method is biased and the other is not.
  3. "Much more significant for the AD subjects" compares p-values (0.001 vs 0.009) that are not the p-values of these tests.

The descriptive content of Table I (means, SDs, group separation 0.36 vs 0.50) is internally consistent and not affected.

Explanations considered

Status

Suggested correction

  1. Replace the four p-values in Section IV.B paragraph 5 with the p-values of the paired t-tests. For the printed t statistics these are, two-sided, 0.264 (proposed method, AD), 0.789 (proposed method, controls), 0.173 (FreeSurfer, AD) and 0.864 (FreeSurfer, controls). Also state whether the tests used percent change or absolute thickness, because the printed t values differ slightly from those implied by Table I.
  2. Revise the opening sentence of paragraph 5 ("Both techniques are able to detect a systematic change … both in the AD group and in controls"), footnote 9 ("is statistically significant"), and the paragraph 9 statements "much more significant for the AD subjects" and "not very large but still statistically significant", together with the inference about FreeSurfer's registration and segmentation errors that rests on them.
  3. Check the p-values for the method-agreement correlations in paragraph 7. For r = 0.583 (n = 13) and r = 0.558 (n = 9) the two-sided values are 0.037 and 0.119, not 0.0011 and 0.0047. Also state whether the tests are one- or two-sided (the all-subjects value of 0.0028 matches a one-sided test).