Paperonium

← Back to the plain-language case

Results text swaps the two condition means for tic frequency, reversing the reported reduction

Curr Biol (2020) · doi:10.1016/j.cub.2020.04.044 · PMC7308804

Summary

In study 3 (16 people with Tourette syndrome), the Results say that rhythmic median nerve stimulation (MNS) "significantly reduced" tic frequency. The same sentence then gives mean tic counts of 87.6 ± 71.4 for no stimulation and 126.3 ± 94.5 for rhythmic MNS, which as printed is a 44.2% increase. The processed data the paper deposits on OSF give exactly these two values with the conditions the other way round: 87.625 ± 71.41 under MNS and 126.25 ± 94.49 without stimulation, a 30.6% reduction. The direction stated in the abstract, highlights, Figure 5 and Discussion is therefore supported, and the error is a label swap in one sentence. Recomputing the tests from the same deposited data turns up a separate problem: the printed t values (2.36, 2.41, 1.83) cannot be reproduced (1.99, 2.05, 1.41). The tic-frequency and tic-intensity effects reach p < 0.05 only under the one-tailed tests the Methods specify.

What the paper reports

Results, "Effect of Rhythmic MNS on the Occurrence of Tics and the Urge-to-Tic", second paragraph:

Quantitative analysis revealed that compared with comparable time periods of no stimulation, rhythmic MNS significantly reduced both tic frequency (i.e., total number of tics recorded over a total period of 4 min) and tic intensity (i.e., the independently, blind, rated intensity of each tic) (means for tic frequency: no stimulation = 87.6 ± 71.4, rhythmic MNS = 126.3 ± 94.5; t(15) = 2.36, p = 0.03, Figure 5 left; means for tic intensity: no stimulation = 3.0 ± 0.6, rhythmic MNS = 2.8 ± 0.6; t(15) = 2.41, p = 0.03, Figure 5 middle).

The same paragraph, next sentence:

quantitative analysis of the continuous self-estimated urge-to-tic data showed an overall reduction in the self-estimated urge-to-tic during MNS of ∼33%, the difference between rhythmic MNS and no stimulation was only marginally significant (means for self-estimated tic-to-tic intensity: no stimulation = 41.3 ± 31.7, rhythmic MNS = 29.1 ± 20.7; t(15) = 1.83, p = 0.09, Figure 5 right).

The same paragraph, on the correlation:

These analyses revealed a positive correlation (r = 0.75, R2 = 0.56, F = 12.74, p = 0.005), indicating that a reduction in tic frequency was associated with a reduction in self-estimated urge-to-tic experiences

Abstract:

it nonetheless leads to a large reduction in tic frequency and tic intensity in individuals with TS.

Highlights:

Mu-band entrainment significantly reduces motor and vocal tics in Tourette syndrome

Discussion, fourth paragraph:

This quantitative analysis clearly demonstrated that both tic frequency and tic intensity were significantly reduced by rhythmic 10 Hz MNS compared with no stimulation

STAR Methods, "Quantification and Statistical Analysis":

In all four studies, statistical analyses were performed using one-tailed paired-samples t tests.

STAR Methods, "Data and Code Availability":

Processed data of tic frequencies, tic intensities, and urge intensities are available on OSF (https://osf.io/zw93x/).

What the paper's own numbers show

Method. First I took the printed means, SDs and t values at face value and worked out the direction of change, the relative change, and the p values that t(15) implies. Then I opened the processed data the paper deposits on OSF (Study3_data.xlsx, dated 2020-04-17). It has one row per participant (n = 16) and three outcome columns, TICS, INTENSITY and SLIDER, each split into "Rhythmic" and "No Stimulation". From it I recomputed the condition means and SDs, the paired t tests (one- and two-tailed), a Wilcoxon signed-rank test, and the correlation reported in the same paragraph. The means and SDs I recomputed equal the AVERAGE and SD rows stored in the spreadsheet itself. recompute.py.txt holds the outcome values only, without participant codes or demographic columns, and prints every number below.

Quantity Location Reported Recomputed
Tic frequency, no stimulation Results, study 3, para 2 87.6 ± 71.4 126.25 ± 94.49 (OSF data, n = 16)
Tic frequency, rhythmic MNS same 126.3 ± 94.5 87.625 ± 71.41 (OSF data)
Change, MNS minus no stimulation same ("significantly reduced") +38.7 tics as printed, a 44.2% increase −38.6 tics, a 30.6% reduction (OSF data; also (126.3 − 87.6)/126.3 = 30.6% with the printed labels exchanged)
Medians, no stimulation vs MNS OSF "median" row — 105 vs 67.5; fewer tics under MNS in 12 of 16 participants
Tic intensity, no stim vs MNS Results, study 3, para 2 3.0 ± 0.6 vs 2.8 ± 0.6 2.97 ± 0.56 vs 2.76 ± 0.56 (consistent; labels correct)
Urge-to-tic, no stim vs MNS Results, study 3, para 2 41.3 ± 31.7 vs 29.1 ± 20.7, df 15 43.06 ± 31.67 vs 29.10 ± 20.73, n = 13 (3 participants have no slider data); reduction 32.4% (text: "∼33%")
p from the printed t, df 15 Results, study 3, para 2 p = 0.03, 0.03, 0.09 two-tailed 0.032, 0.029, 0.087; one-tailed 0.016, 0.015, 0.044
Paired t, tic frequency same t(15) = 2.36 t(15) = 1.99; one-tailed p = 0.032, two-tailed p = 0.065
Paired t, tic intensity same t(15) = 2.41 t(15) = 2.05; one-tailed p = 0.029, two-tailed p = 0.058
Paired t, urge-to-tic same t(15) = 1.83 t(12) = 1.41; one-tailed p = 0.092, two-tailed p = 0.184
Wilcoxon signed-rank, tic frequency not reported — one-sided p = 0.022, two-sided p = 0.044
SD of paired differences, tic frequency implied by printed t 65.6 (= 38.7 × √16 / 2.36) 77.5
Correlation of reductions, tics vs urge Results, study 3, para 2 r = 0.75, R² = 0.56, F = 12.74, p = 0.005 r = 0.748, R² = 0.560, F(1,10) = 12.74, p = 0.0051 (proportional reductions; n = 12, as one no-stimulation urge value is 0)

Three points follow from the table.

  1. The tic-frequency means are attached to the wrong conditions. The printed pairs 87.6 ± 71.4 and 126.3 ± 94.5 match the deposited "Rhythmic" (87.625 ± 71.41) and "No Stimulation" (126.25 ± 94.49) values to every printed digit, but with the labels exchanged. Pairing cannot reverse the sign of a mean difference, so the printed values describe an increase and cannot support "significantly reduced". The tic-intensity and urge values carry the correct labels.
  2. The deposited file is the dataset the paper analysed. It reproduces the printed correlation exactly (r = 0.75, R² = 0.56, F = 12.74, p = 0.005), the "∼33%" urge reduction (32.4%), and the printed SDs of all six condition columns.
  3. The printed t values do not follow from the deposited data. Paired t tests on the deposited data give 1.99, 2.05 and 1.41, against the printed 2.36, 2.41 and 1.83. The printed p values (0.03, 0.03, 0.09) match the one-tailed p values of the recomputed tests (0.032, 0.029, 0.092), which is what the Methods specify. The same printed p values also happen to match the two-tailed p values of the printed t (0.032, 0.029, 0.087). As a numerical observation, and not as an account of how they arose, each printed t equals, to two decimals, the t whose two-tailed p equals the one-tailed p of the deposited data (2.36 and 2.41 at df 15, 1.83 at df 12). For the urge measure the printed degrees of freedom (15) also differ from the 12 that the 13 complete pairs allow, and the printed no-stimulation mean (41.3) differs from the deposited 43.06, although the SD (31.7) matches.

Figure

The printed tic-frequency means show an increase under MNS; the deposited data show the same two numbers as a reduction

Figure 1. Mean tic count per participant (total over the 4 min of video analysed), drawn as an arrow from the no-stimulation condition (open circle) to the rhythmic MNS condition (filled square). Top row (orange): the values printed in the Results text, which put MNS 38.7 tics, or 44.2%, above no stimulation. Bottom row (blue): the same quantities recomputed from the processed data deposited on OSF (n = 16), which give the same two numbers with the conditions exchanged, so MNS is 38.6 tics, or 30.6%, below no stimulation. Labels show mean ± SD. The axis starts at zero.

Printed t statistics are larger than those recomputed from the deposited data, which clear the one-tailed but not the two-tailed 5% threshold

Figure 2. Paired t statistics for the three study 3 outcomes: as printed in the Results (orange), and recomputed from the deposited data with a paired t test (blue). The dashed and dotted lines mark the one-tailed and two-tailed p = 0.05 critical values at df 15. The urge-to-tic data have 13 complete pairs (df 12), with critical values 1.78 and 2.18. Recomputed, the tic-frequency and tic-intensity effects clear only the one-tailed threshold, which is the test the Methods specify. The axis starts at zero, the null value.

Why it matters

Explanations considered

Status

Suggested correction

In the Results, section "Effect of Rhythmic MNS on the Occurrence of Tics and the Urge-to-Tic", replace

means for tic frequency: no stimulation = 87.6 ± 71.4, rhythmic MNS = 126.3 ± 94.5

with

means for tic frequency: no stimulation = 126.3 ± 94.5, rhythmic MNS = 87.6 ± 71.4

The authors may also wish to check the three t statistics in the same paragraph against the deposited data. Paired tests on Study3_data.xlsx give t(15) = 1.99 for tic frequency, t(15) = 2.05 for tic intensity and t(12) = 1.41 for the urge-to-tic (13 complete pairs), with one-tailed p = 0.032, 0.029 and 0.092. They may also wish to check the urge-to-tic no-stimulation mean (printed 41.3; deposited 43.06). Stating "one-tailed" next to each p value in the Results would help readers.