Paperonium

← Back to the plain-language case

Binomial p for 12 of 17 complexes is 0.072, not the reported 0.0245

eLife (2024) · doi:10.7554/eLife.93333 · PMC10942632

Summary

The paper reports that 12 of 17 protein complexes had an ROC-AUC above 0.5 and calls this "a significant excess (binomial test, p=0.0245)" against a null expectation of one half. The exact one-sided binomial p-value for 12 or more successes out of 17 at p = 0.5 is 9402/131072 = 0.0717, and the two-sided value is 0.143, so neither is significant at 0.05. The reported 0.0245 equals P(X ≄ 13), the upper tail starting one success above the observed count. The second binomial test in the same paragraph shows the same pattern: 0.0012 is reported, while P(X ≄ 4 | n = 17, p = 0.05) is 0.0088. That result remains significant. The correction removes one of the paper's three complex-level arguments for a detectable physical-interaction signal in ERC, and the corrected value is now consistent with the paper's own non-significant global permutation test (p = 0.08). The headline conclusion in the title and abstract does not depend on this test.

What the paper reports

The first binomial test, its stated null, and the second binomial test:

"Of the 17 complexes, 12 had an ROC-AUC > 0.5. Since the random expectation would be that half would exceed 0.5, this is a significant excess (binomial test, p=0.0245). Moreover, five complexes had AUCs > 0.7 (Figure 4). These results indicate that physically interacting domains tend to have a higher ERC than nonphysically interacting domains in those complexes. Likewise, four complexes had their physically interacting domains ranked significantly higher than noninteracting domains at an alpha of 5% (one-tailed Mann–Whitney U test), which is also a significant excess of complexes overall (binomial test, p=0.0012)." (Results, "ERC does not distinguish physical interactions from nonphysical interactions within a given protein complex", para 4, which opens "Of the 17 complexes, 12 had an ROC-AUC > 0.5")

The count is restated in the figure legend:

"Of the 17 complexes, 12 have an ROC-AUC > 0.5. The SEC23/24 complex (bright green) has the highest ROC-AUC at 1, and the NUP84 complex (marigold) has the lowest AUC of 0.247." (Figure 4 legend)

The global test on the same AUCs:

"We compared the true average ROC-AUC from the 17 complexes (0.596) to the permuted distribution of average ROC-AUCs, which resulted in a permutation p-value of 0.08." (Results, same subsection, para 5, which opens "We then applied a third test for a general pattern")

The Discussion relies on the 12-of-17 result:

"Looking across all complexes in our study, a proportion higher than random chance showed elevated ERC between interacting domains; this result was reflected in 12 of the 17 complexes with ROC-AUCs over 0.5 and the 4 complexes with individually significant rankings." (Discussion, para 2, which opens "We found that compensatory coevolution due to physical interaction contributes to ERC")

The abstract states the overall conclusion:

"We found no generalizable relationship between physical interaction and high ERC, although a few complexes ranked physical interactions higher than nonphysical interactions." (Abstract)

What the paper's own numbers show

Method. Under the null stated in the text, the number X of complexes with AUC > 0.5 is Binomial(n = 17, p = 0.5). The exact one-sided p-value for the observed 12 is P(X ≄ 12), which includes the observed outcome. It was computed by summing binomial coefficients exactly and checked with scipy.stats.binomtest. The same was done for the second test, X ~ Binomial(17, 0.05) with 4 observed. As an input check, the 17 per-complex AUCs printed in the version-of-record Figure 4 image were transcribed into recompute.py.txt. (The PMC text legend gives only the highest and lowest values.) They give 12 of 17 above 0.5, no values exactly at 0.5, and a mean of 0.596, matching the text. All numbers below are printed by recompute.py.txt when it is run. Paragraph numbers in this write-up are one-based, counted from the first paragraph of the named subsection, and each is given with its opening words.

Quantity Location Reported Recomputed
Complexes with ROC-AUC > 0.5 Results para 4 ("Of the 17 complexes, 12 had
"); Fig. 4 legend; Discussion para 2 12 of 17 12 of 17 (from the Fig. 4 AUCs)
Mean ROC-AUC Results para 5 ("We then applied a third test
") 0.596 0.596 (from the Fig. 4 AUCs)
Binomial p, 12 of 17 at p = 0.5, one-sided P(X ≄ 12) Results para 4 0.0245 9402/131072 = 0.0717
Same test, exact two-sided — — 0.1435
P(X ≄ 13) = P(X > 12) — — 3214/131072 = 0.02452 (equals the reported value)
Smallest count out of 17 with one-sided p < 0.05 — — 13
Binomial p, 4 of 17 significant at α = 0.05, one-sided P(X ≄ 4) Results para 4 0.0012 0.0088 (still < 0.05)
P(X ≄ 5) = P(X > 4) at p = 0.05 — — 0.00116 (equals the reported value)
Global permutation test on mean AUC Results para 5 0.08 not recomputable (needs per-complex ranked lists); consistent with the corrected 0.072
Complexes with AUC > 0.7 (side note) Results para 4 "five" 6 in the Fig. 4 AUCs (1.00, 0.821, 0.736, 0.723, 0.714, 0.713)
Complexes with physical > nonphysical mean ERC (side note) Results para 2 ("We split the proteins
") vs Table 1 12 of 17 14 of 17 in Table 1

Both reported binomial p-values equal the upper tail that starts one success above the observed count. That tail excludes the observed outcome, so it is not the p-value of the observation. For 12 of 17, the excluded term alone is P(X = 12) = 6188/131072 = 0.0472.

Figure

Reported and recomputed binomial p-values for the two complex-level tests

Figure 1. Reported p-values (orange) and exact binomial p-values recomputed from the paper's counts (blue), on one log-scaled axis; the dashed line marks α = 0.05. Top row: 12 of 17 complexes with ROC-AUC > 0.5 under the paper's stated null of one half. The reported 0.0245 falls below 0.05. The exact one-sided p (0.0717, filled blue) and two-sided p (0.143, open blue diamond) are both above it. Bottom row: 4 of 17 complexes individually significant at α = 0.05. The reported 0.0012 and the recomputed 0.0088 are both below 0.05, so only the size of that p-value changes. All values come from the paper's text or from recompute.py.txt.

Which tail of the Binomial(17, 0.5) distribution each p-value sums

Figure 2. The null distribution of the number of complexes with AUC > 0.5 out of 17, Binomial(17, 0.5). The blue bars (12 to 17) form the upper tail that includes the observed count, and they sum to the correct one-sided p = 0.0717 (blue bracket). The reported 0.0245 equals the sum over 13 to 17 only (orange bracket), leaving out the observed outcome, whose probability is 0.0472. This figure shows why a one-step shift in the tail boundary roughly triples the p-value here.

Why it matters

Results paragraphs 4 and 5 of the subsection use three complex-level tests to argue that physical coevolution contributes a detectable signal to ERC: the 12-of-17 sign test, the count of individually significant complexes, and the global permutation test. As reported, the first two are significant and the third is not, and the Discussion says the evidence "was tempered by a global permutation test". After correction, the 12-of-17 test (p = 0.072) is not significant either, so two of the three complex-wide tests fail to reach 0.05. The corrected value is in line with the permutation p of 0.08 rather than in tension with it.

Two statements depend on the 12-of-17 result as written. The Results paragraph calls it "a significant excess", and the Discussion's "a proportion higher than random chance ... this result was reflected in 12 of the 17 complexes with ROC-AUCs over 0.5" cites it as evidence. At α = 0.05 the 12-of-17 result is not a proportion higher than chance. The Discussion's broader claim of a "non-negligible signal from physical interaction" is weakened but keeps some support. The 4-of-17 count remains significant (corrected p = 0.0088), and five complexes are reported as significant in the separate proportional-rank analysis.

The headline conclusion is that ERC does not reliably predict physical interactions ("no generalizable relationship"). The error does not undermine it. If anything the corrected value makes the evidence for a general physical-interaction signal weaker, which fits that conclusion. The error is not in the title or the abstract.

Explanations considered

The count is 13, not 12. Ruled out. The text, the Figure 4 legend and the Discussion all say 12 of 17. The 17 AUCs printed in the version-of-record Figure 4 image have exactly 12 above 0.5, and their mean (0.596) matches the value in the text, so the transcription is consistent.

A tie at AUC = 0.5 was dropped (n = 16). Does not account for it. No Figure 4 AUC equals 0.5 (the closest is SMC5/6 at 0.504). Even with n = 16, P(X ≄ 12 | 16, 0.5) = 0.0384, not 0.0245.

A two-sided test. Does not account for it. The two-sided exact p is 0.143, larger than the one-sided value.

A different null proportion. Ruled out by the text, which states the expectation as "half would exceed 0.5".

A convention that defines the test as strictly greater than the observed count. No such definition appears in the Methods, which describe only the per-complex Mann–Whitney tests and the permutations. P(X > k) is not a valid p-value for an observation of k, because it excludes the observed outcome. The value 0.0245 is what an upper-tail call returns when the observed count itself is passed as the cutoff, for example R's pbinom(12, 17, 0.5, lower.tail = FALSE), which returns P(X > 12). The correct call passes 11. R was not run here; recompute.py.txt evaluates the equivalent upper-tail function, scipy.stats.binom.sf, with both cutoffs (0.02452 for 12, 0.07173 for 11). The second test reproduces the same shift exactly (0.00116 vs 0.0088), which points to one consistent computational step rather than two separate typos.

A conversion or archiving artifact. Ruled out. The eLife version-of-record web page, the version-of-record PDF, the PMC page, the Europe PMC full-text XML and the reviewed preprint v2 all give the same numbers. The only markup difference is in the second test: in the Europe PMC XML its equals sign is wrapped in italic markup, while the eLife web page displays it as "p = 0.0012". The value is the same.

The analysis used different data. This cannot be settled from the paper and does not change the finding as published. The data deposit (see Status) contains a per-complex spreadsheet. An earlier audit stage recomputed AUCs from it and found values that differ slightly from Figure 4 (for example NUP84 0.260 vs 0.247), with 14 of 17 above 0.5. Table 1 also shows physical > nonphysical mean ERC in 14 of 17 complexes, where the text says 12. A count of 14 of 17 would give P(X ≄ 14) = 0.0064. But every statement in the paper, and the published Figure 4, reports 12 of 17, and the p-value is evaluated here for that reported count. If a different count was intended, the text, the figure and the Discussion would all need to change.

Status

As of 2026-10-23:

Sources: PMC10942632 (PubMed Central archive, read with src/show.py); https://elifesciences.org/articles/93333; https://cdn.elifesciences.org/articles/93333/elife-93333-v1.pdf; reviewed preprint and reviews at https://elifesciences.org/reviewed-preprints/93333/reviews; preprint doi:10.1101/2023.10.18.562970; data deposit https://github.com/nclark-lab/erc (subfolder physical_interaction_paper, named in Methods, "Calculating ERC").

Suggested correction

In Results, "ERC does not distinguish physical interactions from nonphysical interactions within a given protein complex", para 4 (opening "Of the 17 complexes, 12 had an ROC-AUC > 0.5"), replace "this is a significant excess (binomial test, p=0.0245)" with "this excess is not statistically significant (one-sided exact binomial test, p=0.072)". Replace "(binomial test, p=0.0012)" with "(one-sided exact binomial test, p=0.0088)". In the Discussion, para 2 (opening "We found that compensatory coevolution due to physical interaction contributes to ERC"), revise "a proportion higher than random chance showed elevated ERC between interacting domains; this result was reflected in 12 of the 17 complexes with ROC-AUCs over 0.5 and the 4 complexes with individually significant rankings" so that only the 4 individually significant complexes are cited as an excess over chance, and note that the 12-of-17 count is not significant. The authors could also: