Paperonium

← Back to the plain-language case

Abstract says gender-stereotype thumbnails increased with engagement; Results, Table 2 and Discussion report a decrease

JAMA Netw Open (2024) Β· doi:10.1001/jamanetworkopen.2024.13855 Β· PMC11137630

Summary

The abstract reports that, over 20 sequential engagements with recommended videos, "the gender stereotypes feature increased with more engagement in the recommendations feed (P for trend < .001)". The body reports the opposite direction for the same feature and the same test. The Results say gender stereotypes "decreased (P for trend < .0001)", the Discussion says prevalence "decreased for gender stereotypes", and Table 2 gives a negative Spearman correlation with screenshot number (ρ = βˆ’0.08), which with the paper's 2880 thumbnails corresponds to t = βˆ’4.31 and P = 1.7 Γ— 10⁻⁡. The feature that did increase was visual loudness, which the Results report with the same P for trend (< .001) that the abstract gives. The trend over screenshots is how the study tests its stated hypothesis (that recommendations drift toward more problematic content), so a reader of the abstract alone gets the direction of that result reversed.

What the paper reports

Abstract, Results paragraph (final sentence):

Contrary to the hypotheses, problematic feature prevalence did not increase over time, but the gender stereotypes feature increased with more engagement in the recommendations feed (P for trend < .001).

Results, section "Trends Over 20 Screenshots":

Both Spearman correlations and tests of trend demonstrated that visual loudness increased over the 20 screenshots (P for trend < .001), while gender stereotypes decreased (P for trend < .0001) (Figure).

Discussion, first paragraph:

Contrary to our hypothesis, the prevalence of most codes did not increase over time when following recommendations over 20 sequential engagements; prevalence only increased slightly for visual loudness and decreased for gender stereotypes.

Table 2 ("Spearman Correlations Between Content Codes Among 2880 Thumbnails"), column "Screenshot No. (1-20)":

Visual loudness | .09 (<.001)

Gender stereotypes | -0.08 (<.001)

Methods, "Statistical Analysis", which defines both trend analyses:

To examine whether the frequency of content features changed over the course of 20 sequential screenshots, we conducted Spearman correlations between screenshot number (1 through 20) and each feature. We also dichotomized each feature as present (codes of levels 1 or 2) vs not present and tested significance of trend over 20 screenshots with the Cochran-Armitage trend test.

Introduction, the hypothesis the trend analysis tests:

We hypothesized that the prevalence of problematic thumbnail features would increase over time with sequential engagement with recommended videos (ie, approximating going down the rabbit hole).

The coding levels are defined in the Methods as "0 = absent, 1 = mild or latent, and 2 = clear or dominant", so a higher code means the feature is more present, and a positive correlation with screenshot number means more of the feature at later screenshots.

What the paper's own numbers show

Method. The paper does not publish screenshot-level counts (the Data Sharing Statement says "Data available: No"), so the Cochran-Armitage statistic itself cannot be recomputed. The direction can be checked in three ways from what is published. (1) I compared the direction stated in each place the paper reports the trend. (2) I converted the Table 2 correlation into a two-sided P value with the usual t approximation, t = ρ√((n βˆ’ 2)/(1 βˆ’ ρ²)), at n = 2880 (the table's stated denominator) and at n = 2574 (the number of unique thumbnails), and computed a Fisher-z 95% interval. Because ρ is printed to two decimals and Spearman ties are ignored, these values are approximate, but the sign is not affected. (3) As a supporting check only, I read by eye the level-0 (absent) bar heights in the paper's Figure, panels E (gender stereotypes) and A (visual loudness), converted them to counts of "present" thumbnails out of 144 per screenshot (12 search terms Γ— 12 thumbnails), and ran a Cochran-Armitage test. Those readings are approximate (about Β±1–2 percentage points per bar). I also checked the prevalence counts against Supplement 1, eTable 1. Every number below is printed by recompute.py.txt.

Quantity Location Reported Recomputed from the paper's own numbers
Direction of gender stereotypes trend Abstract, Results "increased" (P for trend < .001) decrease (opposite sign)
Direction of gender stereotypes trend Results, Trends Over 20 Screenshots "decreased" (P for trend < .0001) decrease (agrees)
Direction of gender stereotypes trend Discussion, first paragraph "decreased" decrease (agrees)
ρ, screenshot no. vs gender stereotypes Table 2 βˆ’0.08 (<.001) t = βˆ’4.31; P = 1.7 Γ— 10⁻⁡ (n = 2880), 4.8 Γ— 10⁻⁡ (n = 2574); 95% CI βˆ’0.116 to βˆ’0.044
ρ, screenshot no. vs visual loudness Table 2 .09 (<.001) t = +4.85; P = 1.3 Γ— 10⁻⁢; 95% CI +0.054 to +0.126 (an increase)
P for trend in the abstract's sentence Abstract vs Results < .001 matches the Results' P for visual loudness (< .001), not gender stereotypes (< .0001)
Other four features, ρ with screenshot no. Table 2 βˆ’0.01 to 0.02, P = .29 to .73 all 95% CIs include 0 (no trend)
Gender stereotypes, prevalence Abstract, Table 1 525 (18.2%); 387 at level 1, 138 at level 2 387 + 138 = 525; 525/2880 = 18.2%; eTable 1 per-term counts also sum to 387 and 138
Gender stereotypes present, screenshots 1–5 vs 16–20 Figure, panel E (read by eye) plotted only about 25.0% vs 17.0%; Cochran-Armitage z = βˆ’4.27, P β‰ˆ 2 Γ— 10⁻⁡ (a decrease)
Visual loudness present, screenshots 1–5 vs 16–20 Figure, panel A (read by eye) plotted only about 70.4% vs 81.4%; z = +5.94 (an increase; checks the reading method)

All six prevalence totals in the abstract equal the Table 1 level 1 + level 2 sums, so the abstract's prevalence figures are sound; the problem is confined to the direction of the gender stereotypes trend.

A separate, minor inconsistency: the Results give drama and intrigue for the search term Roblox as "218 of 240 thumbnails [96.7%]". 218/240 is 90.8%. Supplement 1, eTable 1 lists Roblox drama and intrigue as 8 at level 0, 115 at level 1 and 117 at level 2, so 232 of 240 thumbnails (96.7%) had the feature. The percentage is right and the count in the text is wrong. It does not affect any conclusion.

Figure

The abstract says gender stereotypes increased; the Results, Table 2 and Discussion say they decreased, and the Table 2 correlation recomputes to a significant decrease

Figure 1. The conflicting statements about the same feature and the same trend test. Left (orange outline): the abstract's sentence, which says gender stereotypes increased (P for trend < .001). Right (grey outline): the Results, Table 2 and Discussion, which all say or show a decrease. The double arrow marks the conflict; the note under it records that the abstract's P value is the one the Results give for visual loudness. Bottom (blue outline): the two-sided P implied by the Table 2 correlation, recomputed by recompute.py.txt; the negative sign means fewer gender-stereotype thumbnails at later screenshots. All quoted text is verbatim from the article as archived in PMC.

Spearman correlations of each content feature with screenshot number from Table 2, with recomputed 95% intervals: visual loudness is the only clear increase and gender stereotypes the only clear decrease

Figure 2. The six correlations with screenshot number (1–20) printed in Table 2 (orange dots), each with a 95% interval recomputed by recompute.py.txt (blue lines; Fisher z, n = 2880). The vertical line marks no trend (ρ = 0). Only two features have intervals that exclude 0: visual loudness, above 0 (more at later screenshots), and gender stereotypes, below 0 (fewer at later screenshots). The abstract describes the gender stereotypes trend as an increase. Intervals use the two-decimal ρ printed in the table, so they are approximate.

Why it matters

Explanations considered

Status

As of 2026-10-24:

Sources: the article in PMC (PMC11137630) and on the publisher's site; Europe PMC full text and supplementary files; PubMed; Crossref REST API.

Suggested correction

In the Abstract, Results paragraph, replace

Contrary to the hypotheses, problematic feature prevalence did not increase over time, but the gender stereotypes feature increased with more engagement in the recommendations feed (P for trend < .001).

with wording consistent with the Results, Table 2 and the Discussion, for example:

Contrary to the hypotheses, problematic feature prevalence did not increase over time; the gender stereotypes feature decreased with more engagement in the recommendations feed (P for trend < .0001), and the visual loudness feature increased (P for trend < .001).

In the Results, "Attention-Capture Designs", the Roblox count for drama and intrigue should read 232 of 240 thumbnails (96.7%), as in eTable 1, rather than 218 of 240.