Paperonium

← Back to the plain-language case

The paper calls store sales primary; the registered primary, women's diet quality, is reported as secondary

PLoS Med (2021) · doi:10.1371/journal.pmed.1003729 · PMC8423266 · NCT03518151

Summary

ClinicalTrials.gov NCT03518151 (WRAPPED1) has a single version, submitted on 2018-04-25 (658 days after the registered study start and 360 days after primary completion), which names one primary outcome, women's dietary quality score from a 20-item food frequency questionnaire (FFQ) at 3 months, and lists weekly store sales of fresh fruit and vegetables and of confectionery over 12 weeks as secondary outcomes. The published primary report reverses that order: it names "store sales (primary outcome)", calls women's and children's diets "secondary outcomes", and reports every outcome at 3 and 6 months. The paper cites the registration three times and never mentions that its outcome order differs from the registered one. The registered primary outcome is reported in Table 2 (+0.29 SD, P = 0.04, adjusted for the Index of Multiple Deprivation, IMD; +0.26 SD, P = 0.06, fully adjusted), but the abstract gives no numbers for it.

What was registered

The registry record history lists a single version. It was retrieved through the ClinicalTrials.gov history endpoint (/api/int/studies/NCT03518151/history/0) and checked against the current public API v2 record, which matches it.

Registry version Date Primary outcome Secondary outcomes
v0 (the only version) submitted 2018-04-25; first posted 2018-05-08; last update posted 2018-05-08 Dietary quality score, 3-month follow-up Child's dietary quality score (3 mo); women's daily fruit and vegetable portions (3 mo); women's weekly confectionery intake (3 mo); weekly sales of fresh fruit and vegetables (12 wk); weekly sales of confectionery (12 wk)

The registered primary outcome reads, verbatim:

Dietary quality score — "The primary outcome is women's dietary quality measured using a 20-item food frequency questionnaire (FFQ) to create a standardised dietary quality score (validated against 100-item FFQ and red cell folate)." Time frame: "3 month follow-up post intervention commencement"

The two registered store-sales outcomes read:

Weekly sales of fresh fruit and vegetables (secondary) — "Total weekly sales of fresh fruit and vegetables collected via electronic sales data". Time frame: "12 week period following intervention commencement"

Weekly sales of confectionery (secondary) — "Total weekly sales of confectionery collected via electronic sales data". Time frame: "12 week period following intervention commencement"

Registry dates (all marked "actual"): study start 2016-07-06; primary completion 2017-04-30; study completion 2017-11-30. Enrollment: 150 (actual). Because v0 is the only version, the registry primary outcome was never changed. The record was created after data collection for the primary outcome had ended. The paper agrees: it says the study "took place between April 2016 and March 2017" and that "Recruitment occurred in 3 waves between July and September 2016."

What the paper reports

The paper names store sales as its primary outcome in two places:

It aims to assess whether creating a healthier layout in discount supermarkets in England improves the healthiness of store sales (primary outcome) and the purchasing and dietary behaviours of women customers aged 18 to 45 years (secondary outcomes) after 3 and 6 months.

— Introduction, final paragraph

The data collected included 9 months continuous store sale transaction data (primary outcome) and participant loyalty card data (secondary outcome) provided by the collaborating supermarket to cover 3 time periods

In order to obtain an understanding of intervention effects on household members’ diets (secondary outcomes), interview-administered telephone questionnaires obtained information about participants’ diets and diets of their children aged 2 to 6 years (where applicable) at 3 time points: baseline (time 1) and 3 (time 2) and 6 months (time 3) following intervention commencement.

— Section 2.4, Outcome measures

The store-sales analysis covers four product groups: "Weekly sales per store of fresh fruits and vegetables, frozen vegetables, confectionery, and intervention checkout items" (Section 2.6). The abstract's quantitative results are store-sales and loyalty-card purchasing results. Its first concluding sentence is "Healthier supermarket layouts can improve the nutrition profile of store sales and likely improve household purchasing and dietary quality."

The paper mentions the registration in three places, each time without comment on its content:

The trial registration number is NCT03518151.

— Abstract, Methods and findings

ClinicalTrials.gov NCT03518151 (pre-results)

— Abstract, "Trial registration" line

it was registered with ClinicalTrials.gov (NCT03518151, pre-results).

— Section 2.1, Study design and setting

The paper describes itself as "a pilot study with a prospective matched controlled cluster design" (Section 2.1). The article text (excluding the reference list) contains none of the strings "retrospectiv", "amend", "changed", "protocol", "analysis plan", "prespecified" or "pre-specified". The word "deviat" occurs only in "standard deviation". The one statement about decisions made after data collection concerns analysis methods, not which outcome is primary:

The distributions of the data were unknown in advance of study commencement because this was a pilot study. The research team therefore made decisions about the statistical analyses following (i) completion of initial analyses to determine the shape of the outcome data; and (ii) consultation with the literature to identify appropriate statistical methods for the various outcome variables.

— Section 2.6, Statistical analysis

The discrepancy

Outcome Registered (v0, 2018-04-25) In the paper Status
Women's dietary quality score (20-item FFQ, standardised) Primary, 3-month follow-up Secondary ("household members’ diets (secondary outcomes)"); reported at 3 and 6 months (Results 3.4, Table 2) demoted; time point changed (6 months added)
Weekly sales of fresh fruit and vegetables Secondary, 12-week period after intervention start Primary ("store sales (primary outcome)"); interrupted time series at intervention, 3 and 6 months (Results 3.2) added as primary; time point changed
Weekly sales of confectionery Secondary, 12-week period after intervention start Primary (same); at intervention, 3 and 6 months added as primary; time point changed
Frozen-vegetable sales; intervention checkout-item sales Not registered Part of the store-sales (primary) analysis (Sections 2.6, 3.2) added
Child's dietary quality score Secondary, 3 months Secondary, 3 and 6 months (Table 2) matched; time point changed (6 months added)
Women's daily fruit and vegetable portions Secondary, 3 months Secondary, 3 and 6 months (Table 2) matched; time point changed (6 months added)
Women's weekly confectionery intake Secondary, 3 months Secondary, 3 and 6 months (Table 2) matched; time point changed (6 months added)
Loyalty-card purchasing (fruit and vegetables, confectionery, other categories) Not registered "participant loyalty card data (secondary outcome)" (Sections 2.4, 3.3) added
Economic outcomes (grocery spend, total store sales) Not registered Reported (Section 3.5) added

Every registered outcome is reported, so nothing is missing. The difference is in the outcome hierarchy. The only registered primary outcome is presented as secondary. Two registered secondary outcomes, together with two unregistered product groups, become the primary outcome, and a 6-month time point is added throughout.

Figures

Timeline of NCT03518151: study start, primary completion, first and only registration, and publication

Figure 1. Trial timeline. Blue items are registry dates; orange items come from the paper or the publisher's article page. The only registry version (v0) was submitted on 2018-04-25. That is 658 days after the registered study start (2016-07-06) and 360 days after the registered primary completion (2017-04-30). The orange bars show the study period (April 2016 to March 2017) and the recruitment window (July to September 2016) as the paper reports them. No later registry version exists, so the primary outcome was never changed in the registry. The paper (received 2020-10-15, accepted 2021-07-12, published 2021-09-07) appeared while v0 was still the only version.

Registered vs reported outcome hierarchy for NCT03518151

Figure 2. The registered outcome hierarchy (left, ClinicalTrials.gov v0, 2018-04-25) compared with the hierarchy in the paper (right). Each connector is labelled with what changed. The sole registered primary outcome, women's dietary quality score at 3 months, is reported as a secondary outcome (DEMOTED), and a 6-month time point is added. The registered secondary weekly sales outcomes become the paper's primary outcome (PROMOTED, in the table above "added as primary"), and the time frame changes from 12 weeks to 3 and 6 months. Loyalty-card purchasing, frozen-vegetable and checkout-item sales, and economic outcomes appear only in the paper (ADDED). The bottom panel gives the paper's results for the registered primary outcome (Table 2), with the 3-month headline store-sales results for comparison, and notes that the paper discloses nothing about the difference.

Why it matters

A primary outcome named in advance limits the choices that can be made after the data are seen: which outcome is the main test, which time point counts, and what the abstract leads with. Here the only registration was created after primary completion. The paper's hierarchy then differs from that registration and the paper does not say so. A reader who opens the cited NCT number finds a different primary outcome from the one the paper names, and the paper gives no explanation.

The two hierarchies lead to different headlines. The abstract's numerical results and its first conclusion are about store sales. Fruit and vegetable sales rose by +1.71 SD (95% CI 0.45, 2.96; P = 0.01) at 3 months and +2.42 SD (0.22, 4.62; P = 0.03) at 6 months. Confectionery sales fell by −1.05 SD (−1.98, −0.12; P = 0.03) at 3 months and −1.37 SD (−2.95, 0.22; P = 0.09) at 6 months (Abstract; Results 3.2). The registered primary outcome is reported in Results 3.4 and Table 2:

Women's diet quality score, intervention vs control, change from baseline Beta (SD) 95% CI P n
3 months, adjusted for IMD (registered time point) 0.29 0.01, 0.57 0.04 138
3 months, adjusted for IMD and DAG-selected confounders 0.26 −0.01, 0.54 0.06 132
6 months, adjusted for IMD (not a registered time point) 0.10 −0.16, 0.36 0.45 125
6 months, fully adjusted 0.10 −0.17, 0.37 0.48 119

Table 2 labels the IMD-only model "Unadjusted"; its footnote says "* Unadjusted = adjusted for IMD (to control for similarities between pairs of stores)." The Results text reports the 3-month IMD-adjusted estimate first: "After adjustment for IMD, diet quality score at 3 months follow-up increased from baseline by 0.29 SDs ((95% CI 0.01, 0.57), P = 0.04) among intervention women compared to those shopping in control stores."

So the registered primary outcome points the same way as the headline sales results. Its 95% CI excludes zero only in the IMD-adjusted model, and the 6-month estimate is close to zero. The abstract summarises it only as "Changes in dietary variables were predominantly in the expected direction for health benefit." The Discussion calls the diet findings "more equivocal". The paper also says it is "a pilot study, and, therefore, not powered to detect an effect" (Abstract). If the registered order had been kept, the abstract would have led with a borderline 3-month dietary effect rather than with store-sales effects. Readers and systematic reviewers should know which of these the trial was registered to test.

Explanations considered

There may be legitimate reasons for the paper's hierarchy, for example the store-level sales data being the most complete data source. This write-up does not assess them. The issue documented is only that the difference from the registered primary outcome is not disclosed.

Status

As of 2026-10-23:

Suggested correction

A correction or clarification would make the outcome hierarchy match the registry, or explain why it does not. It could do three things:

  1. State in the Methods (Sections 2.1 and 2.4) and in the abstract's Trial registration line that the trial was registered on 2018-04-25, after data collection, and that the registration names women's dietary quality score (20-item FFQ) at 3 months as the sole primary outcome, with weekly store sales of fresh fruit and vegetables and of confectionery over 12 weeks as secondary outcomes.
  2. Either present the registered primary outcome as primary, or say that store sales were designated primary in the paper, when this was decided and why. Also state that the 6-month time point and the frozen-vegetable, checkout-item, loyalty-card and economic outcomes were not in the registration.
  3. Give the registered primary result in the abstract with its estimate, CI and P value: 3 months, +0.29 SD (0.01, 0.57), P = 0.04 adjusted for IMD; +0.26 SD (−0.01, 0.54), P = 0.06 fully adjusted.