Overestimation of Action-Game Training Effects: Publication Bias and Salami Slicing

We first reproduced the estimates from Bediou et al. (2018a) and then applied all the adjustments for publication bias described in Table 1. We also probed the robustness of the moderation by laboratory (Bavelier and non-Bavelier) and examined whether that moderation could be explained by differences in training duration or differences in publication bias. Data and code are available athttps://osf.io/dhejx/.

Dependency between outcomes within studies. One challenge of adjusting for publication bias in this meta-analysis is that studies have multiple outcomes. Simple meta-analytic models assume a single effect size per study; a more sophisticated model is required to handle multiple effect sizes per study. BAMTGB did so using robust variance estimation (Hedges, Tipton, & Johnson, 2010), a method for modeling and estimating the dependence between outcomes within studies. BAMTGB applied the PET and PEESE adjustments in concert with this robust variance estimation. However, the trim-and-fill adjustment is incompatible with robust variance estimation, so the authors computed their trim-and-fill estimate by first averaging across outcomes within a study and then applying the adjustment. We used the same approach in our PET, PEESE, and trim-and-fill adjustments to reproduce their analyses.

The p-uniform and 3PSM adjustments (Table 1) are incompatible with robust variance estimation and with averaging outcomes within studies. Both methods assume that studies are either published or censored based on a single primary outcome’sp value rather than on the average p value across outcomes. These models are expected to perform poorly when using the average of p values (Simonsohn, Nelson, et al., 2014), so the standard recommendation is to choose the single most appropriate outcome from each study.

Unfortunately, we cannot know which outcome is the most appropriate, nor can we know which outcome or outcomes were subjected to selection bias. We considered creating a specification curve (Simonsohn, Simmons, & Nelson, 2015), running p-uniform and selection modeling on every possible choice of outcome per study. This approach was infeasible, though, as it would require more than 1.5 billion analyses. Instead, we chose to use bootstrapping, which randomly samples from this space of 1.5 billion possible analyses. In this way, bootstrapping gives all outcomes equal weight, which is an appropriate compromise when a single primary outcome cannot be identified. We bootstrapped 1,000 samples to explore the variability in the estimates caused by the choice of outcome from each study. 95% CIs are defined as the 2.5^th and 97.5^th percentile of bootstrapped estimates.

Table 2 summarizes the results of our analyses with and without adjustment for publication bias using the updated data set provided in the erratum. (See the supplement for a sensitivity analysis using other, less appropriate combinations of bias adjustment and study-level aggregation methods.) We reproduced the unadjusted random-effects, PET, and PEESE estimates of the main effect of game training calculated in the authors’ R Markdown document provided in the online data repository (MA_Intervention_erratum.Rmd). Their PET and PEESE calculations mistakenly used the “weights” rather than “modelweights” argument, and we corrected that error in our code. (The differences in the estimates were minor. Other small differences may result from our testing of single moderators rather than multiple moderators.) The unadjusted random-effects estimate was significant and positive (g = 0.34 [0.07, 0.61]), but both PET and PEESE yielded significant and negative estimates for the erratum dataset (PET: g = –1.69 [–2.44, –0.94]; PEESE:g = –0.53 [–0.95, –0.11]) just as they did in the original analysis (Bediou et al., 2018a, Figure S7).

BAMTGB chose to interpret these negative PET and PEESE results as indicating problematic and eccentric behavior of the PET and PEESE estimators. However, ourp-uniform and 3PSM adjustments were also consistent with a substantial effect of publication bias, with both estimating small and nonsignificant effects (p-uniform, g = 0.11, [–0.78, 0.69]; selection modeling, g = 0.16 [–0.09, 0.42]).

None of these estimates, on its own, is necessarily accurate or precise. Most have wide confidence intervals (especially p-uniform). Furthermore, many involve a tradeoff between bias and variance, with some minimizing variance but retaining bias (trim and fill) and others reducing bias in favor of noisier estimates. Nevertheless, these estimates collectively indicate a serious and problematic degree of publication bias and the possibility of no effect of game training on cognition. Although we cannot say for certain that there is no benefit, given limited support for transfer of cognitive training more generally (Melby-Lervåg et al., 2016), we feel the burden of proof rests on those claiming efficacy of game training to show that such effects are not an artifact of publication bias.

Lab effect. The overall effect estimate reported by BAMTGB was qualified by a large lab effect: Research from Bavelier laboratory yielded substantially larger effects (g = 0.92 [0.76, 1.08],p < .001) than studies by other groups (g = 0.22 [–0.01, 0.45], p = .054). BAMTGB reported a moderation analysis and concluded that the lab effect likely resulted from the longer training durations used in their own studies. The weak moderation effect in the original report was no longer statistically significant in the updated data set provided in the erratum (b = 0.016 [–.004, .036], p = .089). Differences in training durations do not, therefore, explain the pronounced laboratory effect.

Although a lab effect might result from differences in methods and research design, it also could result from differences in publication practices. We tested this possibility by including the PET bias adjustment in a multiple regression predicting effect size from the lab producing the effect. If the lab effect is caused by differences in research design, then including the PET bias adjustment in the regression should either have little effect on the estimated lab effect or it should strengthen it by removing noise. If the lab effect is caused in part by differences in publication bias, however, then including the PET adjustment for publication bias in the regression will reduce the size of the estimated lab effect by removing explained variance.

Adding the PET adjustment dramatically reduced the laboratory effect fromb = 0.72, t(4.28) = 7.35,p = .001, ω² = 0.096,τ² = 0.027 without adjustment tob = 0.20, t(5.36) = 1.85,p = .119, ω² = 0.060,τ² = 0 with adjustment. Much of the laboratory effect might be attributable to differences in publication bias. Adding the PET adjustment also accounted for all of the observed variability between study-level effect sizes (i.e., the between-cluster variability,τ²), meaning that after adjusting for small study biases, the variation in the observed effect sizes between studies can be explained by sampling error alone. Taken together, these analyses suggest that the larger effects in studies reported by Bavelier laboratory might result from greater publication bias, not differences in training duration or other design differences.

One might argue that the Bavelier lab results are accurate and that it is the non-Bavelier labs that have underestimated the true effect through selection bias in favor of the null. This seems unlikely given that small-study effects suggestive of bias against the null are detected in the non-Bavelier-lab studies as well. The Egger test among non-Bavelier-lab studies is significant, b = 5.79,t(4.3) = 5.04, p = .006, and the PET-adjusted estimate is negative, g = –1.89,t(4.54) = –3.98, p = .013. Thus, while the lab effect suggests greater overestimation among Bavelier-lab studies, studies by other laboratories also show evidence of publication bias and overestimation. This widespread publication bias indicates serious weakness in the evidence for game training benefits.

Might differences in reporting practices underlie the apparent difference in publication bias between the Bavelier lab studies and other studies? In this section, we examine how one practice—distributing outcomes from the same study across multiple articles—might account for much of the reported lab effect and lead to an inflated overall estimate of the benefits of game training.

Contemporary standards for transparency, such as those from the Office of Research Integrity, require clear documentation whenever data from the same study are reported in multiple publications:

Salami slicing can contribute to overestimated effects in meta-analysis. If a meta-analysis treats the outcomes reported in each paper as if they were independent of those reported in other papers, then salami slicing of outcomes across papers will lead to a single intervention counting multiple times. That is, the analysis will treat these distributed outcomes as if they were independent intervention studies, implying that there was more evidence than actually exists. And, If those papers happened to report some of the larger effect sizes, the overall effect size estimate will be further inflated. Salami slicing is particularly damaging when individual studies collect many outcomes and only significant outcomes are published, a practice that constitutes a form of p-hacking (Simmons, Nelson, & Simonsohn, 2011). Censoring of non-significant outcomes biases the meta-analytic estimate upwards, and when combined with salami slicing, each significant outcome is weighted even more heavily into the overall meta-analytic estimate, amplifying the effects of publication bias.

Independent of the statistical hazards, undisclosed salami slicing might increase the likelihood that a result will be published at all by misleading editors and reviewers. A manuscript presented as a new trial might be evaluated more favorably than one reporting outcomes collected in a previously-reported study.

To what extent might salami slicing contribute to publication bias and the reported lab effect in studies of video game training? Prior to the publication of the BAMTGB meta-analysis, two of us had noted the possibility that data from one trial might have been published in separate articles without clear documentation. We called for more transparency in the extent of overlap (Boot, Blakely, & Simons, 2011). For example, several sets of articles reported experiments with highly similar training methods (same games, same durations, same game improvements), with each reporting only a small number of outcomes.

The original BAMTGB analysis clustered outcome measures from the same study together to account for their non-independence, but they did not cluster outcomes across papers. Although BAMTGB did note that some unspecified papers were not entirely independent, their analysis treated outcomes from different papers as if they were always from different interventions with different participants. In response to an earlier version of this manuscript that we sent to the BAMTGB authors, they issued an erratum that re-analyzed the data by assigning dependent groups of participants across papers to the same cluster. This revised clustering identifies overlapping samples, providing an opportunity to determine which original studies featured undisclosed overlapping samples.

We compared the clusters in the corrected BAMTGB dataset against the original BAMTGB dataset. In addition, we carefully read the published articles and compared their methods and results, looking for similarities. This analysis was aided by an early draft of the BAMTGB erratum, posted publicly on the Open Science Framework, which we quote from below.² This early draft included a section labeled “Subject Overlap” which identified overlaps and documented the extent of those overlaps (complete in some cases, partial in others). This “subject overlap” section was removed in the published version of the erratum, which instead only stated: “In the original publication, cases of partial overlap were treated as independent. A more conservative approach is to code these effects as dependent, which is done here.”

The corrected dataset clustered together a number of papers that had been treated as independent in the original meta-analysis, all of which were among the Bavelier lab subset of studies. To our knowledge, this re-clustering is the first time this overlap has been addressed: none of the primary articles reported the full set of outcomes collected, nor did these articles report that other outcome measures from the same participants were previously reported in other published work.

The earlier draft of the erratum text identified three papers that reported outcomes from the same sample, saying “data from the same subjects (different tasks) were reported in more than one study and should thus have been assigned the same cluster” (Bediou et al., 2018b, p. 1). These articles separately report effects of action game training on visual acuity (Green & Bavelier, 2007), multiple object tracking (Green & Bavelier, 2006a, Cognition), and UFOV (Green & Bavelier, 2006b, JEP:HPP). None of these articles mentioned that the reported outcomes and results came from the same sample.

The “Subject Overlap” section also noted other broad, unreported overlap across multiple papers:

Based on the erratum draft and our own readings of the papers, we document here several additional sets of overlapping papers. The first set involves three papers reporting four studies of action game training effects on backward masking (Li, Polat, Scalzo, & Bavelier, 2010, Study 3), contrast sensitivity (Li, Polat, Makous, & Bavelier, 2009, Studies 2 and 4), and visual motion discrimination (Green, Pouget, & Bavelier, 2010, Study 3). The erratum indicates that the three Li et al. studies are drawn from subsamples of the Green et al. (2010) experiment. These overlaps are not described in the original articles. (Green et al., 2010, supplement p. 15 mentions “subjects underwent 50 hours of training as well as several experiments unrelated to the ones at hand,” but does not indicate what those experiments were or whether “experiments” referred to outcome measures from the intervention that were reported in other publications.)

Another overlapping set, noted in the initial erratum draft, includes outcome measures of task-switching (Green, Sugarman, Medford, Klobusicky, & Bavelier, 2012) and seeing the orientation of a line under visual noise (Bejjanki et al., 2014). The 2014 publication makes no mention of overlap with the 2012 publication. The 2012 paper notes: “Subjects completed two experimental blocks [of task switching]…as well as several other tasks unrelated to the current paper (e.g., motion discrimination, visual search, contrast detection – however note: the data presented here was acquired over the course of 3 separate training studies – and thus the unrelated tasks are not identical in all subjects)” (Green et al., 2012, pg. 992). However, it is unclear which three separate training studies are included or where those outcomes are reported. One possibility is that those outcomes had been reported in earlier papers that also reported the results of 30 hours of training with either Unreal Tournament/Call of Duty or Sims 2. Such effects of training on contrast sensitivity were reported in Li et al. (2009, Study 4), and effects on motion discrimination were reported in Green et al. (2010, Study 3). If these are the outcomes alluded to in Green et al. (2012), this partial overlap was not documented in the first draft of the erratum, was not accounted for by re-clustering in the updated dataset, and is not mentioned in the original papers. (The sample sizes differ across these articles, so the extent of possible overlap is unknown.)

Training interventions are expensive and time consuming, so most such studies include batteries of outcome measures. For example, the ACTIVE trial of cognitive training in older adults collected 10 proximal outcomes, 6 primary outcomes, and 5 secondary outcomes (see Jobe et al., 2001).

How many outcomes are reported in typical action game training articles? Those from Bavelier and colleagues report an average of 1.6 outcomes (SD = 0.8) whereas papers from other laboratories average 4 outcomes per experiment (SD = 2.5; see Figure 1). Collecting only one or two outcome measures in an intervention seems unlikely, and even 4 seems low.

Why so few outcomes? One possibility is that that more outcomes were collected, but those outcomes that did not show a statistically significant difference in improvement between the game training and control conditions were not reported. This selection bias among outcomes within studies would cause overestimation of the effect size. Since the Bavelier lab group reports a greater average improvement per outcome, but fewer outcomes per paper, it is possible that the lab effect could be explained by differences in reporting practices.

Another possibility is that relatively few outcomes are reported per published study because some outcomes are salami-sliced for later publications. If we reconstruct Figure 1 using the number of outcomes collected per cluster in the updated dataset rather than the number reported for each paper, the count from Bavelier and colleagues is closer to that reported in papers by other labs (M = 3.6,SD = 1.82) (Figure 2).

In the absence of trial registration or disclosure statements about the number of outcomes (e.g., Simmons, Nelson, & Simonsohn, 2012), it is impossible to know how many outcomes were collected but not reported. It is possible that many of these intervention studies, not just those from the Bavelier lab, collected more outcomes than were actually reported in published articles. That sort of underreporting could account for the observed publication bias within the studies by labs other than the Bavelier lab as well.

BAMTGB acknowledged the possibility of publication bias but relied on the trim-and-fill adjustment to conclude in favor of significant benefits from action game training. However, trim and fill is inadequate as an adjustment for the substantial publication bias that meta-regression techniques suggest is present (Carter et al., 2019). We applied additional adjustments for publication bias and observed meta-analytic results consistent with a small effect or no effect of action game training on intervention outcomes. That conclusion is consistent across multiple adjustment methods: PET and PEESE estimates were negative, consistent with a small or null effect and strong bias, and bootstrapped p-uniform and selection modeling yielded small, non-significant results.

Many of the largest effect sizes in published evidence for benefits from action-game training come from papers authored by the Bavelier laboratory. In our analysis, the strong lab effect reported by BAMTGB is consistent with differences in publication bias and is not explained by differences in training duration. An earlier draft of the published erratum for the BAMTGB paper documented how various papers from the Bavelier lab reported outcomes from overlapping sets of participants. Unfortunately, the published version of the erratum removed that “subject overlap” section. The overlap that we document above, combined with the small number of outcomes reported in each paper, suggest selective publication of results from each intervention. It is unclear how many additional outcome measures, particularly non-significant ones, were collected but not reported.

In addition to the issues of salami slicing, the erratum draft suggests other issues that potentially undermine the evidence provided by these studies. First, data collection apparently used a form of rolling recruitment over different time periods, with different outcome measures tested during these periods (Bediou et al., 2018b). This form of study design involves a degree of flexibility that makes it difficult to evaluate what the published results mean. For example, if the completed tasks differed across subjects, it is possible that members of the treatment and control conditions performed different sets of tasks. If so, then the effect of the intervention on any individual outcome measure is confounded by differences in the other tasks completed by those participants. Failure to disclose that recruitment procedure and differences between the tasks completed by different participants could mislead readers and reviewers about the intervention (i.e., participants were not randomly assigned to groups that differed only in the intervention they completed). According to best practices for intervention design, the tasks and outcomes should be identical between the intervention and control group, and the target sample size should be predetermined.

An accurate estimate of the benefits of action video games requires a more complete accounting of the collected samples and outcomes. Future studies should fully report all outcome measures and explicitly describe any overlap with previous studies. For already published research, we hope that the BAMTGB authors will expand on the earlier draft of the erratum and provide a more complete accounting of the overlap among results reported across separate papers: How many fully-independent intervention studies have been conducted and what outcomes were collected in each? If samples reported in multiple papers overlapped partially or completely, which participants contributed to each outcome? We also hope other authors will document the results for any unreported outcome measures from their own intervention studies. Without such an accounting, the statistical inferences reported in the published papers are effectively uninterpretable.

BAMTGB’s conclusion in favor of training benefits appears premature in the face of substantial publication bias, uncertainty about the number of distinct, independent interventions among the meta-analyzed results, and the potential censoring of outcomes that did not yield significant results. Although game training might transfer to other cognitive tasks, the studies synthesized in BAMTGB do not support that conclusion; the meta-analyzed data are also consistent with a combination of substantial publication bias and no effect of action games on cognition. Greater transparency in the reporting of interventions and of outcomes within interventions is needed.

Data and code are available at https://osf.io/dhejx/.

Special thanks to John Sakaluk for consultation regarding treatment of dependent effect size estimates. Thanks also to Dan Ispas, Olivia Cody, and Christopher Engelhardt for reading drafts of the commentary. JH, DS, WB, and GS conceptualized, wrote, and edited this commentary. JH and GS performed the statistical analyses. WB and DS documented the possible overlap across clusters reported in the meta-analysis.

GS is a JSPS International Research Fellow (Award #17F17313).

The authors have no competing interests to declare.

The author(s) of this paper chose the Open Review option, and the peer review comments are available at: http://doi.org/10.1525/collabra.231.pr