Considerable research has been carried out in the last two decades on the putative benefits of cognitive training on cognitive function and academic achievement. Recent meta-analyses summarizing the extent empirical evidence have resolved the apparent lack of consensus in the field and led to a crystal-clear conclusion: The overall effect of far transfer is null, and there is little to no true variability between the types of cognitive training. Despite these conclusions, the field has maintained an unrealistic optimism about the cognitive and academic benefits of cognitive training, as exemplified by a recent article (Green et al 2019).

We demonstrate that this optimism is due to the field neglecting the results of meta-analyses and largely ignoring the statistical explanation that apparent effects are due to a combination of sampling errors and other artifacts.

We discuss recommendations for improving cognitive-training research, focusing on making results publicly available, using computer modeling, and understanding participants’ knowledge and strategies.

Given that the available empirical evidence on cognitive training and other fields of research suggests that the likelihood of finding reliable and robust far-transfer effects is low, research efforts should be redirected to near transfer or other methods for improving cognition.

[Keywords: cognitive training, meta-analysis, methodology, working-memory training]

…It is our contention that by and large, the literature on cognitive training has underestimated the role of sampling error and other artifacts, which include issues with measurement, range restriction, and typographical errors, among others. Specifically, many researchers assume that distinct types of interventions will have different effects on far transfer—some interventions will have a positive effect, and others will not. But this is a hypothesis that researchers can test empirically while keeping in mind that the variability in results could be in reality artifactual. We tested this hypothesis in the meta-analyses and second-order meta-analysis that we discuss below and found that the hypothesis is incorrect empirically: The variability is artifactual. Thus, beyond random fluctuations, there are no differences between the different types of intervention: Their effect on far-transfer tasks is null when sampling error, publication bias, and type of control group are taken into account. We get the same results when meta-analyses are carried out within one domain (eg. action video games vs. nonaction video games) or between domains (ie. the second-order meta-analysis comparing the effects of WM training, video-game playing, etc.). Thus, rather than limiting researchers to piecemeal conclusions (eg. Intervention 1 does not lead to far transfer; Intervention 2 does not lead to far transfer), we show that it is possible to reach a conclusion that applies to the broad category of cognitive training. Reaching broad generalizations supported by empirical evidence is the hallmark of scientific progress (Braithwaite1960; Chow1987).

…What do meta-analyses tell researchers about cognitive training? As noted above, we have carried out several meta-analyses about cognitive training.⁴ We have repeatedly found that the true far-transfer effect-size, when estimated from the comparison of treatment versus active control group, is close to zero. This outcome has been found for WM training (Aksayli et al 2019a; Sala et al 2019b; Sala & Gobet2020b), video-game playing (Sala et al 2018), exergames (Sala et al 2021), and music training (Sala & Gobet2017c, Sala & Gobet2020a, Sala & Gobet2020b). The exception is chess (Sala & Gobet2016), for which too few studies with an active control group have been carried out; however, the few available studies with an active control group suggest a lack of far transfer (eg. Sala & Gobet2017a).

…Thus, the meta-analyses allowed us to quantify, with respect to far-transfer effects, the extent to which the literature is mixed and could explain any between-studies true variance. An important conclusion was that the results are not inconsistent and thus do not depend on differences in methodologies between researchers. That is, once baseline differences were controlled for, the only appreciable source of true variance (which is often quite low) is the type of control group. In other words, the debate about the literature being mixed and the results inconsistent is just much ado about nothing. Far-transfer effects do not exist. Cognitive-training researchers seem to incorrectly equate sampling-error variance and true variance: Terms such as “τ²”, “true variance”, or “true heterogeneity” rarely appear in cognitive-training reviews. In addition, it seems that cognitive-training researchers fail to understand that it is absolutely normal that statistically-significantly positive effects are sometimes found (eg. when comparing treatment groups with active control groups on far-transfer measures) even if the true effect is zero. Specifically, by chance, we expect a portion (5%) of the measurements to be statistically-significant (p < 0.05, one-tailed). Effect sizes in a given literature are mathematically bound to differ because of sampling error. Variability across and within the studies is the rule, not the exception.

A step further: second-order meta-analysis: Second-order meta-analysis is a procedure designed by Schmidt & Oh2013 for integrating findings of first-order (ie. conventional) meta-analyses. This technique estimates a grand mean of the first-order overall effect sizes and, most notably, the between-meta-analyses true variance. Second-order meta-analysis represents the current highest level of cumulative knowledge in quantitative research.

In Sala et al 2019c, we applied second-order meta-analysis to cognitive-training data (for results about far transfer, see Table 1 and Table 2). The analysis included 14 statistically independent first-order meta-analyses (332 samples, 1,555 effect sizes, and 21,968 participants) of near-transfer & far-transfer effects in different populations (eg. children, adults, and older adults). As shown in Table 1/Table 2, the training programs covered were WM training, action-game and nonaction-video-game training, music training, chess training, and exergame training.

The key results were as follows. First, near transfer occurs even when placebo effects are controlled for and seems to be moderated by the age of the participants. Second, far transfer is negligible (uncorrected overall effect) or null (when placebo effects and publication bias are ruled out). Third, within-studies (ω²) and between-studies true variance (τ²) are small to null with far transfer. Fourth, second-order sampling error (ie. the residual sampling error from first-order meta-analyses) explains all the between-meta-analyses variance with far transfer. That is, we found no evidence of either within-studies, between-studies, or between-meta-analyses true variance.

These results strongly corroborate the idea that although near transfer is real and the magnitude of its effect is moderated by the population examined, the observed far transfer is due to factors that are unspecific (ie. it occurs regardless of the type of training regimen or population), such as placebos. (This conclusion is buttressed by the results of Kassai et al 2019, who carried out a meta-analysis on training components of children’s executive functions skills, a type of training not covered by our second-order meta-analysis.)

…Finally, note that our meta-analyses do not show that placebo effects occur in all cognitive-training programs. For example, they are not present in either action-game or nonaction-video-game training (Sala et al 2018). However, we did find that placebos always occur in WM training when it comes to far transfer (Sala & Gobet2020b). These placebos are around 0.15 to 0.20 standardized mean difference at best and often affected by publication bias…we estimated a small publication-bias effect (0.05–0.10 standardized mean differences).

…Another common incorrect argument relies on the negative correlation occurring between far-transfer pretest scores and pretest/posttest gains. This correlation is sometimes presented as evidence of an individual-based compensatory effect (eg. Karbach et al 2015). Put simply, a given cognitive-training regimen is believed to be particularly effective for individuals who performed poorly at baseline assessment (ie. Subject × Treatment interaction). However, such negative correlations are likely to be, at least in part, statistical artifacts due to regression to the mean (Smoleń et al 2018). Therefore, correlations between pretest/posttest gains and pretest scores alone cannot be considered as evidence for true individual differences in training-induced transfer effects.

…The need for detailed analyses and computational models: …There is thus an urgent need to understand which cognitive mechanisms might lead to cognitive transfer. As we showed above in the section on meta-analysis, the available evidence shows that the real effect size of cognitive training on far transfer is zero. Prima facie, this outcome indicates that theories based on general mechanisms, such as brain plasticity (Karbach & Schubert2013), primitive elements (Taatgen2013), and learning to learn (Bavelier et al 2012), are incorrect when it comes to far transfer. We reach this conclusion by a simple application of modus tollens: (1) Theories based on general mechanisms such as brain plasticity, primitive elements, and learning to learn predict far transfer. (2) The empirical evidence shows that there is no far transfer. Therefore, (3) theories based on general mechanisms such as brain plasticity, primitive elements, and learning to learn are incorrect.

…The Broader View: As discussed earlier, our meta-analyses clearly show that cognitive training does not lead to any far transfer in any of the cognitive-training domains that have been studied. In addition, using second-order meta-analysis made it possible to show that the between-meta-analyses true variance is due to second-order sampling error and thus that the lack of far transfer generalizes to different populations and different tasks. Taking a broader view suggests that our conclusions are not surprising and are consistent with previous research. In fact, they were predictable. Over the years, it has been difficult to document far transfer in experiments (Singley & Anderson1989; Thorndike & Woodworth1901), industrial psychology (Baldwin & Ford1988), education (Gurtner et al 1990_34ya), and research on analogy (Gick & Holyoak1983), intelligence (Detterman1993), and expertise (Bilalić et al 2009). Indeed, theories of expertise emphasize that learning is domain-specific (Ericsson & Charness1994; Gobet & Simon1996; Simon & Chase1973). When putting this substantial set of empirical evidence together, we believe that it is possible to conclude that the lack of training-induced far transfer is an invariant of human cognition (Sala & Gobet2019).

Obviously, this conclusion conflicts with the optimism displayed in the field of cognitive training, as exemplified by Green et al 2019’s article discussed above. However, it is in line with skepticism recently expressed about cognitive training (Moreau2021; Moreau et al 2019; Simons et al 2016). It also raises the following critical epistemological question: Given that the overall evidence in the field of cognitive training strongly suggests that the postulated far-transfer effects do not exist, and thus the probability of finding such effects in future research is very low, should one conclude that the reasonable course of action is to stop performing cognitive-training research on far transfer?

See Also: