“Maximal Positive Controls: A Method for Estimating the Largest Plausible Effect Size”, 2021-03-01 (; similar):
Some reported effect sizes are too big for the hypothesized process.
Simple, obvious manipulations can reveal which effects are too big.
A demonstration is provided examining an implausibly large effect.
Effect sizes in social psychology are generally not large and are limited by error variance in manipulation and measurement. Effect sizes exceeding these limits are implausible and should be viewed with skepticism. Maximal positive controls, experimental conditions that should show an obvious and predictable effect [eg. a Stroop effect], can provide estimates of the upper limits of plausible effect sizes on a measure.
In this work, maximal positive controls are conducted for 3 measures of aggressive cognition, and the effect sizes obtained are compared to studies found through systematic review. Questions are raised regarding the plausibility of certain reports with effect sizes comparable to, or in excess of, the effect sizes found in maximal positive controls.
Maximal positive controls may provide a means to identify implausible study results at lower cost than direct replication.
[Keywords: violent video games, aggression, aggressive thought, positive controls, scientific self-correction]
[Positive controls eliciting a hitherto-maximum effect can be seen as a kind of empirical Bayes estimating the distribution of plausible effects: if a reported effect size exceeds the empirical max, either something extremely unlikely has occurred (a new max out of n effects ever observed) or an error. For large n, the posterior probability of an error will be much larger.]