--- title: "Regression To A Mean Fallacies" description: "Regression-to-a-mean is a general statistical phenomenon which leads to several widespread fallacies in analyzing & interpreting statistical results, such as ‘residual confounding’ and ‘Lord’s paradox’." created: 2021-05-20 modified: 2021-05-20 thumbnail: /doc/statistics/probability/2024-09-05-gwern-meme-butitcouldworkforus-regressiontoamean.jpg status: finished confidence: certain importance: 5 css-extension: dropcaps-yinit ... [Regression to a mean](!W): If you select something highly unusual in one way, it will probably be more usual in other ways, because stuff is usually more usual than unusual. This has many implications, like ["regression fallacy"](!W), but it also leads to additional errors, particularly when combined with measurement error: - **Residual Confounding**: ["Statistically Controlling for Confounding Constructs Is Harder than You Think"](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152719), Westfall & Yarkoni 2016 ([Phillips & Smith 1992](/doc/statistics/causality/1992-phillips.pdf "'Bias in relative odds estimation owing to imprecise measurement of correlated exposures")/[Phillips & Smith 1991](/doc/statistics/causality/1991-phillips.pdf "How independent are ‘independent’ effects? Relative risk estimation when correlated exposures are measured imprecisely")/[Smith et al 1992](/doc/statistics/causality/1992-smith.pdf "Smoking as ‘independent’ risk factor for suicide: illustration of an artifact from observational epidemiology?"), [Lawlor et al 2004a](https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(04)16925-0/fulltext "Observational versus randomized trial evidence")/[Lawlor et al 2004b](/doc/statistics/causality/2004-lawlor.pdf "Those confounded vitamins: what can we learn from the differences between observational versus randomized trial evidence?"), [Smith et al 2007](https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0040352 "Clustered Environments and Randomized Genes: A Fundamental Distinction between Conventional and Genetic Epidemiology")/[Fewell et al 2007](https://academic.oup.com/aje/article/166/6/646/89040 "The Impact of Residual and Unmeasured Confounding in Epidemiologic Studies: A Simulation Study")/[Smith 2011](https://www.dcscience.net/Davey-Smith-2011.pdf "Epidemiology, genetics and the 'Gloomy Prospect': embracing randomness in population health research and practice"), [Pingault et al 2021](https://journals.plos.org/plosgenetics/article?id=10.1371/journal.pgen.1009590 "Genetic sensitivity analysis: Adjusting for genetic confounding in epidemiological associations"))^[This is part of why results in sociology/epidemiology/psychology are so unreliable: [everything is correlated](/everything) but not only do they usually not control for genetics at all, they don't even control for the things they think they control for! You have *not* controlled for SES by throwing in a discretized income variable measured in one year plus a discretized college degree variable. Variables which correlate with or predict some outcome such as poverty, may be doing no more than correcting some measurement error (frequently, due to the heavy genetic loading of most outcomes---["the environment is genetic"](/doc/genetics/heritable/adoption/2016-plomin.pdf#page=10 "'Top 10 Replicated Findings From Behavioral Genetics', Plomin et al 2016")---correcting the omission of genetic information). This is why within-family designs are desirable even without worries about genetics: they hold constant shared-environment factors so you don't need to measure or model them. Even a structural equation model (SEM) which explicitly incorporates measurement error may still have enough leakage to render 'controlling' misleading. Such confounding where the highly-imperfect correlations drive pseudo-causal effects (which are just regression to a mean) are doubtless a reason why so many apparently-well-controlled & highly-replicable correlations [fail in RCTs](/correlation "'How Often Does Correlation=Causality?', Gwern 2014").] Example: [Parker et al 2021](/doc/sociology/2021-parker.pdf "Maternal Judgments of Child Numeracy and Reading Ability Predict Gains in Academic Achievement and Interest"). - ["Impossibly hungry judges"](https://daniellakens.blogspot.com/2017/07/impossibly-hungry-judges.html), Lakens - A newer twist on residual confounding is to include a polygenic score, which typically measures a small fraction of all genetic influences on many outcomes & environmental measures (usually much less than half of the genetic variance), and declare that "\[all\] genetics have been controlled for" and proceed to interpret all remaining model coefficients as purely-environmental causal variables, and all remaining group differences as various kinds of societal bias or discrimination or environmental/nurture (eg. [Cuevas et al 2021](https://www.pnas.org/doi/10.1073/pnas.2017224118 "Discrimination and anxiety: Using multiple polygenic scores to control for genetic liability"), [Harden et al 2021](https://www.biorxiv.org/content/10.1101/598532.full "'Genetic Associations with Mathematics Tracking and Persistence in Secondary School', Harden et al 2019"), [Engzell et al 2020](https://sociologicalscience.com/articles-v7-10-242/ "It’s All about the Parents: Inequality Transmission across Three Generations in Sweden"), [Lin 2020](/doc/iq/2020-lin.pdf "The social and genetic inheritance of educational attainment: Genes, parental education, and educational expansion"), [Barnes et al 2019](/doc/genetics/heritable/2019-barnes.pdf "The Propensity for Aggressive Behavior and Lifetime Incarceration Risk: A Test for Gene-environment Interaction (G×E) Using Whole-genome Data"), [Sauce et al 2022](https://www.nature.com/articles/s41598-022-11341-2 "The impact of digital media on children’s intelligence while controlling for genetic differences in cognition and socioeconomic background")). An example of correctly using PGSes (using the [Pingault et al 2021](#pingault-et-al-2021) method to extrapolate the known incompleteness of the PGS to full heritability) is [Baldwin et al 2022](/doc/psychiatry/2022-baldwin.pdf "A genetically informed Registered Report on adverse childhood experiences and mental health"). - [**Attenuation**](https://en.wikipedia.org/wiki/Regression_dilution):^[There are countless examples of incorrect interpretations of measured variables which are imperfectly correlated with their latent variables, requiring explicit correction for measurement error or range restriction, particularly when meta-analyzed (see [Hunter & Schmidt 2004](/doc/statistics/meta-analysis/2004-hunterschmidt-methodsofmetaanalysis.pdf "'Methods of Meta-Analysis: Correcting Error and Bias in Research Findings', Hunter & Schmidt 2004")); for example, it is particularly common for researchers to claim that their favorite new trait (SES, personality, "emotional intelligence", etc.) correlates more with an outcome than IQ, without noting that their sample has been selected to be high on IQ, or that their IQ measure has much more random error in it than their alternative (sometimes with an excuse about "second-order sampling error" like [Allen et al 2020](https://onlinelibrary.wiley.com/doi/abs/10.1002/sej.1377 "What matters more for entrepreneurship success? A meta-analysis comparing general mental ability and emotional intelligence in entrepreneurial settings")), and so it is unsurprising yet uninformative if the raw correlation coefficient may be larger because the IQ correlate was biased towards zero much more heavily. This sort of argument-from-attenuated-variables is wrong, but doesn't become a regression-to-the-mean fallacy until combined with something else.] - ["Evaluating the Effect of Inadequately Measured Variables in Partial Correlation Analysis"](/doc/statistics/bias/1936-stouffer.pdf), Stouffer 1936 - ["Gifted Today But Not Tomorrow? Longitudinal Changes in Ability and Achievement in Elementary School"](https://files.eric.ed.gov/fulltext/EJ746292.pdf), Lohman & Korb 2006 (Challenges in gifted education in elementary or earlier: IQ scores are unstable and so regression to a mean implies that few children in G&T programs will grow up to be gifted); [_Genius Revisited_ Revisited](/hunter "‘𝐺𝑒𝑛𝑖𝑢𝑠 𝑅𝑒𝑣𝑖𝑠𝑖𝑡𝑒𝑑 Revisited’, Gwern 2016") (early childhood IQ is measured with great error, and so extremely-high-IQ elementary schools select much less high IQ adults and correspondingly unimpressive results, in contrast to [later selection](/smpy "'SMPY Bibliography', Gwern 2018")); ["To Understand Regression From Parent to Offspring, Think Statistically"](/doc/statistics/bayes/regression-to-mean/1978-humphreys.pdf), Humphreys 1978 - ["Regression Fallacies in the matched groups experiment"](/doc/statistics/bias/1942-thorndike.pdf), Thorndike 1942 - ["Control of Spurious Association and the Reliability of the Controlled Variable"](/doc/statistics/bias/1965-kahneman.pdf "'Control of spurious association and the reliability of the controlled variable', Kahneman 1965"), Kahneman 1965; ["Nuisance Variables and the Ex Post Facto Design"](/doc/statistics/bias/1970-meehl.pdf), Meehl 1970 - **Kelley’s Paradox** (cf. [Lord’s paradox](!W){#lords-paradox}): the Roman poet Terence noted that "When two do the same, it isn't the same."; when we have prior knowledge about score distributions, 2 identical scores may have different implications because they will be shrunk differently by regression to a mean, and this will be stronger the more extreme the scores are & the larger the prior differences. (If measurements are not corrected and their predictive accuracy is reduced by leaving them as 'raw', this may manifest in the real world as [statistical discrimination](!W "Statistical discrimination (economics)") as optimizing agents learn to implicitly correct and 'discriminate'.) This frequently comes up in standardized testing & exams, where one of the first to point out the implications was [Truman Lee Kelley](!W): - [_Statistical Method_](/doc/statistics/order/1923-kelley.pdf), Kelley 1923; [_Interpretation of Educational Measurements_](/doc/iq/1927-kelley-interpretationofeducationalmeasurements.pdf), Kelley 1927; [_Fundamentals of Statistics_](https://archive.org/details/in.ernet.dli.2015.263082/page/n421/mode/2up), Kelley 1947 - [Longitudinal]{.smallcaps}: ["Interpreting regression toward the mean in developmental research"](/doc/statistics/bias/1973-furby.pdf), Furby 1973; ["Lord’s Paradox in a Continuous Setting and a Regression Artifact in Numerical Cognition Research"](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3994136/ "'Lord’s paradox in a continuous setting and a regression artifact in numerical cognition research', Eriksson & Häggström 2014"), Eriksson & Häggström 2014; ["Allocation to groups: Examples of Lord’s paradox"](/doc/statistics/bayes/regression-to-mean/2019-wright.pdf "'Allocation to groups: Examples of Lord's paradox', Wright 2019"), Wright 2019 - ["The relevance of group membership for personnel selection: A demonstration using Bayes’ theorem"](/doc/statistics/bayes/regression-to-mean/1994-miller.pdf "'The Relevance of Group Membership for Personnel Selection: A Demonstration Using Bayes' Theorem', Miller 1994"), Miller 1994 - ["Kelley’s Paradox"](/doc/statistics/bayes/regression-to-mean/2000-wainer.pdf), Wainer 2000 - ["Three Statistical Paradoxes in the Interpretation of Group Differences: Illustrated with Medical School Admission and Licensing Data"](/doc/statistics/bayes/regression-to-mean/2006-wainer.pdf), Wainer & Brown 2006^[The draft version is "Two statistical paradoxes in the interpretation of group differences: Illustrated with medical school admission and licensing data".] - ["Measurement Error, Regression to a mean, and Group Differences"](https://humanvarieties.org/2017/07/01/measurement-error-regression-to-the-mean-and-group-differences/) (eg. [selective attrition from college majors](https://link.springer.com/article/10.1186/2193-8997-1-5 "‘What happens after enrollment? An analysis of the time path of racial differences in GPA and major choice’, Arcidiacono et al 2012") differs by group, and so a measurement like "has an bachelor degree" means different things by group) - **[Winner’s curse](!W)**: ["The Optimizer's Curse: Skepticism and Postdecision Surprise in Decision Analysis"](/doc/statistics/decision/2006-smith.pdf), Smith & Winkler 2006 (decisions with different accuracies/measurement-error will suffer different regression to a mean and be biased towards the most-overestimated options) - ["Predicting the Next Big Thing: Success as a Signal of Poor Judgment", Denrell & Fang 2010](/doc/statistics/prediction/2010-denrell.pdf "'Predicting the Next Big Thing: Success as a Signal of Poor Judgment', Denrell & Fang 2010") constructs a scenario where regression is so severe that being in the tail constitutes evidence for true below-average accuracy - [**Dunning-Kruger Effect**](!W): the famous Dunning-Kruger effect can be caused by regression to a mean from measurement error, due to floor/ceiling effects in measured performance vs expressed confidence; "backfire effects" can also be manufactured this way ([Swire-Thompson et al 2020](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7462781/ "Searching for the Backfire Effect: Measurement and Design Considerations")) - **Placebo Effects**: [just regression](https://www.dcscience.net/2015/12/11/placebo-effects-are-weak-regression-to-the-mean-is-the-main-reason-ineffective-treatments-appear-to-work/ "Placebo effects are weak: regression to a mean is the main reason ineffective treatments appear to work") [to a mean](https://slatestarcodex.com/2018/01/31/powerless-placebos/ "Powerless Placebos") (on [Hróbjartsson et al 2010](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7156905/ "Placebo interventions for all clinical conditions")) after all? 'Nocebo' effects also suffer from this critique (eg. in sports statistics, the [supposed harm of status](/doc/statistics/bias/2009-jordet.pdf "When Superstars Flop: Public Status and Choking Under Pressure in International Soccer Penalty Shootouts") like the ["_Madden_ curse"](!W "Madden NFL#Madden Curse") or the '_Forbes_ cover' effect) - [_A Primer on Regression Artifacts_](https://www.amazon.com/Primer-Regression-Artifacts-Donald-Campbell/dp/1572308591), Campbell & Kenny 1999 - [**Replication Crisis**](/replication "'The Replication Crisis: Flaws in Mainstream Science', Gwern 2010"): because systemic biases like _p_-hacking filter for the most extreme outliers^[Including but not limited to researcher malpractice; eg. the use of "genome-wide statistical-significance" to filter GWAS hits ensures a "winner's curse", and (contra critics) [hits replicate as expected](https://journals.plos.org/plosgenetics/article?id=10.1371/journal.pgen.1006916 "'Statistical correction of the Winner’s Curse explains replication variability in quantitative trait genome-wide association studies', Palmer & Pe'er 2017") given their [statistical power](!W)+regression-to-the-mean.], published effect sizes will predictably decline over time, with additional replication, and with better methodologies - **[Baader-Meinhof Effect](!W)**: [Diaconis & Mosteller 1989](/doc/statistics/bias/1989-diaconis.pdf#page=9 "Methods for Studying Coincidences") propose that one reason that a rare word may seems to abruptly appear repeatedly is simply that for rare words which are not seen for the most unusually over-due period, the duration for the next appearance will be more ordinary and it'll re-appear 'quickly' - **See Also**: [Second product syndrome](!W), [Order statistics: The Probability of a Double Maximum](/order-statistic#probability-of-bivariate-maximum "'Calculating The Gaussian Expected Maximum § Probability of Bivariate Maximum', Gwern 2016"), the [James-Stein estimator](!W) ([Efron & Morris 1977](/doc/statistics/bayes/regression-to-mean/1977-efron.pdf "Stein's Paradox in Statistics: The best guess about the future is usually obtained by computing the average of past events. Stein's paradox defines circumstances in which there are estimators better than the arithmetic average")/[Stigler 1990](/doc/statistics/bayes/regression-to-mean/1990-stigler.pdf "The 1988 Neyman Memorial Lecture: A Galtonian Perspective on Shrinkage Estimators"))