“The Piranha Problem: Large Effects Swimming in a Small Pond”, 2021-04-08 (; backlinks; similar):
In some scientific fields, it is common to have certain variables of interest that are of particular importance and for which there are many studies indicating a relationship with a different explanatory variable. In such cases, particularly those where no relationships are known among explanatory variables, it is worth asking under what conditions it is possible for all such claimed effects to exist simultaneously.
This paper addresses this question by reviewing some theorems from multivariate analysis that show, unless the explanatory variables also have sizable effects on each other, it is impossible to have many such large effects. We also discuss implications for the replication crisis in social science.
…The implication of the claims regarding ovulation and voting, shark attacks and voting, college football and voting etc, is not merely that some voters are superficial and fickle. No, these papers claim that seemingly trivial or irrelevant factors have large and consistent effects, and this runs into the problem of interactions. For example, the effect on your vote of the local college football team losing could depend crucially on whether there’s been a shark attack lately, or on what’s up with your hormones on election day. Or the effect could be positive in an election with a female candidate and negative in an election with a male candidate. Or the effect could interact with your parents’ socioeconomic status, or whether your child is a boy or a girl, or the latest campaign ad, or any of the many other factors that have been studied in the evolutionary psychology and political psychology literatures.
Again, we are not saying that psychological factors have no effect on social, political, or economic decision making; we are only arguing that such effects, if large, will necessarily interact in complex ways. Similar reasoning has been used to argue against naive assumptions of causal identification in economics, where there is a large literature considering rainfall as an instrumental variable, without accounting for the implication that these many hypothesized causal pathways would, if taken seriously, represent violations of the assumption of exclusion restriction (2020).
In this work, we demonstrate that there is an inevitable consequence of having many explanatory variables with large effects: the explanatory variables must have large effects on each other. We call this type of result a “piranha theorem” (2017), the analogy being the folk wisdom that if one has a large number of piranhas (representing large effects) in a single fish tank, then one will soon be left with far fewer piranhas (2021). If there is some outcome on which a large number of studies demonstrate an effect of a novel explanatory variable, then we can conclude that either some of the claimed effects are smaller than claimed, or some of the explanatory variables are essentially measuring the same phenomenon.
There are a multitude of ways to capture the dependency of random variables, and thus we should expect there to be a correspondingly large collection of piranha theorems. We formalize and prove piranha theorems for correlation, regression, and mutual information in §2 & §3. These theorems illustrate the general phenomena at work in any setting with multiple causal or explanatory variables. In §4, we examine typical correlations in a finite sample under a simple probabilistic model.
…For example, an influential experiment from 1996 reported that participants were given a scrambled-sentence task and then were surreptitiously timed when walking away from the lab ( et al 1996). Students whose sentences included elderly-related words such as “worried”, “Florida”, “old”, and “lonely” walked an average of 13% more slowly than students in the control condition, and the difference was statistically-significant.
This experimental claim is of historical interest in psychology in that, despite its implausibility, it was taken seriously for many years (for example, “You have no choice but to accept that the major conclusions of these studies are true” (2011)), but it failed to replicate ( et al 2013) and is no longer generally believed to represent a real effect; for background see et al 2015. Now we understand such apparently statistically-significant findings as the result of selection with many researcher degrees of freedom ( et al 2011).
Here, though, we will take the published claim at face value and also work within its larger theoretical structure, under which weak indirect stimuli can produce large effects.
An effect of 13% on walking speed is not in itself huge; the difficulty comes when considering elderly-related words as just one of many potential stimuli. Here are just some of the factors that have been presented in the social priming literature as having large effects on behavior: hormones (male and female), subliminal images, the outcomes of recent football games, irrelevant news events such as shark attacks, a chance encounter with a stranger, parental socioeconomic status, weather, the last digit of one’s age, the sex of a hurricane name, the sexes of siblings, the position in which a person is sitting, and many others.
A common feature of these examples is that the stimuli have no clear direct effect on the measured outcomes, and in most cases the experimental subject is not even aware of the manipulation. Based on these examples, one can come up with dozens of other potential stimuli that fit the pattern. For example, in addition to elderly-related words, one could also consider word lengths (with longer words corresponding to slower movement), sounds of words (with smooth sibilance motivating faster walking), subject matter (sports-related words as compared to sedentary words), affect (happy words compared to sad words, or calm compared to angry), words related to travel (inducing faster walking) or invoking adhesives such as tape or glue (inducing slower walking), and so on. Similarly, one can consider many different sorts of incidental events, not just encounters with strangers but also a ringing phone or knocking at the door or the presence of a male or female lab assistant (which could have a main effect or interact with the participant’s sex) or the presence or absence of a newspaper or magazine on a nearby table, ad infinitum.
Now we can invoke the piranha theorem. Suppose we can imagine 100 possible stimuli, each with an effect of 13% on walking speed, all of which could arise in a real-world setting where we encounter many sources of text, news, and internal and external stimuli. If the effects are independent, then at any given time we could expect, on the log scale, a total effect with standard deviation 0.5 × √100 × log(1.13) = 0.61, thus walking speed could easily be multiplied or divided by e0.61 = 1.8 based on a collection of arbitrary stimuli that are imperceptible to the person being affected. And this factor of 1.8 could be made arbitrarily large by simply increasing the number of potential primes.
It is ridiculous to think that walking speed could be randomly doubled or halved based on a random collection of unnoticed stimuli—but that is the implication of the embodied cognition literature. It is basically a Brownian motion model in which the individual inputs are too large to work out.