“Inconvenient Truths and the Usefulness of Identifying Unknown Unknowns”, 2022-08-30 (; backlinks):
We studied how the sex of human experimenters affected mouse behaviors and brain functions under normal conditions and in the context of ketamine administration. Identifying such unknown unknowns was critical to understanding how, specifically and quantitatively, they affected experimental outcomes, which led to fresh insight into ketamine’s mechanism as an antidepressant drug.
…In mice, ketamine administration has been shown to produce antidepressant-like effects similar to those of other antidepressant drugs.
Our lab was having difficulty replicating this effect that we had previously shown in mice2. Our earlier findings were achieved with a male experimenter administering ketamine, whereas in these seemingly identical non-replicating studies a female experimenter was administering ketamine. Being familiar with earlier work identifying that the sex of the human experimenter can affect stress and behavioral responses in rodents3, we began to ponder whether our replication issue was related to the sex of the human experimenter, but frankly, did not consider this to be likely
…Our observation is likely to be the tip of an iceberg. There are undoubtedly many unknown unknowns, perhaps of equal importance to the known knowns, that are influencing experimental outcomes4. An inconvenient truth is that the inability to replicate experimental results between and sometimes within laboratories may be due to unrecognized experimental variables that are not controlled. Whereas the known knowns can be accounted for, unknown unknowns are also leading to non-replicating results.
…Behind The Paper: “Prior to this project my primary experimental focus was not ketamine pharmacology. My main field of interest is neuroendocrinology, but it was not in my plans to investigate the differential effects of experimenters’ sex on mouse behaviors. I, a female, noticed the lack of a typical ketamine behavioral response when I was filling in for a male postdoc. We first decided to test possible effects of sex of the experimenter mainly out of curiosity. Following our early results my PI initially discouraged me from extensively pursuing the characterization, but I wanted to know the mechanistic answer and help others in science facing replicability issues. Some people may have chuckled and questioned my scientific approach when asking to swab their skin or for donation of their clothing, but seeing the completed project I have the sense of satisfaction for making an important though unexpected scientific contribution.” —Polymnia Georgiou
See Also:
Publication Bias in Reports of Animal Stroke Studies Leads to Major Overstatement of Efficacy
Genetic Variation in the Social Environment Contributes to Health and Disease
What Can We Learn from Many Labs Replications? 3. Can replication studies detect fraud?
Evaluating the replicability of social science experiments in Nature and Science 2010–52015