“Publication Bias in the Social Sciences: Unlocking the File Drawer”, 2014-08-28 (; backlinks):
The file drawer is full. Should we worry?: Experiments that produce null results face a higher barrier to publication than those that yield statistically-significant differences. Whether this is a problem depends on how many null but otherwise valid results might be trapped in the file drawer. et al 2014 use a Time-sharing Experiments in the Social Sciences (TESS) archive of nearly 250 peer-reviewed proposals of social science experiments conducted on nationally representative samples. They find that only 10⁄48 null results were published, whereas 56⁄91 studies with strongly statistically-significant results made it into a journal.
We studied publication bias in the social sciences by analyzing a known population of conducted studies—221 in total—in which there is a full accounting of what is published and unpublished. We leveraged Time-sharing Experiments in the Social Sciences (TESS), a National Science Foundation-sponsored program in which researchers propose survey-based experiments to be run on representative samples of American adults. Because TESS proposals undergo rigorous peer review, the studies in the sample all exceed a substantial quality threshold. Strong results are 40 percentage points more likely to be published than are null results and 60 percentage points more likely to be written up. We provide direct evidence of publication bias and identify the stage of research production at which publication bias occurs: Authors do not write up and submit null findings.
…We leverage TESS (Time-sharing Experiments in the Social Sciences), an NSF-sponsored program established in 2002 where researchers propose survey-based experiments to be run on nationally representative samples. These experiments typically embed some randomized manipulation (eg. visual stimulus, question wording difference) within a survey questionnaire. Researchers apply to TESS, which then peer reviews the proposals and distributes grants on a competitive basis38. Our basic approach is to compare the statistical results of TESS experiments that eventually got published to the results of those that remain unpublished.
This analytic strategy has many advantages. First, we have a known population of conducted studies, and therefore have a full accounting of what is published and unpublished. Second, TESS proposals undergo rigorous peer review, meaning that even unpublished studies exceed a substantial quality threshold before they are conducted. Third, nearly all of the survey experiments were conducted by the same, high-quality survey research firm (Knowledge Networks, now known as GFK Custom Research), which assembles probability samples of Internet panelists by recruiting participants via random digit dialing and address-based sampling. Thus, there is remarkable similarity across studies with respect to how they were administered, allowing for comparability. Fourth, TESS requires that studies have requisite statistical power, meaning that the failure to obtain statistically-significant results is not simply due to insufficient sample size.
…The initial sample consisted of the entire online archive of TESS studies as of January 1, 201439. We analyzed studies conducted 2002–10201212ya. We did not track studies conducted in 2013 because there had not been enough time for the authors to analyze the data and proceed through the publication process. The 249 studies represent a wide range of social science disciplines (see Table 1). Our analysis was restricted to 221 studies—89% of the initial sample. We excluded 7 studies published in book chapters, and 21 studies for which we were unable to determine the publication status and/or the strength of experimental findings40. The full sample of studies is presented in Table 2; the bolded entries represent the analyzed subsample of studies.
…Why do some researchers choose not to write up null results? To provide some initial explanations, we classified 26 detailed email responses we received from researchers whose studies yielded null results and did not write a paper (see Table S6). 15 of these authors reported that they abandoned the project because they believed that null results have no publication potential even if they found the results interesting personally (eg. “I think this is an interesting null finding, but given the discipline’s strong preference for p < 0.05, I haven’t moved forward with it”).9 of these authors reacted to null findings by reducing the priority of writing up the TESS study and focusing on other projects (eg. “There was no paper unfortunately. There still may be in future. The findings were pretty inconclusive.”). Perhaps most interestingly, two authors whose studies “didn’t work out” eventually published papers supporting their initial hypotheses using findings obtained from smaller convenience samples.