We examine how the evaluation of research studies in economics depends on whether a study yielded a null result.

…In the main experiment, we present participants with 4 hypothetical vignettes that are based on actual research studies but modified for the purposes of the experiment. For each of the 4 vignettes, we randomize whether the point estimate of the main treatment effect was sizable and statistically-significant or close to zero and not statistically-significant. We keep the standard error of the main finding constant across treatments such that null findings are not associated with a lower statistical precision of the estimates. To examine how the evaluation of research studies depends on expert priors (DellaVigna et al 2019), we cross-randomize whether the vignette includes an expert forecast of the treatment effect. For vignettes including an expert forecast, we further randomize whether the experts predict a null or a non-null result. To examine whether a potential penalty for null results depends on the communication of the statistical uncertainty of the main result, we cross-randomize at the respondent level whether the statistical precision of the main finding is communicated in terms of p-values or the standard error of the estimate. Finally, to obfuscate the purpose of the experiment, we further cross-randomize a series of other salient study characteristics, including the seniority of the research team and their university affiliations.

Studies with null results are perceived to be less publishable, of lower quality, less important, and less precisely estimated than studies with statistically-significant results, even when holding constant all other study features, including the precision of estimates. The null result penalty is of similar magnitude among PhD students and journal editors. The penalty is larger when experts predict a large effect and when statistical uncertainty is communicated with p-values rather than standard errors.

[Keywords: null results, publication bias, learning, information, scientific communication]

…Main treatment effects: Table 1 shows the effects of the null result treatment on our main outcomes of interest. As shown in column 1, respondents assigned to the null result treatment indicate that the studies have a 14.1 percentage point lower probability of being published (95% C.I. [−16.2,−11.9]; p < 0.001). This effect-size corresponds to a 24.9% reduction in perceived publication chances.

Column 2 shows the effects on private beliefs about the quality of the studies. Respondents in the null result treatment associate the studies with 37.3% of a standard deviation lower quality (95% C.I. [−49.6,−25]; p < 0.001). Furthermore, as shown in column 3, respondents in the null result treatment similarly think that other researchers would associate the studies with 46% of a standard deviation lower quality (95% C.I. [−58.1,−33.8]; p < 0.001).

Columns 4 and 5 also show sizable treatment effects on the perceived importance of the studies. Respondents in the null result treatment associate the studies with 32.5% of a standard deviation lower importance (95% C.I. [−43,−21.9]; p < 0.001) and think other researchers would associate the studies with 41.7% of a standard deviation lower importance (95% C.I. [−52.6,−30.7]; p < 0.001).

See Also: