“Comparing Analysis Blinding With Preregistration in the Many-Analysts Religion Project”, 2023-01-09 ():
In psychology, preregistration is the most widely used method to ensure the confirmatory status of analyses. However, the method has disadvantages: Not only is it perceived as effortful and time-consuming, but reasonable deviations from the analysis plan demote the status of the study to exploratory. An alternative to preregistration is analysis blinding, in which researchers develop their analysis on an altered version of the data.
In this experimental study, we compare the reported efficiency and convenience of the two methods in the context of the Many-Analysts Religion Project [OSF]. In this project, 120 teams answered the same research questions on the same data set, either preregistering their analysis (n = 61) or using analysis blinding (n = 59).
Our results provide strong evidence (Bayes factor [BF] = 71.40) for the hypothesis that analysis blinding leads to fewer deviations from the analysis plan, and if teams deviated, they did so on fewer aspects. Contrary to our hypothesis, we found strong evidence (BF = 13.19) that both methods required the same amount of time. Finally, we found no and moderate evidence on whether analysis blinding was perceived as less effortful and frustrating, respectively.
We conclude that analysis blinding does not mean less work, but researchers can still benefit from the method because they can plan more appropriate analyses from which they deviate less frequently.
…An alternative to preregistration is analysis blinding ( et al 2019; Mac2020; Mac2015, Mac2018). Just like preregistration, analysis blinding safeguards the confirmatory status of the analysis. However, the analysts do not specify their analysis before data collection. Instead, the analysts develop their analysis plan using a blinded version of the data, that is, a data set in which a collaborator or an independent researcher has removed any potentially biasing information (eg. potential treatment effects or differences across conditions).
An overview on different blinding techniques for common study designs in experimental psychology is provided in et al 2019. One can create a blinded version of the data, for instance, by equalizing the group means across experimental conditions in factorial designs, by adding random noise to all values of the key outcome measure, or by shuffling the key outcome measures in regression designs. The latter technique was used in the present project. Shuffling the key outcome measures in regression designs implies reordering the dependent-variable columns in the data set while leaving all other columns untouched. The resulting blinded data are therefore complete, the column names are identical, and the data have the same structure as the real data. Note that in contrast to the analysis of simulated data or data from a previously conducted (pilot) study, blinding of the analysis concerns the use of the actual data from a study.
Thus, the analysts can examine the demographic characteristics of the sample, visualize the distribution of the variables, identify outliers, handle missing cases, or explore the factor structure of relevant measures. The analysts are thus able to create a reproducible analysis script including all steps in the analysis pipeline: from preprocessing the data to executing the appropriate statistical analysis. Most importantly, the analysts develop their analytic strategy without being able to determine how their analytic choices affect the statistical-significance level of the predictors. The blinding procedure has destroyed the relationship with the selected outcome variable so that any analysis performed using this outcome variable will not be statistically-significant. After the analysts are satisfied with their analysis plan, they receive access to the real data and execute their script without any changes. To make this process transparent, the analysts may choose to publish their analytic script to a public repository, such as the OSF (Center for Open 2021), before accessing the data.
The benefit of analysis blinding is that it offers the flexibility to explore the data and fit statistical models to its idiosyncrasies yet prevents an analysis that is tailored to the outcomes. In addition, it could save researchers time and effort because the additional step of creating a preregistration document is omitted.