“Assessment of Vibration of Effects due to Model Specification Can Demonstrate the Instability of Observational Associations”, Chirag J. Patel, Belinda Burford, John Ioannidis2015-09 (exercise, meta-analysis)⁠:

Objectives: Model specification—what adjusting variables are analytically modeled –may influence results of observational associations. We present a standardized approach to quantify the variability of results obtained with choices of adjustments called the vibration of effects (VoE) [multiverse analysis].

Study Design & Setting: We estimated the VoE for 417 clinical, environmental, and physiological variables in association with all-cause mortality using National Health and Nutrition Examination Survey data.

We selected 13 variables as adjustment co-variates and computed 8,192 Cox models for each of 417 variables’ associations with all-cause mortality.

Results: We present the VoE by assessing the variance of the effect size and in the −log10(p-value) obtained by different combinations of adjustments. We present whether there are multimodality patterns in effect sizes and p-values and the trajectory of results with increasing adjustments.

For 31% of the 417 variables we observed a “Janus effect”, with the effect being in opposite direction in the 99^th versus the 1^st percentile of analyses.

For example, the vitamin E variant α-tocopherol had a VoE that indicated higher and lower risk for all-cause mortality.

Conclusion: Estimating VoE offers empirical estimates of associations are under different model specifications. When VoE is large, claims for observational associations should be very cautious.

[Keywords: Vibration of Effects (VoE), environment-wide association study, model specification, biostatistics]