“Are Fit Indices Used to Test Psychopathology Structure Biased? A Simulation Study”, 2019-07-18 ():
Latent variable models of psychopathology provide dimensional alternatives to traditional categorical classification systems (eg. DSM-5 and ICD-11), with the two most popular being the bifactor model and correlated factors models. These competing structural models of psychopathology are often compared via statistical indices to assess how well each model fits the same data.
The results of our simulation study suggest that bifactor models are often erroneously favored over correlated factor models when the simulated data were generated by a correlated factors model with minor misspecifications. Findings from tests of model equivalence also clarified the conditions under which fit indices’ favoring of the bifactor model was characterized by bias.
This calls into question the common practice of relying on common fit statistics when comparing structural models of psychopathology.
Structural models of psychopathology provide dimensional alternatives to traditional categorical classification systems. Competing models, such as the bifactor and correlated factors models, are typically compared via statistical indices to assess how well each model fits the same data. However, simulation studies have found evidence for probifactor fit index bias in several psychological research domains.
The present study sought to extend this research to models of psychopathology, wherein the bifactor model has received much attention, but its susceptibility to bias is not well characterized. We used Monte Carlo simulations to examine how various model misspecifications produced fit index bias for 2 commonly used estimators, WLSMV [weighted least square mean and variance adjusted] and MLR [maximum likelihood with robust standard errors]. We simulated binary indicators to represent psychiatric diagnoses and positively skewed continuous indicators to represent symptom counts.
Across combinations of estimators, indicator distributions, and misspecifications, complex patterns of bias emerged, with fit indices more often than not failing to correctly identify the correlated factors model as the data-generating model. No fit index emerged as reliably unbiased across all misspecification scenarios. Although, tests of model equivalence indicated that in one instance fit indices were not biased—they favored the bifactor model, albeit not unfairly.
Overall, results suggest that comparisons of bifactor models to alternatives using fit indices may be misleading and call into question the evidentiary meaning of previous studies that identified the bifactor model as superior based on fit.
We highlight the importance of comparing models based on substantive interpretability and their utility for addressing study aims, the methodological importance of model equivalence, as well as the need for implementation of statistical metrics that evaluate model quality.
[Keywords: bifactor model, factor analysis, fit index bias, model evaluation, Monte Carlo simulation]