[OSF] Theories are among the most important tools of science. 1943 already noted “There is nothing as practical as a good theory.” However, although there has been a lot of discussion about improving theories, most theories in psychology are still of low quality. One reason for this is that it is difficult to assess the quality of a theory in practice. For example, when is it worthwhile to add more assumptions to explain more patterns in data?
To give researchers the ability to answer questions like this, we developed a computational model for theory comparison.
We also make the model available in a package for the statistical software R, so researchers can use it with ease.
We hope that the availability of a tool to assess theory quality will improve the state of theory in psychology and beyond.
Theories are among the most important tools of science. 1943 already noted “There is nothing as practical as a good theory.” Although psychologists discussed problems of theory in their discipline for a long time, weak theories are still widespread in most subfields. One possible reason for this is that psychologists lack the tools to systematically assess the quality of their theories. 1989 developed a computational model for formal theory evaluation based on the concept of explanatory coherence. However, there are possible improvements to 1989’s model and it is not available in software that psychologists typically use. Therefore, we developed a new implementation of explanatory coherence based on the Ising model [note: loosely based on belief networks as Ising spin glass models].
We demonstrate the capabilities of this new Ising model of Explanatory Coherence (IMEC) on several examples from psychology and other sciences. [eg. mutualism]
In addition, we implemented it in the R-package IMEC to assist scientists in evaluating the quality of their theories in practice.
[Keywords: theory appraisal, theory development, explanatory coherence, Ising model]
…Example Applications of IMEC § Positive Manifold of Intelligence by Mutualism
The next paragraph shows the usefulness of IMEC as a tool to compare psychological theories by comparing the explanatory coherence of two theories of intelligence, the g-factor explanation (eg. 1904) and mutualism explanation (van der Maas et al 200618ya). We also show that IMEC allows to evaluate the robustness of the theory to weak evidence or weak explanations and to identify critical experiments by thinking through counterfactuals.
Van der et al 2006 propose an alternative model of intelligence that explains the positive manifold (the positive correlation of different components of intelligence) not by commonly used latent variable explanations (g-factor) but by mutualism. Development by mutualism means that the positive correlation between different aspects of intelligence (eg. verbal reasoning, logical-mathematical thinking) occurs solely due to positive interactions between several distinct cognitive processes during development. This idea is inspired by ecology, where for example, the correlation between different aspects of water quality in a lake (eg. vegetation, water quality) is not explained by a “lake-factor” but by the positive interactions between different aspects of water quality (eg. 1997 [Ecology of shallow lakes]; et al 1993).
Based on this model, van der et al 2006 explain a variety of phenomena in intelligence research, some of which latent variable models have struggled to explain for a long time. However, they also introduce some new assumptions (explanatory hypotheses that support the main hypothesis); therefore, it is interesting to investigate whether their new theory has more explanatory coherence than a latent variable explanation.
The explanatory relations can be seen in Figure 8 with edge weights and thresholds based on the default explained above. Looking at the figure shows that it appears difficult to compare these two theories intuitively. The mutualism theory explains more phenomena than the g-factor theory. However, it is also more complex.
Hence, an instrument like IMEC that can assess the trade-off between these different epistemic values seems required for effective theory comparison.
Implementing these explanatory relations in IMEC indicates that the mutualism explanation is superior with an explanatory coherence of 0.788, whereas the explanatory coherence of the latent variable explanation is only 0.504; in other words, the mutualism hypothesis seems preferable.
This exemplifies how IMEC can smoothly compare theories in contexts where it is intuitively hard to decide between theories.
However, van der et al 2006 state regarding E7 (‘differentiation effects’) that this phenomenon has not been replicated consistently. Differentiation effects imply that the g-factor is not uniform in the population. In particular it has been suggested, that the positive manifold declines with age (eg. 2004) and that the positive manifold is stronger in lower IQ groups (eg. et al 1996). However, both of these manifestations could sometimes not be replicated (eg. 2004). In other words, due to the weak evidence for this phenomenon, we should consider stepping away from IMEC’s default settings and lowering the threshold on E7. In addition, van der et al 2006 state that the mutualism model allows for both differentiation and integration; in other words, the model makes a very general prediction with regards to differentiation effects that can easily be confirmed.
Therefore, let us consider what happens if we reduce both the evidence for E7 and the weight between HM1 and E7 1 → 0.5. In other words, to account for the problematic state of this phenomenon and the vague prediction of the mutualism model, we give only half the evidence to the phenomenon E7 and we weaken the connection to show that the phenomenon is implied weaker by the theory than other phenomena.
Computing IMEC with these settings results in an explanatory coherence of 0.779 for the mutualism model and an explanatory coherence of 0.516 for the latent variable model; a change of −0.009 and 0.008, respectively. In other words, the superiority of the mutualism model seems to be robust to the problems with differentiation effects. This example shows how we can incorporate weak predictions as well as weak empirical support when specifying IMEC.
In addition, the latent variable model may be taken to imply the existence of a biological cause (eg. et al 2005; et al 2005; van der Maas et al 200618ya) that constitutes the g-factor (E8). That this predicted phenomenon was never discovered reduces the explanatory coherence of the latent variable theory. Investigating how the discovery of such a biological basis for intelligence would change the explanatory relations of the two theories can tell us whether finding a biological correlate would constitute a critical experiment (eg. 1964). In other words, an experiment that could discard the more coherent mutualism theory in favor of the latent variable theory.
With IMEC we can test this by modeling a counterfactual world in which a common biological cause of g is discovered. To do so we assign a positive instead of a negative threshold to E8 and compare the theories again. Computing the explanatory coherence assuming a positive threshold on NE8 shows that finding a biological correlate to g would indeed constitute a critical test.
After finding such a correlate the explanatory coherence of the mutualism theory would be 0.782, while the latent variable theory increases to 0.836, surpassing even the explanatory coherence of the mutualism theory in the configuration we started with. Thus, in this thought experiment, the latent variable explanation would be the preferable theory after discovering that it designates a biological feature of the human being.
The example illustrates how IMEC can compare psychological theories. We showed that the explanatory coherence of two theories can be compared with identify which of them constitutes a better explanation. In addition, by varying thresholds and weights we can account for higher or lower corroboration of phenomena. Finally, modeling counterfactual states of a theory (eg. assuming a not yet discovered phenomenon to be discovered) can help us to identify critical experiments that would make a previously inferior explanation the better explanation. However, the example should also be considered a cautionary note as it shows how the conclusions derived from IMEC can depend on what are considered individual hypotheses. For example, one could make the case that when including the assumption of the mutualism model that growth can be described by a logistic model, it would be important to also include that the influence of intelligence on test scores is linear in the latent variable model. Determining inclusion and exclusion of explanatory hypotheses can indeed be considered a weakness of our model and we give more guidance about it in § Discussion.