“Language Models Accurately Infer Correlations between Psychological Items and Scales from Text Alone”, Björn E. Hommel, Ruben C. Arslan2024-04-05 (Transformer, psychology)⁠:

Many behavioral scientists do not agree on core constructs and how they should be measured. Different literatures measure related constructs, but the connections are not always obvious to readers and meta-analysts. Many measures in behavioral science are based on agreement with survey items. Because these items are sentences, computerised language models can make connections between disparate measures and constructs and help researchers regain an overview over the rapidly growing, fragmented literature.

Our fine-tuned language model, the SurveyBot3000 [based on Sentence BERT], accurately predicts the correlations between survey items, the reliability of aggregated measurement scales, and intercorrelations between scales from item positions in semantic vector space. In our pilot study, the out-of-sample accuracy for item correlations was 0.71, 0.86 for reliabilities, and 0.89 for scale correlations.

In a preregistered study, we will investigate whether the performance of our model generalizes to measures across behavioral science.

See Also:

• Expanding the methodological toolbox: Machine-based item desirability ratings as an alternative to human-based ratings

• Let the Algorithm Speak: How to Use Neural Networks for Automatic Item Generation in Psychological Scale Development

• A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

• Automated Social Science: Language Models as Scientist and Subjects

• Automatic Personality Assessment Through Social Media Language