“On the Accuracy, Media Representation, and Public Perception of Psychological Scientists’ Judgments of Societal Change”, 2023-04-20 (; similar):
At the onset of the COVID-19 crisis, psychological scientists contributed to the public discourse on COVID-related societal change in the news media through intuition-based reasoning and often made predictions outside their area of expertise.
We assessed the likely accuracy of such judgments by surveying psychological scientists and laypeople at the onset of the pandemic regarding future societal change in different domains and comparing predictions to actual markers of change at 6 months and 1 year after.
We found that psychological scientists and laypeople made similar and largely inaccurate predictions. Neither direct experience, training, nor domain-specific expertise was associated with greater accuracy.
[OSF, pre-registration] At the onset of the COVID-19 pandemic, psychological scientists frequently made on-the-record predictions in public media about how individuals and society would change.
Such predictions were often made outside these scientists’ areas of expertise, with justifications based on intuition, heuristics, and analogical reasoning (Study 1; n = 719 statements).
How accurate are these kinds of judgments regarding societal change? In Study 2, we obtained predictions from scientists (n = 717) and lay Americans (n = 394) in Spring 2020 regarding the direction of change for a range of social and psychological phenomena. We compared them to objective data obtained at 6 months and 1 year. To further probe how experience impacts such judgments, 6 months later (Study 3), we obtained retrospective judgments of societal change for the same domains (Nscientists = 270; Nlaypeople = 411).
Bayesian analysis suggested greater credibility of the null hypothesis that scientists’ judgments were at chance on average for both prospective and retrospective judgments. Moreover, neither domain-general expertise (ie. judgmental accuracy of scientists compared to laypeople) nor self-identified domain-specific expertise improved accuracy.
In a follow-up study on meta-accuracy (Study 4), we show that:
the public nevertheless expects psychological scientists to make more accurate predictions about individual and societal change compared to most other scientific disciplines, politicians, and non-scientists, and they prefer to follow their recommendations.
These findings raise questions about the role psychological scientists could and should play in helping the public and policymakers plan for future events.
See Also:
Behavioral scientists and laypeople misestimate societal effects of COVID-19
Laypeople Can Predict Which Social-Science Studies Will Be Replicated Successfully
Shifting Minds: A Quantitative Reappraisal of Cognitive-Intervention Research
The unlikelihood effect: When knowing more creates the perception of less
Covid-19 is (Probably) Not an Exogenous Shock or Valid Instrument
Harm Hypervigilance in Public Reactions to Scientific Evidence