“The Micro-Task Market for Lemons: Data Quality on Amazon’s Mechanical Turk”, Douglas J. Ahler, Carolyn E. Roush, Gaurav Sood2021-10-25 (psychology)⁠:

While Amazon’s Mechanical Turk (MTurk) has reduced the cost of collecting original data, in 2018, researchers noted the potential existence of a large number of bad actors on the platform.

To evaluate data quality on MTurk, we fielded 3 surveys 2018^–₂2020.

While we find no evidence of a “bot epidemic”, large portions of the data—25–35%—are of dubious quality. While the number of IP addresses that completed the survey multiple times or circumvented location requirements fell almost 50% over time, suspicious IP addresses are more prevalent on MTurk than on other platforms. Furthermore, many respondents appear to respond humorously or insincerely, and this behavior increased over 200% 2018^–₂2020.

Importantly, these low-quality responses attenuate observed treatment effects by magnitudes ranging from ~10–30%.

See Also:

• Too Good to Be True: Bots and Bad Data From Mechanical Turk

• Lizardman’s Constant Is 4%

• Measuring the Prevalence of Questionable Research Practices with Incentives for Truth-Telling

• ‘Mischievous Responders’ Confound Research On Teens

• Generalizability of heterogeneous treatment effect estimates across samples