“I’m Paid Biweekly, Just Not by Leprechauns: Evaluating Valid-But-Incorrect Response Rates to Attention Check Items”, 2019-10-01 (; backlinks; similar):
Carelessness in self-report data can be detected with many methods.
Embedding items in a scale with presumed ‘correct’ responses is one of these.
Properties of these items can impact their usefulness.
Individuals can provide valid justification for ‘incorrect’ responses.
Researchers should know their items, and know the risk of not knowing those items.
Participant carelessness is a source of invalidity in psychological data (Huang, Liu, & Bowling, 2015), and many methods have been created to screen for this carelessness (Curran, 2016; Johnson, 2005). These include items that researchers presume thoughtful individuals will answer in a given way (eg. disagreement with “I am paid biweekly by leprechauns”, 2012). This paper reports on two samples in which individuals spoke aloud a series of these questions, and found that (a) individuals do occasionally report valid justifications for presumed invalid responses, (b) there is relatively high variance in this behavior over different items, and (c) items developed for this specific purpose tend to work better than those drawn from other sources or created ad-hoc.
[Keywords: Carelessness, Data cleaning, Insufficient effort responding, Verbal protocol, Self-report data]
…
Check Justifications “All my friends are aliens” “‘Aliens’ is a relative term; I don’t actually know for sure” · “What does that even mean, we’re all aliens if there’s other life out there” “I am interested in…parabanjology” “Might be real so don’t want to disagree” · “It sounds like it could be interesting” “I work twenty-eight hours in a typical work day.” “It feels like that sometimes” “I am familiar with geological terms such as jpg and firewall.” “I know what those are, but don’t know that they’re geological” “I am fluent in combinatorial English” “I’m fluent in English” “I am able to read the minds of others” · “I can see into the future” “Understand general idea of what others are thinking” · “Close friends know each other” · “Can plan and expect future events” “I sleep less than one hour per night” “When I’m pulling an all-nighter I do” · “I sleep very few hours each night” “All my friends say I would make a great poodle” “They say I’m like a puppy” · “They say I’d make a great koala” · “Friends say I share dog-like personality” · “Friends have said my hair looks like a poodle” · “Have been told I’d make a good dog” · “Don’t know, I’ve never asked them” “I eat cement occasionally” “There was cement in my braces, sure that I ate some” · “There are a lot of things that are in cement in a lot of foods, so maybe eating parts of it” “Answer with ‘Disagree’ for this item” “Item doesn’t say how much to disagree (picked ‘Strongly disagree’)” “I am paid biweekly by leprechauns” “I am paid biweekly, just not by leprechauns” “I can run 2 miles in 2 min” “It doesn’t say run with your feet, can do it in my mind” “I have been to every country in the world” “I’ve been to a lot of countries” · “I have probably been to more countries than most people” “I can teleport across time and space” “Well, time passes, and I can move places, so that’s sort of true” · “Is walking a type of teleportation?” · “In my dreams I can because one of my life goals is to be the doctor’s companion” Table 2: Selected examples of valid justifications for ‘incorrect’ answers.
[I strongly disagree with the authors that these justifications are even remotely “valid”: most of these responses are ‘careless’ or ‘insufficient effort’.]