“Can Linguists Distinguish between ChatGPT and Human Writing?: A Study of Research Ethics and Academic Publishing”, 2023-12 (; similar):
There has been considerable intrigue surrounding the use of Large Language Model powered AI chatbots such as ChatGPT in research, educational contexts, and beyond. However, most studies have explored such tools’ general capabilities and applications for language teaching purposes. The current study advances this discussion to examine issues pertaining to human judgments, accuracy, and research ethics.
Specifically, we investigate: (1) the extent to which linguists/reviewers from top journals can distinguish AI-generated from human-generated writing, (2) what the basis of reviewers’ decisions are, and (3) the extent to which editors of top Applied Linguistics journals believe AI tools are ethical for research purposes.
In the study, reviewers (n = 72) completed a judgment task involving AI & human-generated research abstracts, and several reviewers participated in follow-up interviews to explain their rationales. Similarly, editors (n = 27) completed a survey and interviews to discuss their beliefs.
Findings: suggest that despite employing multiple rationales to judge texts, reviewers were largely unsuccessful in identifying AI versus human writing, with an overall positive identification rate of only 38.9%. Additionally, many editors believed there are ethical uses of AI tools for facilitating research processes, yet some disagreed.
Future research directions are discussed involving AI tools and academic publishing.
See Also:
Do teachers spot AI? Evaluating the detectability of AI-generated texts among student essays
For Chinese Students, the New Tactic Against AI Checks: More AI
Delving into ChatGPT usage in academic writing through excess vocabulary
GPT-4 is judged more human than humans in displaced and inverted Turing tests