“Do Teachers Spot AI? Evaluating the Detectability of AI-Generated Texts among Student Essays”, Johanna Fleckenstein, Jennifer Meyer, Thorben Jansen, Stefan D. Keller, Olaf Köller, Jens Möller2024-06 (GPT-3 nonfiction; similar)⁠:

• Randomized-controlled experiments investigating novice and experienced teachers’ ability to identify AI-generated texts.

• Generative AI can simulate student essay writing in a way that is undetectable for teachers.

• Teachers are overconfident in their source identification.

• AI-generated essays tend to be assessed more positively than student-written texts.

The potential application of generative artificial intelligence (AI) in schools and universities poses great challenges, especially for the assessment of students’ texts. Previous research has shown that people generally have difficulty distinguishing AI-generated from human-written texts; however, the ability of teachers to identify an AI-generated text among student essays has not yet been investigated.

Here we show in two experimental studies that novice (n = 89) and experienced teachers (n = 200) could not identify texts generated by ChatGPT among student-written texts. However, there are some indications that more experienced teachers made more differentiated and more accurate judgments. Furthermore, both groups were overconfident in their judgments.

Effects of real and assumed source on quality assessment were heterogeneous.

Our findings demonstrate that with relatively little prompting, current AI can generate texts that are not detectable for teachers, which poses a challenge to schools and universities in grading student essays. Our study provides empirical evidence for the current debate regarding exam strategies in schools and universities in light of the latest technological developments.

See Also:

• A real-world test of artificial intelligence infiltration of a university examinations system: A ‘Turing Test’ case study

• How to cheat on your final paper: Assigning AI for student writing

• Investigating attitudes of professional writers to GPT text generation AI based creative support tools

• AI composer bias: Listeners like music less when they think it was composed by an AI

• Artificial Intelligence Can Persuade Humans on Political Issues

• Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers

• Can GPT-3 write an academic paper on itself, with minimal human input?

• AI-synthesized faces are indistinguishable from real faces and more trustworthy

Similar Links: