“Is ChatGPT Transforming Academics’ Writing Style?”, Mingmeng Geng, Roberto Trotta2024-04-12 (GPT-4 nonfiction, science)⁠:

Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT’s writing style in their abstracts by means of a statistical analysis of word frequency changes.

Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis.

We find that ChatGPT is having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of ChatGPT-revised abstracts is estimated to be ~35%, if we take the output of one of the simplest prompts, “revise the following sentences”, as a baseline.

We conclude with an analysis of both positive and negative aspects of the penetration of ChatGPT into academics’ writing style.