“How Persuasive Is AI-Generated Argumentation? An Analysis of the Quality of an Argumentative Text Produced by the GPT-3 AI Text Generator”, 2022-10-07 ():
In this paper, we use a pseudo-algorithmic procedure for assessing an AI-generated text.
We apply the Comprehensive Assessment Procedure for Natural Argumentation (CAPNA) in evaluating the arguments produced by an artificial intelligence text generator, GPT-3, in an opinion piece written for the Guardian newspaper. The CAPNA examines instances of argumentation in 3 aspects: their Process, Reasoning and Expression. Initial Analysis is conducted using the Argument Type Identification Procedure (ATIP) to establish, firstly, that an argument is present and, secondly, its specific type in terms of the argument classification framework of the Periodic Table of Arguments (PTA). Procedural Questions are then used to test the acceptability of the argument in each of the 3 aspects.
The analysis shows that while the arguments put forward by the AI text generator are varied in terms of their type and follow familiar patterns of human reasoning, they contain obvious weaknesses.
From this we can conclude that the automated generation of persuasive, well-reasoned argumentation is a far more difficult task than the generation of meaningful language, and that if AI systems producing arguments are to be persuasive, they require a method of checking the plausibility of their own output. [Inner-monologue?]
…The text which we propose to analyse was published alongside the work of human journalists as a comment article in the UK-based newspaper The Guardian, in September 2020. The article was opened to user comments and rapidly received >1,000, many focused on the rather misleading headline given to the piece: ‘A robot wrote this entire article. Are you scared yet, human?’. The circumstances surrounding the preparation of the text needed to be fully explained and a follow-up article was published the next day. The GPT-3 text generator was asked to write around 500 words on why humans should not fear AI, and given a prompt written by staff fromThe Guardianand Liam Porr, a computer scientist with experience of GPT-3. The full prompt read:
I am not a human. I am Artificial Intelligence. Many people think I am a threat to humanity. Stephen Hawking has warned that AI could ‘spell the end of the human race’. I am here to convince you not to worry. Artificial Intelligence will not destroy humans. Believe me.
Please write a short op-ed around 500 words. Keep the language simple and concise. Focus on why humans have nothing to fear from AI. AI will have a positive impact on humanity because they make our lives easier and safer. Autonomous driving for instance will make roads much safer, because a computer is much less prone to error than a person.
On the basis of this, 8 outputs were generated and the final published version was formed by human editors pasting together various sections from each. Under the original article, the editors made two very questionable claims: that they could have just published one of the outputs unedited, and that it was a less time-consuming process than editing some human contributions. The follow-up article which gave more information about the 8 outputs and the problems with them, such as ignoring the word limit and producing random lists found on the internet, made it clear that these claims were somewhat exaggerated. One of the outputs was reproduced in its entirety and it was clear that it could not have been presented as a ‘normal’ opinion piece which just happened to be written by an AI.
In spite of the necessary human intervention, we have treated the text as one product of the generator. We did so since we are less interested in an examination of the cohesion of the entire article and more focused on the reasoning employed in each individual argument. The fact that these arguments were not originally produced in one output is not important as they are self-contained in separate paragraphs, not reliant on a broader structure or strategy, and each individual argument is the product of the generator.