It has become routine to report research results where Large Language Models (LLMs) outperform average humans in a wide range of language-related tasks, and creative text writing is no exception. It seems natural, then, to raise the bid: Are LLMs ready to compete in creative writing skills with a top (rather than average) novelist?

To provide an initial answer for this question, we have carried out a contest between Patricio Pron (an awarded novelist, considered one of the best of his generation) and GPT-4 (one of the top performing LLMs), in the spirit of AI-human duels such as Deep Blue vs Kasparov and AlphaGo vs Lee Sedol.

We asked Pron and GPT-4 to provide 30 titles each, and then to write short stories for both their titles and their opponent’s. Then, we prepared an evaluation rubric inspired by Boden’s definition of creativity, and we collected 5,400 manual assessments provided by literature critics and scholars.

The results of our experimentation indicate that LLMs are still far from challenging a top human creative writer, and that reaching such level of autonomous creative writing skills probably cannot be reached simply with larger language models.

…Also, our study highlights the large role of prompts in creative text writing: titles provided by Pron resulted in GPT-4 texts which are substantially more creative and original than the ones written for its own titles. Even the simplest prompting (short titles in our case) should be considered co-authorship, as it has a profound influence on the results

After all I almost did for you
All love songs are sad songs
Another episode in the Class Struggle
Don’t tell mom
Eclipse in the botanical garden
Edith loves him (we’ll come back to this)
Every picture from when we were young
Future ghosts
I have no fear because I have nothing
I keep trying to forget your promise
Lindsay Hilton visits Paris
Mental illness 3 days a week
Monsters live here
Paradise can’t be seen from here
Pick a card, any card. No, not that one! Another!
Rise and fall of R. S. Turtleneck, children’s author
Silks from Bursa, tiles from Kütahya
Spanish Youth, keep trying
The day after Groundhog day
The delights of the garden of delights
The last journey of Santiago Calatrava
The last laugh of that year
The Lego woman
The national red button
The nightmares of the invisible man
The nocturnal emissions
The tied cow
Two cops stand between us
When you are at the top you can’t fall any lower
Who killed Patricio Pron?

Among clouds and mirages
Between the lines of fate
Beyond the broken horizon
Bits of reality
Echoes of a lost dream
Echoes of the future
Fragments of an invisible yesterday
Parallel paths
Reflections of another world
Shadows in the mist
Song of the captive moon
Sparks in the dark
The awakening of the aurora
The crystal labyrinth
The echo of silenced voices
The forgotten melody
The garden of withered dreams
The inverted city
The journey of the dawn
The last flight of the butterfly
The last night on Earth
The mosaic of time
The painter of memories
The shadows of time
The whisper of the cosmos
The wind in the moorlands
Traces in the sea of sand
Twilight of the titans
Under the copper sky
Whispers from the eternal city