“How Novelists Use Generative Language Models: An Exploratory User Study”, 2020 (; backlinks):
Generative language models are garnering interest as creative tools. We present a user study to explore how fiction writers use generative language models during their writing process.
We had 4 professional novelists complete various writing tasks while having access to a generative language model that either finishes their sentence or generates the next paragraph of text [‘Talk To Transformer’ vs ‘Write With Transformer’ GPT-2 services].
We report the primary ways that novelists interact with these models, including: to generate ideas for describing scenes and characters, to create antagonistic suggestions that force them to hone their descriptive language, and as a constraint tool for challenging their writing practice.
We identify 6 criteria for evaluating creative writing assistants, and propose design guidelines for future co-writing tools…We propose a series of evaluation questions, which could be answered computationally, to guide system design:
Does a suggestion match the tense of the preceding text?
Does a suggestion introduce new characters or objects, or does it reference preceding ones?
Are new characters or objects coherent given the context?
Does a suggestion include description?
Does a suggestion include action?
Given a single request, how diverse are the suggestions?
[Keywords: co-creativity, natural language processing, user interface, writing tools, user-study]