[fails to consider BPE tokenization or tuning] Large language models (LLMs) can now generate and recognize text in a wide range of styles and genres, including highly specialized, creative genres like poetry. But what do LLMs really know about poetry? What can they know about poetry?
We develop a task to evaluate how well LLMs recognize a specific aspect of poetry, poetic form, for more than 20 forms and formal elements in the English language. Poetic form captures many different poetic features, including rhyme scheme, meter, and word or line repetition.
We use this task to reflect on LLMsâ current poetic capabilities [ChatGPT-3, (Chat)GPT-4, Claude-3-Sonnet, LLaMA-3-70bâunclear whether base or tuned], as well as the challenges and pitfalls of creating NLP benchmarks for poetry and for other creative tasks. In particular, we use this task to audit and reflect on the poems included in popular pretraining datasets.
Our findings have implications for NLP researchers interested in model evaluation, digital humanities and cultural analytics scholars, and cultural heritage professionals.
âŚPoetic forms based on topic prove more difficult for the models, depending on the topic (Table 5 & Table 6). Forms centered on more concrete subjects like âdeathâ (elegy) and âartâ (ars poetica, ekphrasis) are more often recognized, while poems about abstract ideas and styles like aubades and odes are less so.
There are fewer forms in our dataset that depend on visual features, but most models except GPT-4 and GPT-4o falter with them, namely with concrete / pattern poetry (ie. poems that rely on visual and typographical elements for their structure) and prose poetry (ie. poems that donât have line breaks and look like prose).