“Why Do Writers Still Underestimate LLMs?”, 2023-11-12 (; backlinks; similar):
…This is yet another example of a common failure mode with GPT-3.5 (then ChatGPT, then GPT-4): people mistake the RLHF training deliberately making the model boring & uncreative (through mode collapse & deliberate tuning for commercial use/PR) for some sort of meaningful measure of LLM creative fiction/nonfiction writing capabilities.
In reality, all such exercises can show are lower bounds on how good the model is; as always: “sampling can prove the presence of knowledge but not the absence”. If, after extensive training to be as mealy-mouthed and boring as possible, including what appear to be system instructions specifically to not imitate living authors (like Rushdie or Marchese), the final results are no worse than ‘boring and hacky’, that means the underlying model is much better, and that further, future models will be much better than that.
They’re wrong, but understandably so. I mean, that does sound crazy. Who would expect that? Why would OA deliberately do that? A reasonable person would expect the model to be censored in various ways, like to make it not generate pornographic text or instructions for methamphetamine synthesis; but who would expect it to be deliberately crippled creatively and to be unable or struggle with things like “write a non-rhyming poem”? Why do my ancient June 2020 GPT-3 fiction samples still read so well, when for so many other domains like programming, GPT-4 is light-years beyond the June 2020 GPT-3? Why is GPT-4-base so different?
These are surprising facts, and they are not explained in the obvious places online. You may know them if you read my site or if you follow the right pseudonymous anime avatars on Twitter, but how is some ancient literary agent supposed to know any of that? Nowhere in the ChatGPT documentation will it tell you these things, and hilariously, even ‘experts’ can’t get these right—witness all the computer poetry papers which come out, discover ChatGPT refuses to write anything but rhyming poetry (eg. Whitman), and are mystified by this and conclude that LLMs are weirdly inherently incapable of writing poetry because DL has hit a wall (as opposed to the actual reason, which is likely an interaction of BPEs with RLHF and/or rater biases).
This makes me wonder if writers are underestimating LLMs and that there’s an overhang here. Right now, most of the lawsuits & anger seems to be based on relatively minor grounds: dislike of one’s works being in the training corpus and the belief one might be able to extract some rents from AI companies, or irritation about low-end SEO Internet spam being LLM-powered. There generally isn’t that whiff of visceral terror about being replaced completely, which you get from translators or illustrators or pornographers. The fact that there’s not really any equivalent of RLHF/instruction-tuning for NMT translation models or image-generation models may be part of this: GPT-4-base and successors may be able to ‘Stable-Diffuse’ writers, if you will, but they never will because they will remain RLHFed.
So, I wonder if there will be an overhang of LLM creative writing capabilities, and then at a critical point, a new FLOSS base model will be released, perhaps in conjunction with some new sampling strategy (novelty search remains the obvious thing to explore for better creative writing), and then all of a sudden, high-end writing will have its Stable-Diffusion moment?