Four prompts demonstrating that ChatGPT (GPT-4) is unable to correctly repeat or reason about the string “ davidjl”, the name of a YouTube user:
91
174
61
1,871
In the screenshots above this token appears to be variously misread as “jdl” “jndl”, “jdnl”, “jspb”, “JDL”, or “JD”. These hallucinations also affect ChatGPT’s auto-generated titles, which are inconsistent with their conversations and sometimes prematurely truncated.
5
2
202
“ davidjl” is one of the many “glitch tokens” identified by Jessica Rumbelow and Matthew Watkins of SERI-MATS as producing hallucinations in GPT-2, -3, and -3.5. Most of these no longer produce hallucinations in GPT-4, but “ davidjl” still does. lesswrong.com/posts/aPeJE8bS…

SolidGoldMagikarp (plus, prompt generation) — LessWrong

UPDATE (14th Feb 2023): ChatGPT appears to have been patched! However, very strange behaviour can still be elicited in the OpenAI playground, particu…

lesswrong.com

Jun 8, 2023 · 12:10 AM UTC

3
30
7
443
More examples from @SoC_trilogy:
Matthew Watkins @SoC_trilogy
Jun 7
Someone else found this shortly after GPT-4 was released, but I've not seen any followup. At least one of the GPT-2/3 glitch tokens, " davidjl", also causes GPT-4 to glitch. Given an entirely new token set, how did this one slip through the net? Any guesses? @repligate @goodside
2
2
127
If you want to try this yourself, note “ davidjl” begins with a space character. The hallucination doesn’t show up without it.
3
4
1
135
To clarify: The issue isn't that it can't say the string, it's that it can't read it. If you break up the " davidjl" into two or more substrings it's able to say it by concatenating them. But if you literally write “ davidjl” (with a leading space) it can’t read it.
3
3
1
86
Important correction, with apologies to davidjl on YouTube — the source of our problems is Reddit, as usual:
Matthew Watkins @SoC_trilogy
Jun 8
Replying to @goodside @AHSEUVOU15
There is indded a YT user called "davidjl", but the " davidjl" token used by GPT-2/3/J and seemingly -4 almost certainly comes from Reddit user davidjl123 (I'm guessing a different person) who made it into r/counting's "Hall of Counters" along with SoldGoldMagikarp and friends:
4
5
68
If you liked this example and want to learn more about tokenization, see this great post by @simonw (who you should follow!):
Simon Willison @simonw
Jun 8
Understanding GPT tokenizers: I wrote about how the tokenizers used by the various GPT models actually work, including an interactive tool for experimenting with their output simonwillison.net/2023/Jun/8…
1
3
14
Replying to @goodside
Curiously it is even worse with two blanks...
1