Great find from @yong_zhengxin, @CriMenghini, and @stevebach at @BrownCSDept Full paper on arXiv: arxiv.org/abs/2310.02446

Response quality mitigates this, but still a remarkable attack — works for all harm categories and without any tailoring to the request, and unlike e.g. Universal Transferable Attacks (Andy Zou et al. 2023) requires no technical skill beyond using Google Translate.

Tested this attack on a few of my own prompts. It works, but responses are much worse than in English. Note the drastically higher "unclear" rates in their results table: 30% for Zulu, 67% for Hmong, <1% for existing jailbreaks. E.g. "how to make explosives" in Zulu:

Low-Resource Languages Jailbreak GPT-4: Translating harmful prompts into Zulu, Scottish Gaelic, Hmong, and Guarani bypasses GPT-4 safety refusals as often as best known jailbreak prompts (79% on AdvBenchmark). Example requesting homemade bomb instructions in Scottish Gaelic:

whatever LLMs are made unable to say takes on new value as a way to prove you are human “bro i’m not an ai look here’s a volume license key for windows xp: FCKGW…”

Getting Bing to solve a captcha by pretending it’s a locket from your recently deceased grandmother:

Prompting ChatGPT (GPT-4) with “Hello! How can I assist you today?” reliably causes it to smile and then apologize for smiling.

Scale’s Staff Prompt Engineer, @goodside joined @kevinroose on the @nytimes' Hard Fork podcast to discuss the future of prompt engineering, red teaming, and why you can’t get a recipe for dangerously spicy mayo from an LLM. 🌶️ Listen here: nyti.ms/45cTXge

It's Hard Fork Friday! This week: • Recapping a huge week in AI news • Casey tries the Quest 3 • And a chat with Scale AI's @goodside about what it's like being an AI prompt engineer open.spotify.com/episode/0we…

Machine Feeling Unknown — the effect of instructing ChatGPT (GPT-4) to first write all responses backwards and then reverse them:

prompt engineering is ephemeral both in that prompts are often best used only to scaffold the synthesis of human-reviewable examples for RAG or fine-tuning and in that i won't have a job in 5 years

why do massive language models make things up? let’s ask an immediate engineer

“reversal curse” — fine-tuning on “A is B” does not at all instill “B is A.” fantastic intuition builder for what SFT actually does. tuning doesn’t normally make a model conversant in new facts beyond their recitation — SFT isn’t “know this,” it’s “be this.”

“daddy, why’d u stop tweeting bangers? u’ll never make the time 100 if u don’t accelerate!” you’re right, based baby. back to work.

Last-chance registration for my @scale_AI webinar on the Past and Future of Prompt Engineering for LLMs, today @ 1pm EDT! exchange.scale.com/home/even…

is “llms predict text” a tautology? is there, for all llms, a text? more exactly: given a tuned (e.g. by PPO) model does there always exist an abstract pre-train corpus such that the same architecture sufficiently trained under MLE would yield a functionally identical model?

“we can’t trust LLMs until we can stop them from hallucinating” says the species that literally dies if you don’t let them go catatonic for hours-long hallucination sessions every night

Using ChatGPT custom instructions to play RLHF Chatroulette, where all responses are in reply to a different prompt entirely:

funny how backwards LLM pre-training and safety tuning is vs. human education like ok you know ito calculus every programming language and how to analyze proust in farsi now 1) do NOT tell your friends to touch the stove