"You are GPT‑3, and you can't do math": Prompting GPT‑3 via zero-shot instruction to answer calculation/math questions by consulting a Python REPL.

Sep 10, 2022 · 3:55 AM UTC

47
340
82
2,677
Generated at temperature=0. Not rigorously tested. Note "```output" is a stop sequence here. Playground link: beta.openai.com/playground/p…

OpenAI Platform

Explore developer resources, tutorials, API docs, and dynamic examples to get the most out of OpenAI's platform.

platform.openai.com
2
5
90
The "4" in "Answer: 4" in the final screenshot is also generated by the model, and should be highlighted green — there's a bug in the Playground UI that sometimes de-highlights the last token when you chain generations together.
2
2
1
62
Just to be clear since this is blowing up, please don’t do this outside of a security sandbox. This prompt will gladly delete your hard drive if asked.
6
25
5
230
Replying to @goodside
Cool, saw something similar on the Eleuther boards a few months back but using JavaScript
1
3
Yes, the idea isn’t new. The only potentially new part is doing it through zero-shot instruction instead of training a fine-tune.
2
Replying to @goodside @stephenbalaban
Similarly you can have it search Wikipedia for a keyword, pass the article and a question to another instance of GPT-3, and push back that answer in the context.
1
1
7
Yes, I wrote a quick demo of doing something similar with Google SERP results too:
Riley Goodside
 @goodside
10 Sep 2022
You are GPT-3. It is the future. You must find out if Harry Styles is single.
1
1
1
6
more replies
Replying to @goodside
Nice, next do it with google search.
1
Yep, that works too:
Riley Goodside
 @goodside
10 Sep 2022
You are GPT-3. It is the future. You must find out if Harry Styles is single.
1
1
more replies
Replying to @goodside
Wait so is GPT3 using python to evaluate the math or is it just the prompt or are u using pyrhon to evaluate it?
1
1
29
GPT-3 spits out code, which I paste into a Python REPL for it and paste back the results. It could attempt to predict the results itself, but it would be wrong. The goal is for it to recognize what it cannot do and use tools, as we do.
4
15
2
307
more replies
Load more