Potentially cool use of GPT-3 logprobs for idea generation (and look inside its 🧠) Using this prompt engineering trick of adding some numeric signifier to your ideas like '(Team rating: x/5)', you can get GPT-3 to 'score' its generations

Nov 14, 2022 · 10:47 PM UTC

3
17
3
117
You can then examine the logprobs for the token on each rating to get a weighted distribution of how often it would pick each number
1
8
Using Python, you can use string manipulation to extract GPT's sampled ratings, as well as its weighted ratings (different run of the same prompt) It gave both the highest rank and weighted rank to Genometrix and lowest to Biogents, with some discrepency in the middle
1
7
Unfortunately you are limited to top 5 logprob tokens in the response, so you can't incorporate weighted rank on the decimal accurately. Potentially a very useful trick with some more sophisticated prompt engineering. Will follow up with further exploration.
6
Replying to @jmilldotdev
@threadreaderapp unroll
1
Replying to @jmilldotdev
Having fun messing around with this one! Even just the first idea of ranking. Seems like a way to train the model as a user purely on the prompt side
2