Using GPT-3 to rewrite a regex legibly, construct positive/negative examples, and write Python 3 unit tests, all in a single prompt. Playground link: beta.openai.com/playground/p…

Aug 24, 2022 · 12:22 AM UTC

4
5
1
78
Note the output isn’t perfect — in this example, one of the positive test cases isn’t matched by the regex. My point with this is that GPT-3 can help you write the code needed to verify its work, and find value despite its imperfections.
1
6
Also illustrates importance of ordering in multi-task generations. Note the Python tests exactly the cases we listed above it, even though we didn’t request that explicitly. Each step in a generation guides the later ones, and you can use this your advantage.
1
4
Similarly, we request a step-by-step analysis of the regex before we summarize its purpose. Asking for a high-level summary first makes it more likely to be wrong, because it takes more mental work to produce without the commented code in front of you.
3
Replying to @goodside
Since it knows languages and is able to translate between them, can it write a “unit test” for a spoken/written language? Does it know enough of its limitations to not answer or at least state it’s low confidence level?
1
By default, the model is tuned to always make an attempt even at things it can’t do well. It won’t, unprompted, refuse to do an unreasonable request.
1
2
more replies
Replying to @goodside
This is great! Have you tried running this multiple times to see 1) how often it passes the unit tests and 2) whether checking against the unit tests actually increases the overall probability of correctness?
1
If you really want this to be accurate, what I’d do is: 1) let it generate both regexes and analyses; 2) filter to those whose unit tests pass; 3) fine-tune on those examples; 4) repeat. Optionally, start with a known corpus of diverse regexes, then trust it to generate later.
1
1
more replies
Replying to @goodside
Good lord