“Word in Context: Agent and Agent Clarification (69% Dev)”, 2020-07-30 ():
The Word in Context benchmark is part of the SuperGLUE benchmark. In the GPT-3 paper the OpenAI team got 49%, or random chance, on the dev set benchmark (85% on the test set is SOTA).
While I submitted a performance eval on the actual test set using multiple outputs chained together from completion and search to to get 67%, you may be able to get similar or better performance by just doing informed few-shot training to have it do its own context stuffing when as a step of producing the output. Additional work in this area may improve ability to few-shot disambiguation tasks.
Given the API’s implicit understanding of a lot of language, clever people should be able to improve these results quickly.
…Agent and Agent Clarification (69% Dev): Code (nbviewer)
#…fewShotsNounDefinitions = [ ] #I’ll build this from the training set for row in train[train.pos=="N"].tail(10).iterrows(): s1 = row[1]["context-1"] target = row[1]["target"] r = getContextualNounExample(s1, target) print(r) fewShotsNounDefinitions.append(" ".join(r.split(" "))) #add the definition to fewShotsDefinition as 1 line Tom said ’The cinema relies on apparent motion .’ I asked Tom what ’motion’ specifically means in this context, he clarified it is another word for ’movement’. Tom said ’It vanished into the night .’ I asked Tom what ’night’ specifically means in this context, he clarified it is another word for ’darkness’. Tom said ’He threw the ball into the air .’ I asked Tom what ’air’ specifically means in this context, he clarified it is another word for ’sky’. Tom said ’Those clouds show little sign of raining soon .’ I asked Tom what ’sign’ specifically means in this context, he clarified it is another word for ’signs’ or ’indications’. Tom said ’We added a new rosebush to our rose bed .’ I asked Tom what ’bed’ specifically means in this context, he clarified it is another word for ’garden’ or ’yard’ Tom said ’His state of health .’ I asked Tom what ’state’ specifically means in this context, he clarified it is another word for ’condition’. Tom said ’Likes a drink before dinner .’ I asked Tom what ’drink’ specifically means in this context, he clarified it is another word for ’alcoholic drink’. Tom said ’Piecas kronas \u2014 five krona .’ I asked Tom what ’krona’ specifically means in this context, he clarified it is another word for ’coin’. Tom said ’The harder the conflict the more glorious the triumph’ --Thomas Paine. I asked Tom what ’conflict’ specifically means in this context, he clarified it is another word for war. Tom said ’An invasion of bees .’ I asked Tom what ’invasion’ specifically means in this context, he clarified it is another word for ’attack’ or ’assault’.