“Gary Marcus Has Co-Authored a Brief Critique of GPT-3”, 2020-08-31 (; backlinks; similar):
I was disappointed by Marcus’ critiques of GPT-2, but this is even worse!
…Then we get to the individual results. It is difficult for me to read many of the authors’ assessments without picturing them as characters in a dystopian satire, administering a dreamlike and impossible “psychological examination” to our hapless protagonist…What do the authors even imagine success to be, here?
Sometimes they deliberately describe a surreal situation, then penalize GPT-3 for continuing it in an identically surreal manner—surely the “right” answer if anything is! (“No one in a restaurant asks their neighbor to share a spoon”—yeah, and no one tries to drink soup with their eyeglasses, either!) Sometimes they provide what sounds like a de-contextualized passage from a longer narrative, then penalize GPT-3 for continuing it in a perfectly natural way that implies a broader narrative world continuing before and after the passage. (“There is no reason for your brother to look concerned.” How in the world do you know that? “The switch to the pig is a non-sequitur.” Is it? Why? “The sentence [about Moshe and ‘the spirit of the season’] is meaningless.” How can you say that when you don’t know what season it is, what its “spirit” is, who this Moshe guy is… And come on, the Janet one is a great story hook! Don’t you want to read the rest?)
I don’t claim to be saying anything new here. Others have made the same points. I’m just chiming in to… boggle at the sheer weirdness, I guess. As I said, GPT-3 comes off here like a sympathetic protagonist, and the authors as dystopian inquisitors!