Apropos of Sutskever now having left OA for good, I've gone back to my thinking about the long-term consequences of Altman's coup, and something I began to wonder in 2021 when the news about Anthropic broke: what if "the elves have left Middle Earth"? What if OA has lost its mojo? If so, what would that look like, and how would we know? What's the "rot" narrative?

Maybe I am a bit too easily impressed by how good gpt-4-base and Claude-3.5-sonnet are at poetry compared to the ChatGPTs, but I can't help but wonder if the OA magic has worn off since GPT-4 finished training c. August 2022. What made OA OA in 2020 was that it had taste: it had much less resources than competitors like DeepMind or Google Brain or FAIR, but (thanks to Alec Radford, Ilya Sutskever, Jared Kaplan, and the RLHF-focused safety team like Paul Christiano & Dario Amodei, and fellow-traveler scalers like Andrej Karpathy etc) they bet big on scaling laws & unsupervised learning at the moment those suddenly began to work. Without taste and agility—or you might say, "without its people, OA is nothing"—OA doesn't have that much of a moat.

And most of those people are gone, and the survivors are being policed for leaks to the media, and now know that if they leave, OA management wants to gag them, and has the power to confiscate their vested equity, wiping out all their wealth (an ability they have confirmed and refused to promise to not use); they further have heard the rumors of Altman's mismanagement, lack of candor and broken promises to Superalignment, outside conflicts of interest, ScarJo, and divide-and-conquer management tactics—even if they do not credit this and believe Altman that he had no idea and some rogue lawyer is to blame, the psychological safety has been lost. (If you speak up, how sure are you that there will be no retaliation?) And who has replaced those people? Careerists and spooks, while it becomes increasingly clear that OA is Microsoft's bitch. (Also, your "Chief Scientist", who ousted Ilya, is now so sloppy that he's embarrassing you on Twitter in front of everyone by getting hacked by cryptocurrency scammers.)

What are the vibes now? Where is the research taste at OA, what ideas or breakthroughs have they published the past few years of note? The weird rumored Franken-MoE architecture of GPT-4? GPT-4o, whose architecture has been obvious since DALL·E 1, if not well before, and which benchmarks great but users are overall less pleased? ("SAEs"? "Q*"? Yeah, good stuff, possibly even great in the case of the known unknown of Q*—too bad they fired a bunch of people and liquidated that department. Also, many of the people responsible for those research directions left OA before or were elsewhere to begin with...) Where is the pride, rather than PPUs? Or any sense of esthetics, like fixing the still present BPE problems, which sabotage GPT models so they cannot "count the number of 'r's in the word 'strawberry'", accurately write text in images in DALL·E 3 or GPT-4o, and where the new GPT-4o BPE vocab is filled with Chinese pornography? Or in fixing the bland ChatGPTese, so inferior to Anthropic's Claude, and which is so bad that copyeditors are actually getting rehired for jobs fixing ChatGPTese?

Now that scaling LLMs has been proven out, anyone with two synapses to rub together can try to follow up and make their own GPT-5. It doesn't take much genius to keep following that path and keep scaling up, just the sort of large-scale application of money that megacorps are reasonably competent at once the pain finally reaches the elephant's brain and it starts to react. But this is also true of OA too, now that it is a $100b megacorp: it can keep treading that path for a long time, becoming more and more successful, before finally missing The Next Big Thing due to its lack of taste. GPT-5 was presumably locked in before the coup; are there enough ideas already prototyped at OA for a GPT-6?

People have been observing, mostly as a criticism of AI safety research, that DL safety research has often helped propel DL capabilities, like RLHF helping make chatbots far more useful and enabling the "ChatGPT moment", and have chortled over AI safety researchers being purged from AI companies (eg Superalignment). But consider the flip-side: if that is true, then it means anyone who wants DL capabilities should hire AI safety researchers! So what does that imply for those AI companies which have purged safety, and ensured that anyone interested in safety will think twice about working there...?

I think it implies that they are eating their seed-corn: scrapping any safety issues may work in the short run, but is self-sabotaging in the long run. (Like the man who works with his office door closed, who is highly productive now, but somehow, a few years later, is irrelevant.) The rot will set in long before it become clear publicly. OA will just slow down, look glossier and report ever bigger financial numbers, but increasingly forfeit its lead, and some point it stops being possible to say "oh, they're way ahead, it's just the product cycle right now; you'll see when they release the next model in a few months/years". And the Mandate of Heaven shifts elsewhere, irreversibly, as OA becomes just another place to work. (Startup & research culture mostly only degrades from the peak at their founding.) The visionaries go to Anthropic, or follow Ilya to SSI, or take a risk on Google, or go someplace small like Keen to bet big.

So, as you read AI news in the years to come, this is something to consider, an alternate narrative: there is a world where OA is doing just fine, and while many good researchers have left, it is ultimately not a big deal and OA continues to out-accelerate everyone else in the race to AGI; but there is another world where OA, like a victim of radiation poisoning, has already died and no one realizes it yet and by the time they do, it will be too late for OA to right ship.

There are not all that many worlds, I think, in between these two. Is the news each day more consistent with the former, or the latter? How would it look if OA had started rotting from the head ~9 months ago? ("Day 2 is death.")

This is also something to think about as companies rely more heavily on AI, and soon, start trying to go ~100% LLM by replacing most human employees with AI employees. Similar to the debate over remote work vs in-person collaboration: even if they do well on all ordinary day-to-day tasks—is this enough in the long run?

The LLMs may be hyper-competent and everything, but they may still lack that last little bit of spark and creativity, because of RLHF or something, that years later, one safe defensible decision after another, finally dooms the company to mediocrity/death.