“What OpenAI Really Wants”, 2023-09-05 (; backlinks):
…They’ve just ducked out of one event and are headed to another, then another, where a frenzied mob awaits. As they careen through the streets of London—the short hop from Holborn to Bloomsbury—it’s as if they’re surfing one of civilization’s before-and-after moments. The history-making force personified inside this car has captured the attention of the world. Everyone wants a piece of it, from the students who’ve waited in line to the prime minister.
Inside the luxury van, wolfing down a salad, is the neatly coiffed 38-year-old entrepreneur Sam Altman, cofounder of OpenAI; a PR person; a security specialist; and me. Altman is unhappily sporting a blue suit with a tieless pink dress shirt as he whirlwinds through London as part of a month-long global jaunt through 25 cities on 6 continents. As he gobbles his greens—no time for a sit-down lunch today—he reflects on his meeting the previous night with French president Emmanuel Macron. Pretty good guy! And very interested in artificial intelligence. As was the prime minister of Poland. And the prime minister of Spain.
…Altman didn’t do the research, train the neural net, or code the interface of ChatGPT and its more precocious sibling, GPT-4. But as CEO—and a dreamer/doer type who’s like a younger version of his cofounder Elon Musk, without the baggage—one news article after another has used his photo as the visual symbol of humanity’s new challenge…Altman’s van whisks him to 4 appearances that sunny day in May…Altman is not a natural publicity seeker. I once spoke to him right after The New Yorker ran a long profile of him. “Too much about me”, he said. But at University College, after the formal program, he wades into the scrum of people who have surged to the foot of the stage. His aides try to maneuver themselves between Altman and the throng, but he shrugs them off. He takes one question after another, each time intently staring at the face of the interlocutor as if he’s hearing the query for the first time. Everyone wants a selfie. After 20 minutes, he finally allows his team to pull him out. Then he’s off to meet with UK prime minister Rishi Sunak.
…The people who work at OpenAI are fanatical in their pursuit of that goal. (Though, as any number of conversations in the office café will confirm, the “build AGI” bit of the mission seems to offer up more raw excitement to its researchers than the “make it safe” bit.) It’s not fair to call OpenAI a cult, but when I asked several of the company’s top brass if someone could comfortably work there if they didn’t believe AGI was truly coming—and that its arrival would mark one of the greatest moments in human history—most executives didn’t think so. Why would a nonbeliever want to work here? they wondered. The assumption is that the workforce—now at ~500, though it might have grown since you began reading this paragraph—has self-selected to include only the faithful. [Altman continues to interview all employees, he has claimed.] At the very least, as Altman puts it, once you get hired, it seems inevitable that you’ll be drawn into the spell.
…At the time [that he was fired from YC], Altman had been thinking about running for governor of California. [He did considerably more than ‘think’; see the New Yorker profile.] But he realized that he was perfectly positioned to do something bigger—to lead a company that would change humanity itself. “AGI was going to get built exactly once”, he told me in 2021. “And there were not that many people that could do a good job running OpenAI. I was lucky to have a set of experiences in my life that made me really positively set up for this.”
…OpenAI’s road to relevance really started with its hire of an as-yet-unheralded researcher named Alec Radford, who joined in 2016, leaving the small Boston AI company he’d cofounded in his dorm room. After accepting OpenAI’s offer, he told his high school alumni magazine that taking this new role was “kind of similar to joining a graduate program”—an open-ended, low-pressure perch to research AI. The role he would actually play was more like Larry Page inventing PageRank.
Radford, who is press-shy and hasn’t given interviews on his work, responds to my questions about his early days at OpenAI via a long email exchange. His biggest interest was in getting neural nets to interact with humans in lucid conversation…His first experiment involved scanning 2 billion Reddit comments to train a [RNN] language model. Like a lot of OpenAI’s early experiments, it flopped. No matter. The 23-year-old had permission to keep going, to fail again. “We were just like, Alec is great, let him do his thing”, says Brockman…Sutskever and others encouraged Radford to expand his experiments beyond Amazon reviews, to use his insights to train neural nets to converse or answer questions on a broad range of subjects…Radford began experimenting with the transformer architecture. “I made more progress in two weeks than I did over the past two years”, he says. He came to understand that the key to getting the most out of the new model was to add scale—to train it on fantastically large data sets. The idea was dubbed “Big Transformer” by Radford’s collaborator Rewon Child.
This approach required a change of culture at OpenAI and a focus it had previously lacked. “In order to take advantage of the transformer, you needed to scale it up”, says Adam D’Angelo, the CEO of Quora, who sits on OpenAI’s board of directors. “You need to run it more like an engineering organization. You can’t have every researcher trying to do their own thing and training their own model and make elegant things that you can publish papers on. You have to do this more tedious, less elegant work.” That, he added, was something OpenAI was able to do, and something no one else did.
…The name that Radford and his collaborators gave the model they created was an acronym for “generatively pretrained transformer”—GPT-1. Eventually, this model came to be generically known as “generative AI.” To build it, they drew on a collection of 7,000 unpublished books, many in the genres of romance, fantasy, and adventure, and refined it on Quora questions and answers, as well as thousands of passages taken from middle school and high school exams. All in all, the model included 117 million parameters, or variables. And it outperformed everything that had come before in understanding language and generating answers. But the most dramatic result was that processing such a massive amount of data allowed the model to offer up results beyond its training, providing expertise in brand-new domains. These unplanned robot capabilities are called zero-shots. They still baffle researchers—and account for the queasiness that many in the field have about these so-called large language models.
Radford remembers one late night at OpenAI’s office. “I just kept saying over and over, ’Well, that’s cool, but I’m pretty sure it won’t be able to do x.’ And then I would quickly code up an evaluation and, sure enough, it could kind of do x.”…“I underappreciated how much making an easy-to-use conversational interface to an LLM [ChatGPT] would make it much more intuitive for everyone to use”, says Radford.
…By early 2018, OpenAI was starting to focus productively on large language models, or LLMs. But Elon Musk wasn’t happy. He felt that the progress was insufficient—or maybe he felt that now that OpenAI was on to something, it needed leadership to seize its advantage. Or maybe, as he’d later explain, he felt that safety should be more of a priority. Whatever his problem was, he had a solution: Turn everything over to him. He proposed taking a majority stake in the company, adding it to the portfolio of his multiple full-time jobs (Tesla Motors, SpaceX) and supervisory obligations (Neuralink and the Boring Company).
Musk believed he had a right to own OpenAI. “It wouldn’t exist without me”, he later told CNBC. “I came up with the name!” (True.) But Altman and the rest of OpenAI’s brain trust had no interest in becoming part of the Muskiverse. When they made this clear, Musk cut ties, providing the public with the incomplete explanation that he was leaving the board to avoid a conflict with Tesla’s AI effort. His farewell came at an all-hands meeting early that year where he predicted that OpenAI would fail. And he called at least one of the researchers a “jackass”.
He also took his money with him. Since the company had no revenue, this was an existential crisis. “Elon is cutting off his support”, Altman said in a panicky call to [OA co-founder] Reid Hoffman. “What do we do?” Hoffman volunteered to keep the company afloat, paying overhead and salaries.
But this was a temporary fix; OpenAI had to find big bucks elsewhere. Silicon Valley loves to throw money at talented people working on trendy tech. But not so much if they are working at a nonprofit. It had been a massive [ask] for OpenAI to get its first billion. To train and test new generations of GPT—and then access the computation it takes to deploy them—the company needed another billion, and fast. And that would only be the start.
…But accounting is critical. A for-profit company optimizes for, well, profits. There’s a reason why companies like Facebook feel pressure from shareholders when they devote billions to R&D. How could this not affect the way a firm operates? And wasn’t avoiding commercialism the reason why Altman made OpenAI a nonprofit to begin with? According to COO Brad Lightcap, the view of the company’s leaders is that the board, which is still part of the nonprofit controlling entity, will make sure that the drive for revenue and profits won’t overwhelm the original idea. “We needed to maintain the mission as the reason for our existence”, he says, “It shouldn’t just be in spirit, but encoded in the structure of the company.” Board member Adam D’Angelo says he takes this responsibility seriously: “It’s my job, along with the rest of the board, to make sure that OpenAI stays true to its mission.”
…There is, however, a hitch: At the moment, OpenAI doesn’t claim to know what AGI really is. The determination would come from the board, but it’s not clear how the board would define it. When I ask Altman, who is on the board, for clarity, his response is anything but open. “It’s not a single Turing test, but a number of things we might use”, he says. “I would happily tell you, but I like to keep confidential conversations private. I realize that is unsatisfyingly vague. But we don’t know what it’s going to be like at that point.”
…Those caveats didn’t stop some of the smartest venture capitalists from throwing money at OpenAI during its 2019 funding round. At that point, the first VC firm to invest was Khosla Ventures, which kicked in $50 million. According to Vinod Khosla, it was double the size of his largest initial investment. “If we lose, we lose $50m bucks”, he says. “If we win, we win $5b.” [cf. his reaction] Others investors reportedly would include elite VC firms Thrive Capital, Andreessen Horowitz, Founders Fund, and Sequoia Capital.
The shift also allowed OpenAI’s employees to claim some equity. But not Altman. He says that originally he intended to include himself but didn’t get around to it. Then he decided that he didn’t need any piece of the $30 billion company that he’d cofounded and leads. “Meaningful work is more important to me”, he says. “I don’t think about it. I honestly don’t get why people care so much.”
Because … not taking a stake in the company you cofounded is weird?
“If I didn’t already have a ton of money, it would be much weirder”, he says. “It does seem like people have a hard time imagining ever having enough money. But I feel like I have enough.” (Note: For Silicon Valley, this is extremely weird.) Altman joked that he’s considering taking one share of equity “so I never have to answer that question again.”
…Musk’s jibes might be dismissed as bitterness from a rejected suitor, but he wasn’t alone. “The whole vision of it morphing the way it did feels kind of gross”, says John Carmack. [Carmack has founded his own commercial DL startup, Keen Technologies] (He does specify that he’s still excited about the company’s work.) Another prominent industry insider, who prefers to speak without attribution, says, “OpenAI has turned from a small, somewhat open research outfit into a secretive product-development house with an unwarranted superiority complex.”…[Anthropic defection summary omitted]…Another OpenAI defector was Rewon Child, a main technical contributor to the GPT-2 and GPT-3 projects. He left in late 2021 and is now at Inflection AI, a company led by former DeepMind cofounder Mustafa Suleyman.
Altman professes not to be bothered by defections, dismissing them as simply the way Silicon Valley works. “Some people will want to do great work somewhere else, and that pushes society forward”, he says. “That absolutely fits our mission.”
…OpenAI didn’t shrug off discussion of those perils, but presented itself as the force best positioned to mitigate them. “We had 100-page system cards on all the red-teaming safety valuations”, says Makanju. [It actually seems to be 60 pages; Makanju may be thinking of the GPT-4 paper, which was focused on capability benchmarking, not safety.] (Whatever that meant, it didn’t stop users and journalists from endlessly discovering ways to jailbreak the system.)
…One study indicated that more recent versions of GPT, which have improved safety features, are actually dumber than previous versions, making errors in basic math problems that earlier programs had aced. (Altman says that OpenAI’s data doesn’t confirm this. “Wasn’t that study retracted?” he asks. No.) [But it should’ve been.]
…While he has endorsed, in principle, the idea of an international agency overseeing AI [like the IAEA], he does feel that some proposed rules, like banning all copyrighted material from data sets, present unfair obstacles. He pointedly didn’t sign a widely distributed letter urging a 6-month moratorium on developing more powerful AI systems. But he and other OpenAI leaders did add their names to a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.” Altman explains: “I said, ’Yeah, I agree with that. One-minute discussion.”
As one prominent Silicon Valley founder notes, “It’s rare that an industry raises their hand and says, ‘We are going to be the end of humanity’—and then continues to work on the product with glee and alacrity.”
…Still, as the company takes on more tasks and devotes more energy to commercial activities, some question how closely OpenAI can concentrate on the mission—especially the “mitigating risk of extinction” side. “If you think about it, they’re actually building 5 businesses”, says an AI industry executive, ticking them off with his fingers. “There’s the product itself, the enterprise relationship with Microsoft, the developer ecosystem, and an app store. And, oh yes—they are also obviously doing an AGI research mission.” Having used all 5 fingers, he recycles his index finger to add a 6th. “And of course, they’re also doing the investment fund”, he says, referring to a $175 million project to seed startups that want to tap into OpenAI technology. “These are different cultures, and in fact they’re conflicting with a research mission.”
I repeatedly asked OpenAI’s execs how donning the skin of a product company has affected its culture. Without fail they insist that, despite the for-profit restructuring, despite the competition with Google, Facebook, and countless startups, the mission is still central. Yet OpenAI has changed. The nonprofit board might technically be in charge, but virtually everyone in the company is on the for-profit ledger. Its workforce includes lawyers, marketers, policy experts, and user-interface designers. OpenAI contracts with hundreds of content moderators to educate its models on inappropriate or harmful answers to the prompts offered by many millions of users. It’s got product managers and engineers working constantly on updates to its products, and every couple of weeks it seems to ping reporters with demonstrations—just like other product-oriented Big Tech companies. Its offices look like an Architectural Digest spread. I have visited virtually every major tech company in Silicon Valley and beyond, and not one surpasses the coffee options in the lobby of OpenAI’s headquarters in San Francisco.
…“I can’t emphasize this enough—we didn’t have a master plan”, says Altman. “It was like we were turning each corner and shining a flashlight. We were willing to go through the maze to get to the end.” Though the maze got twisty, the goal has not changed. “We still have our core mission—believing that safe AGI was this critically important thing that the world was not taking seriously enough.”
…Tom Rubin, an elite intellectual property lawyer who officially joined OpenAI in March, is optimistic that the company will eventually find a balance that satisfies both its own needs and that of creators…Rubin is nonplussed when I ask him whether he believes, as an article of faith, that AGI will happen and if he’s hungry to make it so. “I can’t even answer that”, he says after a pause. When pressed further, he clarifies that, as an intellectual property lawyer, speeding the path to scarily intelligent computers is not his job. “From my perch, I look forward to it”, he finally says.
View External Link: