“Programming Possibility: Kevin Scott on AI’s Impact on Cognitive Work”, 2022-10-18 ():
[Reid Hoffman podcast] …So for those who don’t know, GitHub Copilot is a programming assistant tool that can take natural language prompts, like you can express a program you would like to exist and it generates code for you. And shockingly, the performance of the system is improving at a pretty steady clip. But when we made it generally available a couple of months ago, it was producing more than 40% of the code that its users were producing overall. And qualitatively one of those tools where someone gets access to it, not everyone, they’re people who don’t like Copilot, which is fine, but many, many of its users are like, “This is so valuable to me. You will get it from my cold dead hands.”
So, two interesting things about Copilot. One is when we started development on it, we had evidence that large language models were actually going to be able to do this translation from natural language to code. And even when we showed people inside of Microsoft that this might be possible, we got a range of reactions from, “No, this isn’t real, this is impossible. It’s never going to work”, to, “Maybe it’ll work, but I’m highly skeptical.”
And so a big part of what we had to do is to overcome that sort of negative bias to actually get going because we had evidence not only that it was going to work, but we had a really concrete plan for how we were going to make it better, and better, and better over time. And so I think that is a thing that we’re seeing across the board with these foundation models. People sort of look at them, and there’s certainly a degree of hype around them, but there’s skepticism that they actually are going to be useful for the things that people want to build.
And I think the second thing, maybe the more profound thing about GitHub Copilot is it is just one Copilot of a potential, very many. What we were able to do with Copilot of automating this particular type of, not even automating, just assisting people with a particular type of cognitive work is going to be just directly applicable, replicable to a whole bunch of other domains. So, any sort of repetitive cognitive work is likely going to have a Copilot in the future. And the model that powers GitHub Copilot, OpenAI’s Codex model, really does let you think about software development in a different way. So there is now a mode of software development that you can do, which is having a conversation iteratively describing an application into existence.
So, it’s not one utterance or one prompt that generates an entire program. But if you say, “Here’s what I would like”, it generates something. And you’re like, “Okay, that’s good, but augment it in this way, change this way.” And so it’s a multi-turn dialogue that you’re having with this system to get an app. And I’ve got dozens of these demos that we built inside of Microsoft using the API. And increasingly as people get access to the Codex API itself, lots and lots of people are seeing the power of this.
…So, you make them bigger (and I’m making this sound way easier than it actually is), but you make them bigger, and they become more powerful at the task to which they’ve already been put in their smaller incarnations and they also become broader at the same time. So they can be used for a broader set of things than the previous smaller incarnations of the model we’re able to do. And so there’s just plenty of incentive to go invest in bigger and bigger iterations of these models and to make sure that the foundation that you’re building is more and more powerful over time.
The way that we think about it and the way that OpenAI, who’s our partner, thinks about this is we want to make them accessible through APIs so that you actually have a pretty rich third party developer ecosystem that’s building on top of the models. I don’t think… It’s hard to imagine what individual company—even one that was worth a trillion, a trillion and a half, two trillion [dollars]—whatever these big companies are that some of us work for are going to have enough imagination, and resources to build all of the things that can be built that will serve the public good, and humanity, and produce a whole lot of value ourselves. And so it’s just exciting.
I was browsing through the news yesterday, and just the number of excited articles about what people are doing with GPT-3, which is now a relatively ubiquitous thing, it’s 2.5 years old at this point, but there’s just a huge amount of energy around people building things on top of this, and that’s super exciting, and it just gets better and more interesting over time.
…Like in these numerical optimization systems, you just make a whole bunch of different assumptions. It’s like, “I’m going to make approximations to how I solve this wave equation. I’m going to compromise on the resolution of the system. I’m going to compromise on the number of time steps, or how big the time steps are that I’m making.”
And so what we are seeing in both of those styles of systems now (and you can pick up a copy of Nature or Science any given week and see someone using these techniques), is that you can put an AI self-supervised system into these simulation loops where it’s learning from the full granularity system. You just run it grindingly slow, at full resolution, and you train a model that learns something about that domain. And once you have the model, you put it into the core of the optimization loop and then things just sort of go faster.
There’s a bunch of research papers, a really good one from folks at Caltech. They won the best award for their papers on neural differential operators. They basically came up with a method of solving Navier-Stokes, which is the computational fluid dynamics, partial differential equations. And they applied it to airfoil design and they were getting 100,000× speed ups over the previous best-in-breed system without losing anything in terms of quality. Extraordinary.
I think there’s just a lot of opportunity there. It means better medicines. It means maybe we find the carbon-fixing catalyst that we don’t know about now. I’m just as excited, maybe more excited about that than some of the things that we get when we finally have a working quantum computer with more than 50 qubits.
…I think one of the things that we maybe have done over the past decade is we have overestimated the amount of change that AI is going to produce for industrial applications, and manufacturing, and these interfaces of technology in the real world. And we’ve underestimated how much impact it’s going to make to cognitive work.
And so I think in particular, any repetitive cognitive work, no matter how sophisticated it is, whether it’s programming or it’s thinking about experiment design, if you’re a physicist, or just pick your thing: marking up contracts, diagnosing illness, most of those things are entirely in scope for these AI systems. And I think people are going to be shocked this year to see how big a step we’re going to make again.
I think every year we get surprised by what happens. You and I are both friends with Demis Hassabis and even though they’re Google and not Microsoft, you have to just be awed by what DeepMind has done with AlphaFold and the contribution that they’ve made to science.
We had Copilot, we had AlphaFold’s protein data bank last year. I think the things coming this year are going to be even bigger, and most of them will directly impact cognitive work. And so that doesn’t mean that there are going to be a bunch of… I don’t think they’re going to be a bunch of AI lawyers or AI programmers that are going to do 100% of those jobs. It’s that we’re going to have real productivity gains for knowledge work in ways that we really haven’t had since maybe the onset of the Internet. And maybe more than the Internet.