all 89 comments
sorted by:
best
formatting helphide helpcontent policy

[–]VectorSpaceModel 47 points48 points  (0 children)

TAs around the world are rejoicing

[–]phobos_0 168 points169 points  (35 children)

Dude this is dope

[–]uoftsuxalot 147 points148 points  (33 children)

Not to take anything away from this project, but it’s just an api call to gpt3 with prompt “fix this error {error}”. I thought there was some training and fine tuning, but I guess LLMs can do it all now a days

[–]jsonathan[S] 104 points105 points  (12 children)

Yeah, right now it’s just a thin wrapper around GPT-3, but there’s a lot that could be done to improve it, like using static code analysis to build a better prompt or even training a more specialized model (like this).

[–]2Punx2Furious 7 points8 points  (2 children)

Even fine-tuning the prompt could get much better results. Prompt engineering is important.

[–]datamakesmydickhard 1 point2 points  (1 child)

Has it really come to this

[–]2Punx2Furious 0 points1 point  (0 children)

Yes, it's been like this for a while now.

[–]ginger_beer_m 2 points3 points  (4 children)

How did you deal with incorrectness from ChatGPT?

[–]jsonathan[S] 39 points40 points  (0 children)

I didn't. Adrenaline won’t always correctly fix your error, but it can at least give you a starting point.

[–]kelkulus 17 points18 points  (2 children)

Well for one, he's not using ChatGPT. GPT-3 is not the same.

[–]danielswrath 7 points8 points  (1 child)

GPT-3 has the same problem though. ChatGPT is a successor of GPT-3, so it's not the same but it's not extremely different either.

[–]Glum-Bookkeeper1836 2 points3 points  (0 children)

I'm not sure if we know this for certain, but it appears to be davinci instruct 3 with a custom prompt prefix.

[–][deleted]  (2 children)

[deleted]

    [–]jsonathan[S] 3 points4 points  (1 child)

    I haven’t used this yet, but my understanding is it can explain code and generate unit tests. It can’t explain and fix errors.

    [–]cloudedleopard42 0 points1 point  (0 children)

    is it possible to fine tune GPT for static code analysis ? if yes...what would be the training set looks like?

    [–]satireplusplus 17 points18 points  (19 children)

    LLMs are our new overlords, it's crazy

    [–]2Punx2Furious 1 point2 points  (18 children)

    And it's not even AGI yet. The singularity is closer than a lot of people think.

    [–]TrueBirch 3 points4 points  (17 children)

    I don't think AGI will ever happen, but with enough task-specific applications, the difference may become academic.

    [–]iamnotlefthanded666 1 point2 points  (3 children)

    Why don't you think AGI will ever happen?

    [–]TrueBirch 0 points1 point  (2 children)

    Check out this comment. Some things that we take for granted from low-wage humans are incredibly hard for computers and robots. Think about valet parking. Our society doesn't think "Oh my goodness, valet parkers are geniuses!!!" But it's really really hard to build a robot that can do what they do.

    [–]TradeApe 0 points1 point  (1 child)

    If they can automate huge chunks of super busy cargo harbors, they can automate valet parking...and they won't even need AGI for that. Hell, valet parking will likely become obsolete once full self driving is here.

    People also didn't think AI will make artists obsolete...but here we are.

    [–]TrueBirch 0 points1 point  (0 children)

    Artists are hardly obsolete. Photoshop didn't make them obsolete and generative AI won't either. And I say that as someone who has extensively used Stable Diffusion for work and personal projects.

    Regarding valets, I'm referring to the ability to toss your keys to a robot and have it drive your car. Even when true self driving cars are first produced (which always seems to be ten years away), we'll be a long way away from a robot being able to park a non-automated car. That's just one example of a task that seems really easy for humans but is shockingly hard for robots. Folding laundry is another one, which is especially relevant since I'm ignoring the fact that my dryer just finished a load.

    [–]2Punx2Furious 0 points1 point  (12 children)

    Yeah, I see a lot of goalpost-moving, but in the end, it depends on how you define "AGI", some people have varying definitions. I think even a language model can become AGI eventually.

    [–]TrueBirch 1 point2 points  (11 children)

    There are some things that are incredibly hard. Imagine you work on a farm. You toss the keys to the ATV to a 17yo farmhand who's never worked for you before. You say, "Head over to field 3 and tell me if it's dry enough to plow. You can see where it is on this paper map. Radio back using this handheld." The farmhand duly drives the ATV to field 3, sees that it's muddy, picks up the radio, and says, "Sorry boss, field 3's a no-go."

    We're a long way from a robotic farmhand being able to perform those skills, certainly not for a price comparable to a farm laborer.

    You could definitely train an application-specific AI to monitor fields and report on their moisture levels. You could even have an algorithm that schedules all of your farm equipment based on current conditions and other factors. So it's not that AI can't revolutionize how we work, it's just that it'll be different from true AGI.

    [–]2Punx2Furious -1 points0 points  (4 children)

    We're a long way from a robotic farmhand being able to perform those skills, certainly not for a price comparable to a farm laborer.

    If we get AGI, we automatically get that as well, by definition. Those you listed are all currently hard problems, yes, but an AGI would be able to do them, no problem.

    The issue is, will AGI ever be achieved, and if yes, when?

    I think the answer to the first one is simple, the second one not as much.

    The answer (in very short) is: Most likely yes, unless we go extinct first. Because we know that general intelligence is possible, so I see no reason why it shouldn't be possible to replicate artificially, and even improve it, and several, very wealthy companies are actively working on it, and the incentive to achieve it is huge.

    As for the when, it's impossible to know until it happens, and even then, some people will argue about it for a while. I have my predictions, but there are lots of disagreeing opinions.

    I don't know how someone even remotely interested in the field could think it will never happen for sure.

    As for my prediction/opinion, I actually give it a decent chance of it happening in the next 10-20 years, with probability increasing every year until the 2040s. I would be very surprised if it doesn't happen by then, but of course, there is no way to tell.

    [–]TrueBirch 0 points1 point  (3 children)

    A true AGI has way too many edge cases to be possible in the timeframe you describe. It's also not necessary to create AGI in order to make a lot of money from AI. You can find the specific jobs that you want to replace and create a task-specific AI to do it.

    [–]2Punx2Furious 0 points1 point  (2 children)

    True that you don't need AGI to disrupt everything. But I don't think the edge cases matter, it's not like it will be coded manually.

    [–]eldenrim 0 points1 point  (5 children)

    I'm curious how you feel about the following:

    There are humans that can't do the task you outlined. Why use it as a metric for AGI? Put in other words, what about a "less intelligent" AGI, that crawls before it walks? An AGI equivalent to a human with lower IQ, or some similar measurement that correlates with not being capable of the same things as those in your example?

    Second, if an A.I can do 80% of what a human can, and a human can do 10% of what an A.I can, would you still claim the system isn't an AGI? As in, if humans can do X, A.I can do X * 100 things, but there's a venn diagram with some things unique to humans and many things unique to A.I, does it not count because you can point to human examples of tasks it cannot complete?

    Finally, considering a human system has to account for things irrelevant to an AGI (body homeostasis with heart rate and such, immune system, etc) and an AGI can build on code before it, what do you see as the barrier to AGI? Is it not a matter of time?

    [–]TrueBirch 0 points1 point  (4 children)

    I think "AGI" is a silly concept overall and never really happening. Computers are good at doing things in different ways from humans. Rather than chasing AGI, you can make a lot more of an impact by leveraging a computer's strengths and avoiding its weaknesses.

    For my example, I picked an occupation with an average salary south of $30,000/year (source). I'm not saying everybody can do it, but the market puts a price on this kind of labor that suggests many people can do it. A true AGI system could replicate how a low-salary human does a job. In reality, a computerized system would use a few wireless sensors that call home instead of physically driving around looking at fields.

    Similarly, consider meter readers, another low-wage job. Imagine what it would take to create a robot that could drive from house to house, get out of the car, find the power meter, gently move anything blocking it, and take a reading. Instead, utilities use smart meters that call home. It's cheaper, more reliable, and simpler.

    It's beyond hard to create a true AGI system, and there are plenty of ways to make tons of money with application-specific systems.

    [–]eldenrim 0 points1 point  (3 children)

    I'm currently interested in ML to alleviate the suffering of my disabled partner and myself, I just enjoy theoretical discussion with AGI.

    Maybe making money will come later. :)

    [–]jsonathan[S] 6 points7 points  (0 children)

    Thanks! Feel free to try it out here. Let me know if any of y’all get some impressive bug fixes.

    [–]jsonathan[S] 51 points52 points  (11 children)

    Try it out here: https://useadrenaline.com

    [–]jsonathan[S] 45 points46 points  (9 children)

    Right now, this is just a simple demo of what’s possible with AI-driven debugging. But I’d like to build it out so that instead of just explaining errors, Adrenaline provided a ChatGPT-style assistant that can answer questions about your error, and teach you during the debugging process.

    This is open-source, so if anyone’s interested in contributing, here’s the GitHub repository: https://github.com/shobrook/adrenaline

    [–]ddproxy 52 points53 points  (7 children)

    Does it come with an animated assistant in the shape of a paperclip?

    [–]_swnt_ 10 points11 points  (6 children)

    That's would be an actually useful paperclip 😂

    [–]StuntHacks 2 points3 points  (5 children)

    Maybe we should make more of em

    [–]jsonathan[S] 8 points9 points  (4 children)

    What could go wrong?

    [–]lucidrage 1 point2 points  (3 children)

    They become self replicating.

    [–]0x2113 1 point2 points  (1 child)

    That just means we'd have more paperclips. I see no downside here.

    [–]Glum-Bookkeeper1836 0 points1 point  (0 children)

    Paperclip stan

    [–]ImPetarded 1 point2 points  (0 children)

    ...it didn't end like we thought it would in the movies. There were no killer machines....there were paper clips, trillions of them....

    [–][deleted] 6 points7 points  (0 children)

    Yep

    [–]Accomplished-Low3305 9 points10 points  (0 children)

    It would be nice to have some metric to evaluate how good is GPT-3 solving bugs. In my experience it only works fine for simple bugs, such as using an incorrect variable.

    [–]GoofAckYoorsElf 12 points13 points  (7 children)

    This is all great. The only problem is that I can't use it due to non-disclosure and IP protection of my employer. As long as I have to send code over the web, it's a no-no.

    [–][deleted] 11 points12 points  (2 children)

    Yeah I imagine that will be an issue for lots of people. What's the SotA in open source LLMs?

    I looked it up. Apparently it's BLOOM. Slightly bigger than GPT-3. No idea if it is better.

    You need a DGX A100 to run it (only $150k!).

    [–]Soundwave_47 4 points5 points  (0 children)

    Anecdotally, it is comparable.

    [–]LetterRip 1 point2 points  (0 children)

    I'd do GLM-130B

    With INT4 quantization, the hardware requirements can further be reduced to a single server with 4 * RTX 3090 (24G) with almost no performance degradation.

    https://github.com/THUDM/GLM-130B

    I'd also look into pruning/distillation and you could probably shrink the model by about half again.

    [–]--algo 1 point2 points  (1 child)

    How do you deal with source code hosting?

    [–]GoofAckYoorsElf 1 point2 points  (0 children)

    A cloud hosted GitLab with customer managed keys. We have a very detailed IP and security agreement with our cloud provider.

    [–]keepthepace 0 points1 point  (1 child)

    I am willing to be that 99% of the code is overprotected and no one in OpenAI would spend valuable time looking at it.

    These protections mostly exist to justify some bullshit jobs within the company.

    [–]GoofAckYoorsElf 1 point2 points  (0 children)

    Probably. I'm still getting fired if I do something like that without permission.

    [–][deleted]  (4 children)

    [deleted]

      [–]ksblur 10 points11 points  (3 children)

      Just wait till you see what the rate for management will be after LLMs come for their jobs.

      Managers are mostly people-interaction-managers, and LLMs are already 10x better at that than they are at creating novel code.

      [–]scaredandconfussled 4 points5 points  (0 children)

      Don't give me hope like that.

      [–]keepthepace 1 point2 points  (0 children)

      I was going to argue that employees will be able to bullshit their automated manager easily but well, it is not like humans are much better at handling it.

      [–]yerop82726 2 points3 points  (0 children)

      Nice one. Keen to see the vscode extension!

      [–]Jbonez87 1 point2 points  (0 children)

      This is pretty cool dude!

      [–]troubletmill 1 point2 points  (0 children)

      This is really good mate.

      [–]davidswelt 1 point2 points  (0 children)

      OK, how did you evaluate it? How do you tell it's working well or not?

      [–]fnetma 1 point2 points  (0 children)

      An AI debugger would be very helpful. I actually see that as a use case...

      [–]Mikatron3000 1 point2 points  (0 children)

      Not sure if this is already there but it might be worth adding some license information here since sending closed source code over an open sourced API / model might become a no no in the future legally. I guess that would be the problem of making this an Intellij / vscode plugin

      [–]devinhedge 1 point2 points  (0 children)

      This is cool. How do we give feedback to the training engine so that it improves over time?

      [–]EarthAdmin 1 point2 points  (0 children)

      Would love this to be a VSCode plug-in! Happy to drop our OpenAI api key in there.

      [–]RKHS 2 points3 points  (0 children)

      This is a fairly useless example. It's simply a rewording of the error. Do you have any examples that are non trivial?

      [–]naiq6236 0 points1 point  (0 children)

      This would be a game changer dude

      [–]KuzonFire11 0 points1 point  (0 children)

      u/jsonathan Shoot me a dm, I'd love to do a VSCode ext.

      [–]coolcake 0 points1 point  (0 children)

      amazing

      [–]outthemirror 0 points1 point  (0 children)

      If this was a pycharm/vscode plugin….

      [–]sublimegeek 0 points1 point  (0 children)

      Shared with my discord

      [–]ItsAllJustASickGame 0 points1 point  (2 children)

      Bruuuuuh is this generally reliable? And if so, where can I get it?!

      [–]jsonathan[S] 1 point2 points  (1 child)

      You can use it here: https://useadrenaline.com

      [–]ItsAllJustASickGame 0 points1 point  (0 children)

      Awesome thank you!

      [–]Eastern_Care_6369 0 points1 point  (1 child)

      Can you turn this into an IOS app

      [–]Think_Olive_1000 0 points1 point  (0 children)

      Why the fuck are you coding on an iPhone - if you're going to use a phone at least be android

      [–]Funny_Willingness433 0 points1 point  (0 children)

      That is good.

      [–]Eastern_Incident4922 0 points1 point  (1 child)

      Can i try this

      [–]Own-Cherry6760 0 points1 point  (0 children)

      Which model are you using from openAI ?

      [–]HoneyEatingPunkKid 0 points1 point  (0 children)

      vscode extension pls

      [–]the_night_question 0 points1 point  (0 children)

      Really cool!!!!!

      [–]LucasLeroy19 0 points1 point  (0 children)

      My question to you is, why the name Adrenaline? How did you come up with that?