A year ago we spent Christmas in the hospital as my wife had a serious health issue Doctors misdiagnosed and mistreated her for weeks, but eventually a specialist figured it out Months later, GPT4 came out and correctly diagnosed given only the notes from the first doctor

Dec 27, 2023 · 3:59 PM UTC

Since this experience, whenever I make friends with a new doctor, I quiz them to see if they would have correctly diagnosed the issue Around 50% get it right, and to the ones who get it right, it's obvious and they're astounded anyone with a medical license could get it wrong
I'm not in the habit of actively wishing for AI to take anyone's jobs... but in the case of the 50% who might've killed my wife... ¯\_(ツ)_/¯
Replying to @GrantSlatton
My sister did the same thing with her husband's ailment (pneumonia) just a few weeks ago and got exactly the final diagnoses the doctors settled on several days sooner just from the notes and blood test. Copy, paste, ask for explanation->correct diagnoses.
It's also great for translating radiology reports into non-jargon English
Replying to @GrantSlatton
Wow. And I imagine it doesn’t have access to that many medical records yet
In this case, just having read all of Wikipedia would have gotten you there. But I bet you could do much better if you trained on real medical records.
Replying to @GrantSlatton
Thanks for sharing this incredible story Grant! (I honestly thought that GPT usefulness outside coding was a myth) I know it might not be emotionally easy, but did you try other models, especially OSS ones to do the same?
Just tested, gave doctor notes and asked for some chain of thought + top 3 guesses GPT4 - 1st guess Claude 2.1 - 1st guess Claude 1.2 - 2nd guess Cohere - 3rd guess GPT3.5 - not in top 3 guesses Mistral 7B - not in top 3 guesses
Replying to @GrantSlatton
Did you use a specific introductory prompt or custom instructions? Curious how to squeeze it into accuracy.
All I did was: {doctor notes} Write some initial thoughts about the notes. Then, give the top 3 most likely diagnoses.