“ChatGPT Goes to Law School”, 2023-01-25 ():
How well can AI models write law school exams without human assistance?
To find out, we used the widely publicized AI model ChatGPT to generate answers on 4 real exams at the University of Minnesota Law School. We then blindly graded these exams as part of our regular grading processes for each class. Over 95 multiple choice questions and 12 essay questions, ChatGPT:
performed on average at the level of a C+ student, achieving a low but passing grade in all 4 courses…In general, ChatGPT performed better on the essay components of the exams than on the multiple choice. Its average percentile performance on the essay questions (equally weighted across questions and exams) was the 17th percentile; its average performance on the multiple choice questions (equally weighted across exams) was the 7th percentile.
With respect to the essays, ChatGPT’s performance was highly uneven. In some cases, it matched or even exceeded the average performance of real students. On the other hand, when ChatGPT’s essay questions were incorrect, they were dramatically incorrect, often garnering the worst scores in the class. Perhaps not surprisingly, this outcome was particularly likely when essay questions required students to assess or draw upon specific cases, theories, or doctrines that were covered in class.
After detailing these results, we discuss their implications for legal education and lawyering. We also provide example prompts and advice on how ChatGPT can assist with legal writing.
[Keywords: ChatGPT, law school, AI, natural language processing, legal data, NLP, legal NLP, legal analytics, natural language understanding, evaluation, machine learning, artificial intelligence, artificial intelligence and law]
View External Link: