AI chatbot beats most aspiring lawyers on national legal ethics exam, study finds

OpenAI's GPT-4 chatbot scored higher than most people who took a legal ethics exam required for prospective lawyers in nearly every state, according to a new study.

The artificial intelligence software answered 74% of the questions correctly on a simulated Multistate Professional Responsibility Exam (MPRE), beating the estimated 68% average for human test takers nationwide, a report released on Thursday by LegalOn Technologies said. 

LegalOn Technologies sells AI software that reviews contracts, Reuters reported.

"Our study indicates that in the future it may be possible to develop AI to assist lawyers with ethical compliance and operate, where relevant, in alignment with lawyers’ professional responsibilities," the study authors wrote. 

MICROSOFT RESEARCHERS SAY GPT-4 SHOWS ‘SPARKS’ OF HUMAN-LEVEL PERFORMANCE

There is a growing body of research examining how AI will impact legal education and attorney licensure. An earlier study found that the previous version of GPT-4 earned passing but not stellar scores on law school final exams. Another more recent study found that GPT-4 can pass the bar exam. Earlier this month, researchers found that access to GPT-4 improved speed on legal writing assignments but didn’t bolster the quality of law students’ work.

GPT-4 is generative AI tech released in March by startup OpenAI, a partner of Microsoft. It is a large multimodal model – meaning it can be fed both images and text to come up with answers – and reportedly "exhibits human-level performance on various professional and academic benchmarks."

THOMSON REUTERS LAUNCHES GENERATIVE AI TOOLS FOR LEGAL RESEARCH

A spokesperson for the National Conference of Bar Examiners, which develops the MPRE, told Reuters it could not assess the study's claims that GPT-4 can pass its ethics test.

"The legal profession is always evolving in its use of technology, and will continue to do so," said National Conference spokesperson Sophie Martin. She added that "attorneys have a unique set of skills that AI cannot currently match."

YOUTUBE REQUIRING DISCLOSURE OF AI-GENERATED CONTENT, ADDING LABELS

Nearly every state except Wisconsin requires law students to pass the 60-multiple-choice MPRE before they are granted a license to practice. The test covers subjects including conflicts of interest, lawyer-client relationships and confidentiality. 

GPT-4 crushed the questions on conflict of interest, giving correct responses on 91% of those questions. It also scored 88% on questions about lawyer-client relationships. However, its answers on communications about legal services and safekeeping funds were less accurate, giving the correct response 71% and 72% of the time, respectively, Reuters reported.

GOOGLE RELEASES ‘AI OPPORTUNITY AGENDA’ FOR POLICYMAKERS

 "This research demonstrates for the first time that top-performing generative AI models can apply black-letter ethical rules as effectively as aspiring lawyers," the study said. 

Fox Business' Julia Musto and Reuters contributed to this report.