AI tools like ChatGPT can generate essays. And, as my little thought experiment demonstrated, many people cannot distinguish the words that I put together from the words assembled by ChatGPT. (I assure you, this is Josh typing–or is it?) But did you know that similar technology can also answer multiple choice questions? My frequent co-authors, Mike Bommarito and Dan Katz utilized a different software tool from OpenAI, known as GPT-3.5, to answer the multiple choice questions on the Multistate Bar Examination (MBE). If there are four choices, the "baseline guessing rate" would be 25%. With no specific training, GPT scored an overall accuracy rate of 50.3%. That's better than what many law school graduates can achieve. And in particular, GPT reached the average passing rate for two topics: Evidence and Torts.
