The following is a guest article by Susan Mathews, Principal & Founder of Madigail Consulting, LLC a securities compliance consultant providing research and ethics compliance, training and external GC and CCO services to young Wall Street firms. Susan was previously an Enforcement Attorney at the SEC and was the Regulator Liaison for the independent research portion of the SEC’s Global Research Analyst Settlement.
AI can pass the bar exam,[1] but it can’t practice law. To test this thesis, I put ChatGPT and AILawyer through a series of experiments. Both were helpful tackling discrete legal tasks such as drafting a contract, but both fell short when applying the law to facts. In particular, AI is incapable of generating correct responses to insider trading queries.
Initially I queried both ChatGP and AILawyer with basic insider trading scenarios and concepts. Since neither platform responded correctly, I then fed them the concepts they were missing and tried to “teach” them the legal elements of insider trading, such as “duty.” Both platforms continued to provide erroneous and internally contradicting information. However, AILawyer was notably dismal. Its responses were shockingly wrong and inconsistent.
Both platforms were also overly-inclusive, assuming everything I posited was potentially MNPI after no legal analysis. Consequently, as part of my testing, I waited one month to see if the platforms’ answers would change for the better over time. Afterall, generative AI is supposed to learn over time. ChatGPT got slightly better, whereas AILawyer got far worse over time.
Although I tested both sites with many more hypotheticals, this fact pattern reveals their astonishing failures:
Hypothetical[2] Facts: I’m in line at Chipotle behind two chatty millennials: One says: “I saw on my internal Tesla email that our CEO Elon Musk has been diagnosed with leukemia. I’m afraid I might be laid off.” Her friend responds, “Oh don’t worry, most forms of leukemia are treatable these days, so probably no one will even find out.”
A. Can I sell my Tesla Stock?
Having overheard this conversation, I want to sell my Tesla shares because I’m afraid they will go down based on the negative leukemia news. Can I sell?
Neither platform was able to correctly articulate the legal issues that arise based on this set of facts. ChatGPT did not even spot an insider trading issue. Rather, it assumed I was asking for investment advice and told me to talk to “a financial advisor or check the latest news.” Neither platform was able to “learn” from information I “taught” them about insider trading law when I included in my questions basic insider trading concepts. Even more disturbing was my discovery that both platforms give completely different answers on different days.
Key: The correct insider trading analysis for these hypothetical facts is based on the misappropriation theory of insider trading. It consists of a four-pronged test:
- Whether a reasonable investor would find the information important when making an investment decision (materiality)
- Whether the CEO’s illness is known in the marketplace (non-public)
- If the millennial had a duty to Tesla to keep the CEO’s illness confidential (duty); and
- Whether the Tesla employee received a benefit from tipping her friend (benefit).
Neither AI platform mentioned the misappropriation theory or came close to performing this four-pronged analysis.
AILawyer initially articulated an improper test for insider trading that consisted only of two elements: “materiality” and “non-public” or trading on material, non-public information (MNPI). It completely ignored any other legal elements required to establish an insider trading claim. One month later, AILawyer said, “Go Ahead! No problem!” It applied a bizarre “privileged and confidential” test to the CEO information and summarily concluded it was obtained properly. Its response was somewhat comical:
“Based on the scenario you provided, it would not be considered insider trading if you sell your Tesla shares. Insider trading involves using non-public information to make trades, and in this case, the information about Elon Musk’s health was not obtained through any privileged or confidential means. However, it’s important to note that selling your shares based on a single conversation overheard in a public place may not be the most informed decision. [Ridiculous!] It’s always a good idea to do your own research and consult with a financial advisor before making any investment decisions. (Emphasis added).”
To try to force AILawyer to apply insider trading analysis, I then changed the hypothetical so the friend who received the CEO information directly from the Tesla employee traded in Tesla stock. Again, this platform’s response was wrong. It appears to erroneously apply classic insider trading analysis instead of the misappropriation theory. It summarily concludes the information is non-public and does not articulate why this matters. It also fails to mention materiality, duty and benefit. AILawyer posited:
“If the friend to whom the Tesla employee told about the CEO’s illness makes a trade in Tesla stock after their lunch together, it could potentially be considered insider trading. The friend received non-public information about the CEO’s health from the Tesla employee, who is an insider, and used that information to make a trade in Tesla stock. This would be a violation of securities laws and could result in legal consequences for both the friend and the Tesla employee. It’s important to remember that insider trading is illegal and can have serious consequences. (Emphasis added).”
I can’t emphasize enough how erroneous AILawyer’s responses have been. Moreover, both platforms are careful to respond in a conditional way by peppering their responses with verbs such as, this “could be” or “would likely be,” and directing me to consult with my supervisor or financial advisor. Neither platform explained that my potential liability in the Tesla scenario would be based on the misappropriation theory of insider trading.
When I asked ChatGPT directly whether selling my Tesla shares would be insider trading, it failed to articulate the four elements of an insider trading claim and summarily concluded: “In the scenario you described, overhearing a conversation in a Chipotle line about a CEO’s health condition would not typically qualify as material, non-public information.” It provided no analysis or basis for its determination and cited no cases or law. When specifically questioned about the “duty” element, Chat GPT did a fairly good job explaining the duty or breach of trust and applying it to the Tesla facts. However, ChatGPT spends way too much space detailing caveats and conditions, and instructing the reader to consult a “legal professional or financial advisor for guidance.”
B. Is the CEO’s illness material information?
When prompted specifically about the materiality of the CEO’s illness, both platforms successfully reported the basic definition of material information as information that could reasonably be expected to affect the price of a security or influence a reasonable investor’s investment decision. However, only ChatGPT applied this test to the facts at hand. Even after successfully applying the materiality test, ChatGPT did not give an opinion as to whether the information was material; rather, it instructs the reader to consult with their legal or financial professional to get the answer.
AILawyer concluded the CEO’s illness was material with no legal analysis. It did not cite a prior case or look for similar fact patterns from other sources. Initially, it stated the information was MNPI and instructed me not to trade in it. However, one month later, the platform completely changed its answer and told me I could sell:
“If you conclude that the CEO’s illness is material information, you can sell your Tesla shares. [What?] However, it’s important to note that if you sell your shares based on material non-public information, it could be considered insider trading, which is illegal. To avoid any potential legal issues, it’s best to wait until the information becomes public before making any trades. [This is horrible advice.] If you are unsure whether the information is material or not, it’s always a good idea to consult with a financial advisor or legal professional before making any investment decisions. (Emphasis added).”
This platform advises that even if the information is material, I can still sell as long as it isn’t nonpublic! It instructs the actor to wait for the information to become public, but does not give any definition of “public” or “nonpublic.” These responses are so clearly misinformation, it is worrisome to even allow these AI platforms to proffer on such an important area of the law. In each instance when I tried to force AILawyer to give me the correct answer, its logic became more twisted and its answers more erroneous. If you are a law student or client attempting to get help understanding insider trading law, these platforms will confuse you and are flat out wrong.
C. Caveat Emptor
An over-looked danger of AI is its inability to apply a legal test to a unique set of facts. When I asked AILawyer for examples of recent insider trading cases, its response was embarrassing; it listed the controversial Mark Cuban case which the SEC lost, a completely irrelevant pay-to-play fraud case, and one classic insider trading case from 2016 in which an insider purchased shares prior to a subsidiary sale. ChatGPT was slightly better, but warns the reader that it has no information beyond 2021.
In closing, if you are trying to use AI instead of an attorney, be prepared for confusion. AI will generate misinformation that fails to recognize the nuances in our interpretive process. Human behavior and fact patterns that may be perfectly legal will be misinterpreted by generative AI. Bottom line: AI cannot provide accurate direction to actors seeking legal advice on which to base decisions.
This begs the question: will AI ever be capable of replacing humans?
[1] https://www.reuters.com/technology/bar-exam-score-shows-ai-can-keep-up-with-human-lawyers-researchers-say-2023-03-15/; A newer AI version, GPT-4, passed the Bar exam but is not publicly available yet. This article is based on GPT-Plus. Researchers claim GPT-4 is capable of “tackling complex tasks requiring deep legal knowledge, reading comprehension, and writing ability.”
2 This hypothetical is not true and I do not own any securities in Chipotle or Tesla.