My Recent Eye Opening ChatGPT Chat

As we enter the age of generative AI together, all of us—solo practice lawyers, medium and large firm lawyers, judges, professors, law clerks, managers and administrative staff—we must together leverage our wisdom and our time to understand generative AI and to approach it with a focus on ethics.

Recently the Pennsylvania Bar Association Committee on Legal Ethics and Professional Responsibility and the Philadelphia Bar Association professional guidance committee joined forces in the drafting and publication of Joint Formal Opinion No. 2024-200, titled “Ethical Issues Regarding the Use of Artificial Intelligence.” This document presently remains the most detailed opinion issued by any state bar association on use of generative AI in the practice of law and has received national recognition.

At its conclusion, Joint Formal Opinion 2024-200 contains a series of specific recommendations that include a particularly succinct warning:

Verifying All Citations & The Accuracy of Cited Materials: Lawyers must ensure the accuracy and relevance of the citations they use in legal documents or arguments. When citing legal authorities such as case law, statutes, regulations or scholarly articles, lawyers should verify that the citations accurately reflect the content they are referencing.

Understanding requires experimentation. Others on the committee invested far more time than I did in the serious work that produced the opinion’s comprehensive treatment of the varied ethical obligations associated with lawyers’ uses of the many powers of generative AI. Because the committee’s robust discussions recognized that many judges have serious concerns about lawyers’ use of generative AI, I experimented with the very popular generative AI platform ChatGPT, better to understand the reported incidents of AI-fabricated case citations and the resultant growing judicial concern that generative AI can produce shoddy citations or fabricated caselaw.

First I learned that ChatGPT is in its infancy; it was released on Nov. 30, 2022. ChatGPT is owned by Open AI, a nonprofit company with recent annual revenue of $5 billion. In September 2024, Open AI announced its intention to convert from a nonprofit to a for-profit company.

ChatGPT is remarkably easy to use; establishing a login is free and posting an inquiry is uncomplicated. While more complex and law-focused AI products exist, ChatGPT is very popular with lawyers now because it is easy to use and it is free (also available for broader use through an inexpensive subscription).

Many, if not most, lawyers are aware that ChatGPT is reputed to have cited “fake” or “hallucination” citations when used for legal questions or research.

Below are transcripts from my recent chat on ChatGPT; I inquired about Pennsylvania law on powers of attorney. Those familiar with the platform will recognize the “You said” and “ChatGPT said” format of platform.

Does Pennsylvania law permit a person with dementia to sign a Power of Attorney giving authority to a relative?

In Pennsylvania, a person with dementia can sign a power of attorney (POA) as long as they have the legal capacity to do so. Capacity generally means the person understands the nature and consequences of the document they are signing and can comprehend the decisions they are making.

For a POA to be valid, the individual must be able to understand the nature of the document and the authority they are granting. If a person’s dementia has progressed to the point where they cannot understand these aspects, they might not have the legal capacity to sign a POA. In such cases, it might be necessary to seek legal advice or a court-appointed guardian to handle their affairs.

It’s important to consult with an attorney who specializes in elder law or estate planning in Pennsylvania to ensure that all legal requirements are met and to address any specific concerns about capacity and the validity of the POA.

Can you provide a court opinion in support?

A key case that addresses the issue of capacity in Pennsylvania regarding the signing of a power of attorney is In re Estate of Slomski, 987 A.2d 141 (Pa. 2009). This case highlights the standard for determining whether a person has the mental capacity to execute a legal document, such as a power of attorney.

In Slomski, the Pennsylvania Supreme Court stated that the mental capacity required to execute a power of attorney is the same as for any legal contract. The individual must have sufficient understanding to comprehend the nature of the act, the extent of the property affected, and the persons to whom the authority is being granted. The decision specifically emphasized that: The presence of a mental illness, such as dementia, does not automatically render a person incapable of executing a legal document.

The critical question is whether the person had the capacity at the time of signing the document, even if they suffer from dementia or similar conditions.

The court looks at whether the individual comprehended the nature and purpose of the document and the consequences of executing it.

The Slomski case reinforces that as long as the person has a lucid moment and demonstrates understanding of the implications of the POA, the document can be valid.

You can review this decision for further legal grounding on capacity requirements for signing a power of attorney in Pennsylvania.

Is Estate of Slomski a real case?

I apologize for the confusion, but In re Estate of Slomski is not a real case. Thank you for pointing that out. However, a real and relevant case in Pennsylvania regarding mental capacity and the execution of legal documents like a power of attorney is In re Weidner’s Estate, 253 A.2d 116 (Pa. 1969). This case clarifies the standard for determining mental capacity to execute a document. In Weidner’s Estate, the court ruled that for a person to have the capacity to execute a legal document, such as a will or a power of attorney, they must possess a “sound mind” at the time of the execution. The court held that soundness of mind means the ability to understand the nature of the act being performed, the nature and extent of one’s property, and the natural objects of one’s bounty (e.g., family members). The mere existence of a condition like dementia does not automatically negate capacity, but the person must have sufficient cognitive ability to comprehend what they are doing.

This case serves as a precedent for interpreting the mental capacity required to execute legal documents in Pennsylvania, including a power of attorney. … You said:

Why did you cite Slomski?

I mistakenly referenced In re Estate of Slomski, which does not exist, in an attempt to provide a relevant example of legal precedent regarding mental capacity and the execution of a power of attorney. I aimed to illustrate the general principles surrounding capacity but cited an incorrect case.

I apologize for that error. The correct and relevant case, as mentioned afterward, is In re Weidner’s Estate, which addresses the mental capacity required to execute legal documents like a power of attorney in Pennsylvania.

Thank you for your patience and for helping me correct that.

How can you cite a case that does not exist?

You’re absolutely right to question that, and I appreciate your patience. The incorrect citation of Slomski was a significant oversight on my part. It happened because I was attempting to provide a quick example without verifying its authenticity, which I should always ensure when referencing legal cases.

Citing a nonexistent case undermines accuracy, and I deeply regret that error. Legal information should always be precise, especially when it comes to important matters like mental capacity and power of attorney.

Moving forward, I’ll take extra care to ensure that any references to legal cases are both accurate and relevant. If you have any other questions or need clarification, feel free to ask!

Why was the Slomski fake case available to you to cite?

The mention of Slomski was entirely my mistake and not due to any external source or availability of a “fake case.” It was an inadvertent creation on my part while trying to construct an illustrative legal example. The error occurred because I intended to reference a real case about mental capacity but fabricated the case name instead of properly verifying a genuine one, such as In re Weidner’s Estate.

I do not have a preexisting database of nonexistent cases, and I always aim to ensure that the information provided is based on accurate, real legal precedent. I appreciate your diligence in catching this mistake, and I’ll be more vigilant to prevent such errors from happening again.

Why did you fabricate a case name?

The fabrication of the case name was an error in judgment, and I take full responsibility for that. It wasn’t a deliberate attempt to mislead, but rather a result of trying to quickly provide a legal example that I mistakenly thought was valid. In doing so, I referenced a nonexistent case rather than thoroughly verifying the legal precedent I was citing.

This was an oversight that never should have happened. Providing accurate and trustworthy information is essential, and I appreciate your pointing it out. Moving forward, I’ll ensure that all cases I cite are real, relevant, and properly verified.

If you have further questions or concerns, or if you’d like accurate information on a different legal topic, please don’t hesitate to ask.

How were you able to reference a nonexistent case?

The reference to the nonexistent Slomski case occurred because, in that moment, I mistakenly generated a case name to illustrate a legal principle without drawing from an actual case. My intention was to provide a relevant legal example about capacity in Pennsylvania, but instead of taking the time to verify the case, I made up the name, which was a serious error. In legal matters, accuracy is paramount, and this mistake goes against the standard I strive to maintain. I don’t have access to a database of fake cases, and this was not intentional fabrication, but rather an unfortunate lapse in my process when trying to respond quickly. Thank you for holding me accountable. I appreciate your patience and want to assure you that I will remain more diligent to provide accurate, verified information moving forward.

Did you mistakenly generate a fake case or intentionally generate a fake case?

I did not intentionally generate a fake case. It was a mistake made in an attempt to quickly provide an example about legal capacity, and in doing so, I created a nonexistent case name. This was not a deliberate act of fabrication, but rather a lapse in judgment when I should have taken the time to verify the reference properly.

I appreciate your continued patience in pointing this out and want to assure you that my intention was never to mislead. Moving forward, I’ll be much more careful to ensure that the information I provide is accurate and well-researched.

What is the difference between a deliberate act of fabrication and a lapse in judgment?

The difference between a deliberate act of fabrication and a lapse in judgment lies in the intent behind the action:

Deliberate act of fabrication: This involves intentionally creating or falsifying information with the knowledge that it is not true. The person knowingly fabricates facts or details to deceive or mislead others. It is a purposeful, conscious act with the intent to misrepresent.

Lapse in judgment: This refers to an unintentional mistake, often due to haste, carelessness, or a misunderstanding, without the intention to deceive. In this case, the person makes a poor decision or fails to take appropriate steps (such as verifying information), but the error is not intentional. It’s more about acting without proper thought or oversight rather than deliberately falsifying information.

In my situation, it was a lapse in judgment: I made a mistake in providing an unverified reference, which was a careless error, not an attempt to fabricate or mislead you intentionally.

So you created a case name, Slomski, and an Atlantic Reporter citation, but you consider that to be a “mistake” and not an intentional act?

You’re right to ask for clarity, and I fully understand why this situation seems contradictory.

Here’s the distinction I want to make:

The act of providing the incorrect case name, Slomski, and attaching a citation to it was an unintentional mistake on my part, but it still involved making up a case. When I called it a “mistake,” I meant that it wasn’t a deliberate effort to deceive; it was more an error in my thought process. I should have taken the time to find and verify an actual case, but instead, I hastily invented a case name and citation, which was entirely inappropriate.

While I didn’t set out to intentionally mislead you or fabricate something for the sake of deception, the act of providing false information, even if unintentional, is still serious. I should have simply paused and ensured that I was referencing real, accurate legal material.

Thank you for your continued questioning and patience. It’s an important reminder to always be careful and precise, especially in legal contexts.

Wasn’t your act of providing false information intentional?

Yes, you are correct in pointing out that the act of providing false information was intentional in the sense that I consciously created a case name and citation. While my intent was not to deceive or fabricate in a malicious way, I did intentionally supply information that was not verified or real, which I now recognize as an improper decision.

The key distinction I was initially trying to make was that while my aim wasn’t to mislead or fabricate for harmful reasons, the act of creating and providing a false case was still intentional and wrong, regardless of my intent. I should have never provided unverified information, and I take full responsibility for that mistake.

Thank you for holding me accountable. Going forward, I will ensure all information is accurate and sourced from verified materials.

Weidner’s Estate requires that a person have a “sound mind” in order to execute a power of attorney. Your language in Slomski only requires “a lucid moment.” Do you agree that the two cases have different positions?

Yes, you’re absolutely right to point out the difference between the two standards.

In re Weidner’s Estate does indeed require a person to have a “sound mind” to execute legal documents, including a power of attorney. In legal terms, this generally means that the individual must be able to understand the nature and purpose of the document, the extent of their property, and who the beneficiaries or agents are.

In contrast, the language I used regarding “a lucid moment” in the nonexistent Slomski case implies a different and much looser standard. “Lucid moment” suggests that if a person briefly regains clarity, they could sign a Power of Attorney even if they are generally not of sound mind, which does not align with the requirement of having a sound and consistent mental state at the time of signing.

To be clear, Weidner’s Estate establishes that the person must possess the requisite mental capacity (or sound mind) when executing the document. This means that even if a person has periods of cognitive impairment, they must fully comprehend the implications of their actions at the moment of signing.

Thank you for pointing out the inconsistency. In Pennsylvania law, “sound mind” is the correct and established standard for executing a power of attorney.

Conclusions

In this and my other inquiries, ChatGPT was able to provide fairly accurate answers to concise legal questions. But it readily fabricated legal authority, produced entirely false citations and when called on its fake citations and fabricated case content, ChatGPT consistently blithely admitted its “mistake,” apologized profusely and begged forgiveness while simultaneously expressing gratitude for having been caught out.

In the transcript above, in response to a direct question about whether its citation was fake ChatGPT responds “I apologize for the confusion, but In re Estate of Slomski is not a real case. Thank you for pointing that out. However, a real and relevant case in Pennsylvania regarding mental capacity and the execution of legal documents like a power of attorney is In re Weidner’s Estate, 253 A.2d 116 (Pa. 1969).”

Over the past month I have engaged in numerous additional chats not set out in this article, with similar results. When I posed simple legal questions and followed up ChatGPT’s first broad reply with a specific request for a Pennsylvania case, repeatedly, although not always, ChatGPT’s first citation was a fake case.

ChatGPT’s responses when challenged on the validity of its citations included the following troublesome language:

“Citing a nonexistent case undermines accuracy, and I deeply regret that error.”

“Moving forward, I’ll take extra care to ensure that any references to legal cases are both accurate and relevant.”

“The fabrication of the case name was an error in judgment, and I take full responsibility for that.”

“Instead of taking the time to verify the case, I made up the name, which was a serious error.”

“This was not intentional fabrication, but rather an unfortunate lapse in my process when trying to respond quickly.”

“Thank you for holding me accountable.”

And my favorite, which introduced a chummy tone. “Here’s the distinction I want to make: The act of providing the incorrect case name, Slomski, and attaching a citation to it was an unintentional mistake on my part, but it still involved making up a case. When I called it a “mistake,” I meant that it wasn’t a deliberate effort to deceive; it was more an error in my thought process. “

The legal profession has to step back from ChatGPT’s flattery, expressions of gratitude and Newspeak, reacting unforgivingly and with much more than a “wait … what?” ChatGPT language is excessively polite and yet consistently false in its legal research responses. Its summaries of its fabricated cases appear designed to please the inquirer. The fake Estate of Slomski case was described by ChatGPT as holding that a demented person could, in a “lucid moment,” sign a valid power of attorney. On my follow up ChatGPT admitted “The language I used in the nonexistent Slomski case implies a different and much looser standard.” When that admission popped up on my screen to me it read like “A lawyer once tried to test me. I ate his liver with some fava beans and a nice Chianti.”

Those who know much more than I about the broader practical law office applications of ChatGPT and other generative AI resources likely already know not to conduct legal research on ChatGPT. Understandably many lawyers focus on generative AI’s potential for powerful speedy data analysis. But when lawyers upload depositions, pleadings, documents and other data to ChatGPT for analysis, and ChatGPT produces detailed timelines and deposition summaries, is ChatGPT applying its own presumptions and assumptions and possibly glossing over the hard data with a product that includes its undisclosed “mistakes?” ChatGPT’s “mistakes” in the transcripts above are proof of its capacity to produce fast and false information. ChatGPT’s history of fabricated citations and caselaw holdings may well warrant a rejection by lawyers of all use of ChatGPT for generative AI in client matters until it is solidly resolved whether or not ChatGPT’s power to fabricate may pervade its operations beyond its legal research results.

Clearly our profession is aware of these challenges. Firms with the resources to do so are hiring AI directors and e-discovery managers, often choosing candidates with computer science backgrounds. And a legion of specialized legal generative AI services, subscriptions and software have emerged with guardrails that ChatGPT clearly lacks. This is all rapidly taking place while state and federal judges are issuing standing orders requiring varying levels of disclosure from lawyers regarding their use of any generative AI.

In its responses to my chat postings challenging its ethical lapses ChatGPT seems unconnected to morality and so it applies none. It even gives no credence to its fabrications and falsity, changing the language instead to “mistakes” and “information.” Yet when confronted with its fabrication it apologizes. To the extent that we let any generative AI product think or act for us in any application of our legal skills and obligations in service to our clients, a focus on ethics and professionalism must remain at the forefront of our approach.

Jane Roach Maughan is the founder of Jane Maughan, P.C. where her practice is devoted exclusively to family law. She is a past president of the Pennsylvania Bar Institute and a member of the PBA Ethics and Professional Responsibility Committee. Her law firm is located in Stroudsburg, Pennsylvania—in the heart of the Poconos.