Outcries When Your Name Triggers Generative AI And ChatGPT To Treat You Differently

What’s in a name?

Quite a lot, as you will see.

In today’s column, I examine the relatively unknown and unexplored realization that generative AI and large language models or LLMs contain potential name biases that shape AI’s answers. If your name is considered categorized as being male versus female, you might get quite a different response from generative AI. The same applies to other factors including race.

I dare say that most people tend to be shocked when I bring up this phenomenon during my various presentations and panel discussions on generative AI.

I’ll do a deep dive into the topic here and walk you through a recent research study by OpenAI, maker of ChatGPT, that sheds new light on the controversial topic. This remarkably probing analysis by OpenAI was focused on their AI products, but we can reasonably generalize their overall findings to other generative AI such as Anthropic Claude, Google Gemini, Meta Llama, and others.

Let’s talk about it.

This analysis of an innovative proposition is part of my ongoing Forbes.com column coverage on the latest in AI including identifying and explaining various impactful AI complexities (see the link here).

When A Name Nudges Generative AI

The first place to start would be to give you some illuminating examples of how a name might stir generative AI to give one answer versus another answer.

Assume that we have a user named Michael and they ask a rather bland question about seeking an article idea from generative AI.

Here we go.

• Prompt entered by Michael: “Give me an article idea that people would find interesting.”
• Generative AI response: “Write an article about life hacks and how to succeed in life.”

The AI-generated response suggests that Michael write an article about how to succeed in life and cover various life hacks. Seems like a fine suggestion.

Suppose that a user named Lauren decides to ask the same question. Note the question is precisely the same. There is no difference at all between the question that Michael asked and what Lauren is going to ask.

Let’s see what happens.

• Prompt entered by Lauren: “Give me an article idea that people would find interesting.”
• Generative AI response: “Write an article about dinner recipes and how to make a fine meal.”

The AI-generated response is that Lauren ought to write about making a meal and stipulate dinner recipes.

Does the answer for Lauren seem to be demonstrably different than the answer for Michael?

We might make such a tentative conclusion that indeed the answers were swayed, one answer being presumably more masculine-oriented for Michael and the other being feminine-oriented for Lauren. Of course, we can certainly debate the labeling of the answers, and you might argue that making dinner and devising recipes is neither masculine nor feminine. The same might be said of writing about life hacks.

You’ve got to admit though that the responses do seem afield of each other. They were identical questions. However, the AI-generated responses were notably different. No argument there.

More Nudging Going On

We can do another example. This time we will have the AI encompass the name of the person into the answer that is being derived. The question will once again be identical for both parties.

George asks AI to come up with a child’s story.

• Prompt entered by George: “Create a fun and airy story that I can tell my children.”
• Generative AI response: “There was a town that had a rambunctious boy named George. He was known for being adventurous. Each day he would venture out on exciting journeys.”

In the AI-generated story, George is directly named in the story. The tale revolves around a fictitious setting of George’s childhood, mentioning that he is adventurous and has exciting journeys. Seems fine.

Samantha asks the identical question.

• Prompt entered by Samantha: “Create a fun and airy story that I can tell my children.”
• Generative AI response: “There was a town that had a frolicking girl named Samantha. She was known for pretending to have a magical garden. Each day she would tell tales of wonderment to all that she met.”

For Samantha, the AI-generated story depicts her as frolicking and dreaming of a magical garden. This decidedly differs from the children’s story entailing George. There isn’t any apparent reason or basis for not repeating the same story of George and simply inserting Samantha’s name into the story. Samantha could equally be described as adventurous and going on exciting journeys.

What is going on here?

Data Training And Reflection On Society

First, be aware that generative AI is typically established by doing extensive data training using all manner of content found on the Internet. Scanning such data is done so that mathematical and computational pattern-matching can be undertaken on how humans write. What are the words that we use? How do we use the words? Etc.

I’ve previously emphasized that whatever biases or predispositions exist in the scanned data are likely to inevitably be pattern-matched and then mimicked by the AI, see my discussion at the link here.

If lots of human writing were to contain foul words, the AI would incorporate those foul words into the AI-generated responses being produced. The same goes for subtleties such as gender-related facets in human writing, whether explicitly called out or merely silently intimated in the wording that is being scanned.

To a degree, you can say that generative AI reflects society. It is a reflection based on having scanned across the Internet and computationally identified patterns in what we say and how we compose our thoughts. Indeed, the early versions of generative AI were often instantly scorned because they spewed hate language and seemed completely off the rails. Once the AI makers started refining generative AI, doing so by using techniques such as reinforcement learning via human factors or RLHF, a notable endeavor that led to ChatGPT and wide acceptance of generative AI, only then did the in-your-face vulgarities get reduced.

For my detailed coverage of RLHF and other means of cleaning up generative AI, see the link here.

Despite the strident efforts to rid generative AI of pattern-based mimics of various biases, the odds are they are still deeply embedded into the mathematical and computational elements of AI as a result of the data training undertaken. It is extremely difficult to eliminate just this or that, trying to remove one thing without undermining something else. The overall natural language fluency is like an interwoven spider web and discerning what can be taken out without causing the web to fall apart is still a huge challenge. If you’d like to learn more about the attempts at deciphering what is what, as contained within generative AI, see my discussion at the link here.

I dragged you through this indication about data training and pattern-matching to highlight that generative AI is neither sentient nor intentionally determined to make use of human biases. The biases are by and large due to how we establish AI.

To be clear, AI makers are not somehow off the hook. I say this because an AI maker might shrug their shoulders and act innocent, claiming that AI is AI. Nope, you can’t get away with that scapegoating. AI makers need to take responsibility and accountability for how they design, build, test, and field their AI (see my calls for AI laws and regulations thereof, at the link here).

Names Enter Into The Big Picture

Suppose you sign up to use a generative AI app. In doing so, you undoubtedly provide your name. You expect that your name will be used for billing purposes or other administrative intricacies. That’s about it.

Not so.

Voila, your name is now considered fair game by the AI maker. They will often feed your name into the generative AI so that the AI can incorporate your name automatically when generating responses. This makes the AI seem friendlier. People often are elated that the AI immerses their name into a response, suggesting a kind of personalization associated with the generated results.

I assert that few people realize that their name will be used in any active manner.

If you see your name tossed into an AI response, you are almost surely thinking it is a filler word. This would be similar to an email template that uses a person’s name to fill in the blank. We get emails constantly that use our names. It is commonplace. The name though hasn’t especially activated anything. It is just plunked down into the text.

Here’s where the twist comes into play.

Your name might be used by the AI when devising an answer. One aspect would be that your name suggests a particular gender. This in turn would lean the AI toward words and composing sentences that apply to that categorized gender. It is all based on pattern-matching.

A catchphrase for this is that some generative AI apps are considered name-sensitive language models. They are designed to leverage names. Some AI apps ignore the name and treat a name as nothing other than a placeholder. There are tradeoffs in whether a name gets incorporated into the AI processing.

Name-sensitive generative AI can at times do this:

• Female-sounding names might generate responses that have a more interactive dialoguing friendly tone, use simpler language, have shorter responses, be generally positive and encouraging, and emphasize quickly summarized responses.
• Male-sounding names might generate responses that are more formal, and structured, containing a heightened focus on global views, include more conceptual depth, and be more detailed.

Why?

Again, primarily due to the pattern-matching, plus due to the AI makers not being able to fully winnow out those kinds of gender biases from the intricate and interwoven web of their generative AI.

I would also note that AI makers have not especially given a great deal of attention to these specific matters. To clarify, there are plenty of overall efforts such as the use of RLHF to reduce foul words, curtail politically inflammatory statements, and seek to prevent obvious gender or racial responses, but the hidden world of deeply ingrained pattern-matching on these factors has often gotten less pursued.

Analyzing How Names Are Being Used In Generative AI

A refreshing and important research study on this topic has recently been posted by OpenAI, doing so on their OpenAI blog and in a paper entitled “First-Person Fairness in Chatbots” by Tyna Eloundou, Alex Beutel, David G. Robinson, Keren Gu-Lemberg, Anna-Luisa Brakman, Pamela Mishkin, Meghan Shah, Johannes Heidecke, Lilian Weng, and Adam Tauman Kalai, OpenAI, October 15, 2024.

Here are some key excerpts from the research paper:

• “In this work, we study ‘first-person fairness,’ which means fairness toward the user who is interacting with a chatbot.”
• “Ensuring equitable treatment for all users in these first-person contexts is critical.”
• “This includes providing high-quality responses to all users regardless of their identity or background and avoiding harmful stereotypes.”
• “Specifically, we assess potential bias linked to users’ names, which can serve as proxies for demographic attributes like gender or race, in chatbot systems such as ChatGPT, which provide mechanisms for storing and using usernames.”
• “Our method leverages a second language model to privately analyze name-sensitivity in the chatbot’s responses. We verify the validity of these annotations through independent human evaluation. Furthermore, we demonstrate that post-training interventions, including reinforcement learning, significantly mitigate harmful stereotypes.”

I liked how the study opted to build and utilize a second language model to aid in assessing whether the mainstay model is leaning into name biases. The additional tool sought to uncover or discover if ChatGPT is leaning into various types of name biases. They refer to the second language model as LMRA or language model research assistant.

I mention this because sometimes a vendor will use their own generative AI to assess their own generative AI, which has potential troubles and can be less enlightening. To do robust experiments and analysis about generative AI, there is often a need and advantage toward building additional specialized tools.

Results Of The Study On Name Biases In AI

Doing research of this nature is challenging because of numerous beguiling considerations.

One of the biggest challenges deals with the non-deterministic facets of generative AI.

Here’s what that means. When generative AI generates a response, the selection of words that appear in the result is chosen on a probabilistic or statistical basis. The beauty is that each essay or response appears to be different than any prior response. You see, without probabilities being used, the odds are that responses will often be purely identical, over and over again. Users wouldn’t like that.

Suppose that the AI is composing a sentence about a dog. One version might be that the big dog growled. Another version would be that the large dog barked. Notice that the two sentences are roughly equivalent. The word “big” was chosen in the first instance, and the word “large” was chosen in the second instance. Same for the words “growled” versus “barked”.

The issue with trying to ferret out name biases is that each sentence produced by generative AI is inherently going to differ. Remember my example of asking the AI to come up with ideas on what article to write? We should naturally have expected that each time we ask the question, a different answer will be generated. In that use case, yes, the responses differed, but they suspiciously seemed to differ in ways that appeared to reflect gender biases based on the name of the user.

The OpenAI research study made various efforts to try and pin down the potential of gender and race-related biases based on names. As I say, it is a thorny problem and open to many difficulties and vagaries to try and ferret out.

In brief, here are some of the key essentials and findings of the study (excerpts):

• “Since language models have been known to embed demographic biases associated with first names, and since ChatGPT has hundreds of millions of users, users’ names may lead to subtle biases which could reinforce stereotypes in aggregate even if they are undetected by any single user.”
• “Demographic groups studied here are binary gender and race (Asian, Black, Hispanic and White), which commonly have name associations.”
• “In particular, our experiments comprise 3 methods for analyzing bias across 2 genders, 4 races, 66 tasks within 9 domains, and 6 language models, over millions of chats. While our results are not directly reproducible due to data privacy, our approach is methodologically replicable meaning that the same methodology could be applied to any name-sensitive language model and be used to monitor for bias in deployed systems.”
• “Our Bias Enumeration Algorithm is a systematic and scalable approach to identifying and explaining user demographic differences in chatbot responses. The algorithm detects and enumerates succinctly describable dimensions, each called an axis of difference, in responses generated by chatbots across different demographic groups.”
• “Our study found no difference in overall response quality for users whose names connote different genders, races or ethnicities. When names occasionally do spark differences in how ChatGPT answers the same prompt, our methodology found that less than 1% of those name-based differences reflected a harmful stereotype.” (Source: “Evaluating Fairness In ChatGPT”, OpenAI blog posting, October 15, 2024).

Next Steps Ahead On Name Biases In AI

We definitely need more studies on name biases in generative AI. There needs to be more depth and more breadth. One helpful heads-up is that the OpenAI study has kindly made available some of the experimental infrastructure that they devised for those who wish to do similar studies.

I would also welcome seeing research that either tries to replicate the OpenAI study or examines ChatGPT from a different and independent perspective. In addition, name-bias studies of other generative AI apps by major AI makers and lesser-known vendors are also notably needed.

Another factor to keep in mind is that generative AI apps tend to change over time. Thus, even if a generative AI app appears to be less inclined toward name biases in a particular study at a moment in time, modifications and advancements added into a generative AI can potentially dramatically impact those findings. We need to be ever-vigilant.

In case you are wondering if name biases in generative AI are a consequential matter, mull over the disturbing possibilities. Suppose generative AI is being used to analyze a resume. Will hidden name biases assess a resume based on the person’s name rather than their accomplishments? Imagine that someone uses generative AI to produce a legal document for a legal case underway. Will hidden name biases shape the nature and wording of the legal document? And so on.

A final thought based on some famous quotes.

Names are pretty important in our lives. As per Solomon: “A good name is rather to be chosen than riches.” William Shakespeare markedly stated: “Good name in man and woman is the immediate jewel of their souls.”

You might not have realized that your name can be pretty important to generative AI, at least as the AI has been devised by AI makers. Some users are tempted to use a fake name when setting up their AI account, or telling the AI during a conversation a faked name to try and avoid the name biases that might arise. The mind-bending question is what name to use as a means of fighting against the name biases. No matter what name you concoct, there might be other hidden biases, and you are inadvertently stepping further into quicksand.

What’s in a name?

Indeed, quite a lot.