Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum
Listen on your favourite platform
| Platform | Link |
|---|---|
| YouTube | Listen on YouTube → |
What Does “Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum” Talk About?
This episode of the James Dooley Podcast dives into the mechanics of how large language models like ChatGPT, Gemini, Perplexity, Claude, and Grok can return different answers to the exact same query from different users. James Dooley and Benjamin Tannenbaum explore the critical distinction between true personalisation and probabilistic variability, explaining how what looks like a personalised result might simply be the outcome of token selection randomness, which Benjamin describes as a next token lottery. The conversation draws on original research conducted by Benjamin's team, who built a controlled testing environment using real AI conversation datasets and ran identical questions under varying conditions, starting with no login or memory and gradually adding more context.
The episode breaks down the specific factors that drive genuine personalisation, including location signals that are appended to every query regardless of user preference, memory settings that allow systems like ChatGPT to reference past conversations, and login state. Benjamin also explains how query fan out becomes weighted by personal attributes and how Gemini is beginning to integrate broader Google signals such as past search history. A particularly insightful moment comes when Benjamin points out that personalisation can actually reduce randomness rather than increase it, because a highly specific user preference narrows the candidate pool dramatically, for example from 30 broadly similar options down to just two strong matches for someone seeking Japanese restaurants in New York with jazz and premium whisky.
“It's not random. It's probabilistic. If you test the same query thousands of times, patterns emerge. You can measure share of voice rather than expecting deterministic rankings.”
— Benjamin Tannenbaum
Who Are the Guests on “Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum”?
Benjamin Tannenbaum is an AI search researcher and entrepreneur who owns multiple AI tools, including Get AISO. He focuses on understanding how large language models retrieve and rank information, with a particular emphasis on query fan out, personalisation mechanics, and AI visibility measurement. Benjamin conducts original research using controlled experimental setups built from real AI conversation datasets and regularly shares his findings with the SEO and AI search community on LinkedIn.
James Dooley is the host of the James Dooley Podcast and an established figure in the SEO and digital marketing space. In this episode he serves as the interviewer, guiding the conversation through increasingly technical territory while grounding the discussion in practical implications for marketers and business owners trying to understand and optimise their presence in AI-generated search results.
What Are the Key Takeaways From “Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum”?
Here are the key points discussed in this episode:
- Different answers from large language models often reflect probabilistic token generation variability rather than true personalisation, making it important to distinguish between the two before drawing conclusions.
- Location is the most consistent personalisation signal because it is automatically appended to every query, even when a user explicitly instructs the model to ignore it, which has significant implications for both local and non-local businesses.
- Memory-based personalisation in ChatGPT is a deliberate strategic priority for OpenAI because more personalised answers drive user satisfaction and long-term monetisation, though its real-world impact is currently limited since around 95 percent of users are on the free tier with less memory depth.
- Personalisation can actually stabilise AI responses rather than make them more unpredictable, because weighting the query fan out by specific user attributes narrows the candidate pool and reduces the number of roughly equivalent options the model must choose between.
- AI visibility should be measured using share of voice rather than fixed rankings, because optimisation increases the probability of a brand appearing in responses rather than guaranteeing a deterministic position, and consistent patterns do emerge when the same query is tested thousands of times.
“If your brand appears in 60 percent of responses, that's a measurable advantage. Optimisation increases your probability of being selected, even if you won't appear 100 percent of the time.”
— Benjamin Tannenbaum
Is “Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum” Worth Listening To?
This episode is especially valuable for anyone trying to make sense of why their brand or content appears inconsistently across AI search results. Benjamin Tannenbaum brings genuine research rigour to a topic that is frequently discussed with more speculation than evidence, and his explanation of the difference between token generation variability and true personalisation gives listeners a much more accurate mental model for understanding what is actually happening inside these systems. The concrete example of the pepperoni pizza restaurant in New York makes an abstract concept immediately tangible.
The discussion of share of voice as the right measurement framework for AI visibility is practically useful for marketers and SEO professionals who have been frustrated by the lack of stable rankings to track. Rather than dismissing AI visibility measurement as impossible, Benjamin reframes optimisation as a probability game, which is both intellectually honest and actionable. The insight that personalisation can narrow candidate pools and actually reduce randomness is counterintuitive and thought-provoking, and it opens up a clear strategic opportunity for brands that can strongly own a distinctive niche attribute.
Who Should Listen to “Personalizing LLMs: How Your ChatGPT & Gemini Differ | James Dooley x Benjamin Tannenbaum”?
This episode is ideal for:
- SEO professionals and digital marketers who want to understand how to measure and improve brand visibility within AI-generated search results
- Business owners and brand strategists trying to understand why their company appears inconsistently across different AI platforms and different users
- AI and technology enthusiasts interested in how large language models handle personalisation, memory, and probabilistic response generation
- Content creators and agencies advising clients on AI search optimisation who need a research-backed framework for explaining variability and setting realistic expectations
Where Can You Listen to James Dooley Podcast?
You can listen to James Dooley Podcast on all major podcast platforms:
- Apple Podcasts – Search for “James Dooley Podcast” in the Podcasts app
- Spotify – Available on Spotify for free
- Amazon Music / Audible – Listen through your Amazon account
- Overcast – For iOS users who prefer a dedicated podcast app
- Pocket Casts – Cross-platform podcast player
You can also subscribe using the RSS feed: https://feeds.transistor.fm/james-dooley-podcast
What Are Listeners Saying About This Episode?
“The distinction Benjamin draws between token generation variability and actual personalisation is something I've never heard explained this clearly before. It completely changed how I interpret the inconsistent results I see when testing ChatGPT for client projects. Essential listening for anyone doing AI SEO work.”
“The share of voice framework is exactly the mental shift I needed. I've been frustrated trying to track fixed AI rankings and this episode reframed the whole problem in a way that actually makes it tractable. The example about Japanese restaurants with jazz and whisky narrowing the candidate pool to just two options really stuck with me.”
“Short, dense, and genuinely informative. I appreciated that Benjamin based the conversation on real research rather than speculation. The point about 95 percent of ChatGPT users being on the free tier with limited memory depth is the kind of reality check that is often missing from AI hype discussions.”
James Dooley: Personalisation of large language models. How each and every search, whether it’s ChatGPT, Perplexity, Claude, Grok or Gemini, can give different answers. Today I’m joined by Benjamin Tannenbaum, who has a lot of information and evidence on this. He owns multiple AI tools, including Get AISO. So Benjamin, let’s jump straight in. When someone does a search in a large language model like ChatGPT or Gemini, why do different answers appear for different people?
Benjamin Tannenbaum: That’s a very good question. We conducted research on this that we’re preparing to publish in a leading SEO publication. We approached the problem from scratch because there’s already a lot written about it. We built a setup where we took realistic questions from real AI conversation datasets and ran them repeatedly. First, we used a clean baseline with no login and no memory. Then we gradually added more context and reran the same question many times to observe how the answers changed. One complexity is that even without personalisation, large language models can give different answers to the same question. Unlike Google Search, where results are relatively stable, LLMs generate responses probabilistically. If you and I ask the same question, we might receive different brand recommendations. That doesn’t automatically mean it’s personalisation. It could simply be variability in token generation, what some describe as a next token lottery. For example, if we both ask about the best pepperoni pizza restaurant in New York, you might get a fancy Fifth Avenue restaurant and I might get a cheaper Brooklyn option. That difference might look like personalisation, but it could just be variation within equally strong candidates in the query fan out results. So what appears to be personalisation may simply be variability in how the model selects from the top candidate sources.
James Dooley: So what are the real personalisation factors?
Benjamin Tannenbaum: The first straightforward one is location. Every time you send a query, your location is appended to it. This happens even if you explicitly ask the model not to consider your location. It still gets added. That’s significant for local businesses. But even non local businesses should consider where their ideal customers are located, because localisation affects AI responses. The second layer is memory based personalisation. If memory is enabled, which is the default in ChatGPT when logged in, the system references past conversations. If it knows you have a dietary restriction, it may automatically exclude restaurants that don’t meet that requirement. This is a strategic focus for companies like OpenAI. More personalised answers increase satisfaction and, in the long term, improve monetisation potential. But we should add a reality check. Around 95 percent of users are on the free tier of ChatGPT. Those models have less context window and less memory depth. So while personalisation exists, it is often limited in practice. Gemini is beginning to integrate broader Google signals, including past search history, but this is still evolving.
James Dooley: Some people in the SEO space argue there’s no point tracking AI visibility because responses change constantly. What’s your view?
Benjamin Tannenbaum: It’s not random. It’s probabilistic. If you test the same query thousands of times, patterns emerge. You can measure share of voice rather than expecting deterministic rankings. If your brand appears in 60 percent of responses, that’s a measurable advantage. Optimisation increases your probability of being selected, even if you won’t appear 100 percent of the time. Interestingly, personalisation can actually reduce variability. When a model heavily weights certain attributes in the fan out based on user preferences, the candidate pool narrows. For example, if your preference is highly specific, like Japanese restaurants in New York with jazz and premium whisky, the fan out becomes very narrow. Instead of 30 broadly similar candidates, you might only have two strong matches. That reduces randomness and stabilises results.
James Dooley: That’s interesting. So personalisation can make responses more consistent rather than less.
Benjamin Tannenbaum: Exactly. It narrows the search space. So while it introduces complexity for marketers, it also creates opportunities. If your brand strongly owns a distinctive attribute, you can dominate those personalised niches.
James Dooley: I like the idea of measuring share of voice. If you appear 40 percent of the time, aim for 50 percent. It becomes a probability game rather than a fixed ranking. Benjamin Tannenbaum, it’s been a pleasure doing this series. We’ve covered how queries differ in AI search, query fan out, and now personalisation. If people want to follow your work, where can they find you?
Benjamin Tannenbaum: The best place is LinkedIn. My name is Benjamin Tannenbaum. I post regularly about AI search, query fan out, and related research.
James Dooley: Benjamin Tannenbaum, thank you again. That wraps up our discussion on AI personalisation and visibility within large language models.
Creators & Guests
Host
James Dooley is a UK entrepreneur.