AI SEO & Business Automation Podcast

James Dooley: Personalization of large language models, how each and every search whether it's ChatGPT, Perplexity, Claude, Gemini, how it can be giving you different answers. Today I'm joined with Ben who has a lot of information, a lot of evidence about this happening. He owns multiple AI tools, one which is Get AISO. So Ben, let's jump straight into it. When someone does a search within a large language model, within an AI basically like ChatGPT or Gemini, please can you explain how different answers keep coming up for different people?

Benjamin Tanenbaum: Yeah, that's a very good question. So we did some research on it that we're going to publish in a leading SEO publication. We tried to look at the problem with a fresh eye because there has been a lot written on it already. I’ll give a flavour for what the consensus is in the industry at this stage, but we really tried to look at it from scratch. I have a setup where we take a realistic question based on our dataset of real conversations and run that question lots of times. First with a baseline session with no memory, not logged in, then gradually add more and more context and rerun the same question many times to see how the answer changes.

Benjamin Tanenbaum: The concept of personalization is subtle to measure because there are simple facts and there are more complex factors compounded by variability. Unlike Google search, if you ask the same question multiple times in an LLM, you might get different answers. Let’s say James you ask a question to Gemini and I ask the same question. We may get different answers, but it may not be because of personalization. It may just be variability in token selection, like a lottery for the next token. You may get a restaurant recommendation in New York that is fancy on Fifth Avenue. I may get a cheaper one in Brooklyn. It may feel like personalization, but it could just be that both rank equally well in the fan out and one was picked this time and the other the next time. What looks like personalization may just be variability.

Benjamin Tanenbaum: There are straight facts about personalization that are true. The first is that every time you send a query, your location is sent with it. That is true for ChatGPT and mostly for Gemini. Even if you say do not take into account my location, it will still effectively append your location. So if you are in Manchester, Manchester goes with the query. That has a big impact for local businesses. Even if you are not a local business, you should think about where your ideal customer lives because you want localized content.

Benjamin Tanenbaum: The second aspect is more subtle personalization. Labs like OpenAI want to understand who you are. If you search for pizza and previously mentioned you are celiac, it may not even mention restaurants that are unsafe for you. That is a big bet because more customized answers increase satisfaction and eventually support better advertising. The difference with AI search versus Google search is that AI has much richer conversational data. It can know who you are in much more depth. That is why when people ask ChatGPT to draw who it thinks they are, it can be eerily accurate.

Benjamin Tanenbaum: That said, we should calm the debate. Many people say answers are so personalized that we cannot even make content anymore. But 95 percent of people use ChatGPT for free. On free plans, personalization is much less because cheaper models are used and less context is retained. There is some personalization but not that much at this stage. Gemini is starting to use what the rest of Google knows, including past searches, but it is not fully live in all contexts. Most consumer search happens on Gemini and ChatGPT.

James Dooley: I want to touch on tracking. Some people in the SEO community say there is no point tracking AI visibility because it changes every time. There is no point doing LLM optimization. My view is yes it changes, but the more optimization you do, the more raffle tickets you buy. The more chance you have of being cited. Coming back to personalization, Ben, I understand when I am logged in and Gemini knows my history. But when someone is not logged in and has no history, why are they getting different results from one search to the next?

Benjamin Tanenbaum: If you are not logged in and there is absolutely no information about you, the degree of personalization is minimal. It is basically only location. What looks like personalization is usually variability in generation. It is not random, it is distributions. If you run the same query 10,000 times you will see patterns. If your website is clearly described and featured in the query fan out, you may appear in 60 percent of cases. So you want to aim for share of voice rather than a deterministic ranking. It is something you can optimize for.

Benjamin Tanenbaum: What is interesting is that once you are logged in and memory is on, variability is reduced by personalization. If James prefers fancy Japanese restaurants with excellent whiskey and jazz, that preference will shape the query fan out. The generated web searches will heavily weight those features. That narrows the result pool. Instead of one out of 30 possible picks, it may become one out of two. So if you repeat the same question over and over with personalization active, the response becomes more stable. It becomes less of a lottery the more personalized it is.

James Dooley: I like the idea of share of voice. If you are cited 40 percent of the time, aim for 45 percent, then 50 percent. You may never get 100 percent because of volatility, but you can improve. Ben, it has been a pleasure doing this series on AI SEO, LLM optimization, and personalization. If someone wants to follow you and keep up with changes over the next few months, where can they get hold of you?

Benjamin Tanenbaum: The best place is LinkedIn. It is Benjamin Tanenbaum and my LinkedIn slug is B N T Am Bentan. I post a lot, probably too much, but some of it is interesting.

James Dooley: Ben, it has been an absolute pleasure. Hope everyone likes the series on AI SEO and this episode about AI personalization and AI visibility.