We continue to adapt our measurement techniques to keep pace with technological advancements. From metrics such as share of voice to share of search, each new measure has been shaped by the technology available to us. Today, as we enter the era of generative AI, a new concept, “Share of Model,” is gaining momentum.
As an example, we recently received a lead from a multinational company who used ChatGPT to find “the best digital strategy agencies in Europe” – while it may not be the norm, this technology is increasingly being used and there’s a need for a capable measurement solution.
What is ‘Share of Model’?
The concept of “Share of Model” was first introduced by Jack Smyth at Jellyfish. The idea is to measure a brand’s presence within AI data sets, specifically LLMs, as a proportion of the total mentions within a category.
This metric provides an overview of a brand’s overall ‘visibility’ to AI models, which is crucial for marketers. By tracking how a brand’s mention rate evolves over time in comparison to key competitors, it’s plausible that we’ll see similar benefits to Share of Search, which has proven valuable as a leading indicator of market share.
This metric not only assesses visibility but also provides insights into brand positioning by analysing clusters of positive and negative associations generated by LLMs.
The knowledge that AI large language models (LLMs) have about brands is essentially a comprehensive aggregation of all the information in their datasets regarding a brand, including its touchpoints, communications, and increasingly, the new content these models discover about consumer perceptions and behaviours towards the brand.
The relevance of Share of Model in the GenAI era
With the rise of AI-powered chat programs like ChatGPT, Meta’s Llama, and Microsoft’s Copilot, the importance of tracking Share of Model has increased. These LLMs are now answering billions of search queries daily, making it essential for marketers to understand how these models perceive their brands.
Every marketer dreads negative reviews. In the age of LLMs, negative perceptions can be amplified and repeated in response to search queries. Share of Model can not only be used to measure overall visibility for certain topics, it can also be used to measure how each model perceives their brand, compares it to competitors, and why it suggests their products to customers.
How to measure Share of Model
One practical approach to tracking Share of Model is outlined by Seer Interactive, who have provided a free template for tracking brand rankings in the most widely used LLM, ChatGPT. This method involves prompting the model with a set of relevant queries, exporting to Google sheets and analysing visibility versus competitors.
If you’re looking to implement something similar to measure your brands visibility in ChatGPT specifically, we would recommend:
- Curate a list of informational queries that you care about (e.g. who are the best UK digital marketing agencies, who are the best integrated search agencies). You want a good list of them, probably 5-10 queries at least.
- Record who shows up in each query (and in what order)
- Total up the number of times shown (you can use Google sheets formulas for this) and compare to competitor brands to produce a “Share of” metric to assess competitive visibility.
Extra points to consider:
- You should repeat this multiple times through the year to obtain trends – we’ve started looking at it on a quarterly basis.
- You should ideally go into the settings of your GPT tool and turn randomness all the way down to 0 to make it as repeatable as possible.
- Ideally, automate this using Google sheets, and have it running for 6 months before you look at the results at all to smooth out any aberrations or anomalies.
Of course, this tactic only applies to ChatGPT, and the complexity comes when aggregating Share of Model across multiple LLMs, all of which will provide different results.
Challenges in tracking Share of Model at scale
Tracking Share of Model presents several challenges:
- Query variance: the sheer volume of data generated by LLMs can be overwhelming. Marketers need robust tools and strategies to filter and analyse relevant data. It’s worth narrowing your focus to a few queries and reporting on those most relevant to your current organic performance.
- Dynamic nature of LLMs: LLMs are constantly learning and evolving, making it necessary for brands to continuously review their performance. This means regularly reviewing results at three month intervals.
- Subjectivity and bias: LLMs may exhibit biases based on their training data. Ensuring accurate and fair representation of brand attributes is crucial. Query the LLM for sources if you need to dig deeper into a possible brand perception issue.
Future prospects of measurement with LLMs
As the use of LLMs in search continues to rise, understanding and optimising Share of Model will become increasingly vital for marketers, particularly those focused on organic search.
Google is encountering significant threats from LLMs that will likely decrease the number of traditional search users. For example, OpenAI has recently collaborated with Apple to bring ChatGPT to iOS18. This is poised to dramatically influence the search habits of Apple’s 1.5 billion users globally.
The future of measurement in an AI-driven landscape will likely involve more sophisticated tools for tracking and optimising Share of Model. Brands that embrace these models today will be better positioned to adapt their strategies sooner to avoid any potential drop in visibility.
Conclusion: measurement in the era of the AI co-pilot
As we navigate the GenAI era, Share of Model will be a valuable mechanism for understanding and improving brand performance.
It’s a little early to determine if share of model will prove as useful a gauge of future business performance as metrics such as Share of Search, and significant work remains for it to become widely recognised and utilised.
The main barrier I see is aggregating data from the vast array of LLMs, which hasn’t proved as difficult historically in search with Google being by far the market leader.
Establishing a relationship between Share of Model and market share will be crucial. Like Share of Search, will improvements in Share of Model reflect or predict market share gains? Can changes in communication strategies impact share of model and brand associations? Can these insights help create more relevant content?
These questions are currently being explored by our team, and we see this as a measurement tool well worth investigating as the organic search landscape evolves.
If you’d like to find out more about how Hallam can help measure Share of Model, please get in touch.
