Why the same AI gives a different answer every time, and what to measure instead

A founder we work with asked us the obvious question after looking at two runs side by side: these are the same prompts and the same models, so why is every answer different every time? Here is what the two runs held.

Both runs ran on Gemini 2.5 Flash with Google Search grounding, Perplexity Sonar and gpt-5.6-luna, each with search on, each request cold, each asked as a buyer in India. The environment was identical, which is the only reason the two are worth comparing at all. And still the lists of recommended names were different in every one of the nine pairs.

Three things vary, and the screenshots show all three

1. The search is different every time

Every engine we ask searches before it answers, and a search does not return the same pages twice. In run 9, for the branded prompt “what is [the brand]”, the engines read four unrelated sites that happened to share a word with the brand’s name: a cashback site, a company-registration service, a startup-services page. In run 10 they read three different ones, two of them venture-funding directories. Different pages read, different names recommended. Where the engine looks is most of what it says.

2. The model samples

Language models are deliberately non-deterministic at their default settings. Two identical requests produce different wording and, at the margins, different picks. We could pin the temperature to zero and get steadier lists. We do not, because a zero-temperature answer is less like what a real buyer receives, and the thing being measured is what real buyers are told.

3. Some prompts are guesses to begin with

A “what is [the brand]” prompt for a new brand is a question about an entity the engines do not know. So each run invents a category for it from the string: registration services in one, venture funds in another. Perplexity even searched the prompt’s typo verbatim. A prompt about an unknown entity is the least stable prompt there is, and it will stay that way until the entity is defined where the engines read.

The lists are unstable. The standing is fairly stable. The share moved by exactly one answer.Runs 9 and 10, same prompts, same engines

What a measurement has to do about it

If single answers vary this much, a tool that shows you single answers is showing you weather. A few things follow.

• Ask many prompts, not one. A panel in the dozens, across the questions buyers actually ask, gives a share that a few noisy answers cannot swing. Nine cannot hold a stable value, which is exactly what these two runs demonstrate.
• Record the instrument. Every answer we stored carried the engine, the model build, whether it searched, the location, the language and the context state. Two runs were compared only when all of those matched. Runs 9 and 10 matched, so the comparison above is honest. Most of the eight runs before them did not, and we did not pool them.
• Put an interval around the share. Ten runs of one question are not ten independent facts. The statistics we used treated repeated runs as clustered by question, so the interval was wider and truer than a naive count.
• Compare standing, not lists. Recommended, named only, not named. That coarser reading moved on five of nine answers between the runs, and four held. The lists moved on all nine.

The one thing worth keeping from the noise

In run 10, OpenAI recommended the brand on “AI cofounder for Indians”, the first unbranded recommendation in ten runs. Its recorded sources say why: it had read the brand’s own site before answering. In run 9 the same engine read six unrelated blogs and recommended Lore AI, Replit and Cursor. Read you, recommended you. Not read, not recommended. That is not noise. That is the signal the noise was hiding, and it is the subject of the next note.