How many prompts before an AI visibility number means anything?
Published 23 August 2026 · 8 min read
Almost every AI visibility tool reports a percentage. Very few report how many times they asked. Those two facts together are the reason most of these numbers cannot support the decisions being made on them.
This post is arithmetic, not opinion. Every figure below is computed with a 95% Wilson score interval and a standard two-proportion power calculation, and you can reproduce all of them.
What a 40% visibility rate is actually worth
Suppose a tool tells you that you appear in 40% of relevant answers. Here is the same 40%, measured at different sample sizes, with the 95% interval it earns.
| Runs | Result | 95% interval | Interval width |
|---|---|---|---|
| 10 | 4/10 = 40% | 17% to 69% | 52 points |
| 25 | 10/25 = 40% | 23% to 59% | 36 points |
| 50 | 20/50 = 40% | 28% to 54% | 26 points |
| 100 | 40/100 = 40% | 31% to 50% | 19 points |
| 200 | 80/200 = 40% | 33% to 47% | 13 points |
| 500 | 200/500 = 40% | 36% to 44% | 9 points |
| 1000 | 400/1000 = 40% | 37% to 43% | 6 points |
A widely repeated piece of advice in this category is that a 40% visibility rate across 200 prompt runs is meaningful data. By the arithmetic above, 200 runs buys you a band from 33% to 47%. That is real information. It is not enough to tell a 40% quarter from a 45% quarter, and it will be reported to you as though it were.
How many runs it takes to detect a change
Stating a rate is the easy half. The reason anyone buys this software is to know whether something moved. That is a much more expensive question.
At a 40% base rate, with 80% power and a 95% significance threshold, here is what each size of change costs to detect:
| Change you want to detect | Runs needed per side |
|---|---|
| 5 points | 1,534 |
| 10 points | 388 |
| 15 points | 173 |
| 20 points | 97 |
| 30 points | 42 |
Read that against a tool sampling 25 to 300 prompts per month in total, across every question it tracks. Split across even ten question clusters, that is a few dozen runs each. Such a tool can tell you that a thirty-point collapse happened. It cannot tell you that your content programme produced a ten-point gain, and it will not say so.
The small-sample trap
Small samples do not merely produce vague numbers. They produce confident, specific, wrong ones, because a percentage hides its own denominator.
| Observed | Reads as | Actual 95% interval |
|---|---|---|
| 1 of 2 | 50% | 9% to 91% |
| 2 of 5 | 40% | 12% to 77% |
| 3 of 10 | 30% | 11% to 60% |
| 30 of 100 | 30% | 22% to 40% |
Rows one and three both round to a tidy figure a slide will happily carry. Neither distinguishes a serious problem from a rounding artefact. This is why we suppress rates entirely below a floor of five runs per question cluster per window and print "insufficient data" instead. A blank is annoying. A number that reads like a measurement and is not one is worse.
Three rules that follow from the arithmetic
- Never publish a rate without its sample size and interval. The same percentage from 10 runs and 1,000 runs are different findings, and only the denominator distinguishes them.
- Never average across intent families. Being named when asked about you by name and being recommended when asked for a vendor are separate questions with separate base rates. Averaging them raises n while destroying the meaning.
- Correct for multiple comparisons. Testing forty question clusters at p < 0.05 produces roughly two false alarms per round by construction. A Benjamini-Hochberg correction across everything tested in the round is the cheapest fix.
What this costs
Sampling properly is not expensive, which is the frustrating part. At list prices reviewed on 21 August 2026, a grounded answer of roughly 2,000 input and 700 output tokens with one search call costs about $0.0095 on Gemini 2.5 Pro, $0.0195 on GPT-5.1, $0.0215 on Sonar Pro and $0.0375 on Claude Opus 4.5. Blended, about $0.022 a run.
So 388 runs per side, the number that buys you a defensible ten-point detection, costs roughly $8.50 of provider spend. The reason most tools sample 25 prompts is not the cost of the tokens.
Questions people ask about this
How many prompts do you need to measure AI visibility?
It depends entirely on what you intend to conclude. To state a rate with a margin of roughly ten points, you need about 100 runs per question cluster. To detect a ten-point change between two periods at a 40% base rate and 80% power, you need roughly 388 runs per side. Sampling 25 to 50 prompts supports the observation that something occurred, and does not support a percentage.
Why does a 40% AI visibility score need a confidence interval?
Because 40% from 10 runs and 40% from 1,000 runs are different findings presented identically. At 10 runs the 95% Wilson interval is 17% to 69%. At 1,000 runs it is 37% to 43%. Without the interval and the sample size, a reader cannot tell which one they are looking at.
What is a Wilson score interval and why use it for AI answer measurement?
A Wilson score interval is a confidence interval for a proportion that stays correct at small samples and at the boundaries. The commonly used normal approximation breaks down exactly where AI answer measurement operates: small n, and rates near 0% or 100%. At zero hits the normal approximation reports an interval of zero width, which is plainly false.
Is a single wrong AI answer a finding?
No. One run out of two reads as 50% and carries a 95% interval of 9% to 91%. That is compatible with almost any true rate. A single observation is a reason to look, not a result to report.