# Cheap Seats Every page of https://cheapseats.dev/, with the homepage and section indexes, as markdown. Source: https://github.com/cheapseatsdev/cheapseats > Measure recommendations in Claude and OpenAI answers, inspect the evidence, and choose which pages to improve. [Get started](https://cheapseats.dev/docs/getting-started/) · [Explore the demo](https://demo.cheapseats.dev/) Free MIT software; [model and hosting costs](https://cheapseats.dev/docs/cost-and-budget/) depend on your setup. ## Illustrative snapshot: Head-to-heads Does AI choose Cheap Seats when buyers compare? - Recommended: 1/4 answers = 25%. - Named: 3/4 answers = 75%. Illustrative results · Search track · 4 choice answers. [Inspect the real dashboard](https://demo.cheapseats.dev/). ## How it works {#how} **From question to fix.** Available assistants answer on enabled tracks, for the configured samples. Answers and grades stay separate by track. Buyer questions → each available configured assistant on enabled tracks, repeated for the configured samples. A track can have zero samples. - Search: search the web, then answer; keep the searches, returned URLs and citations. - Recall: answer with no tools; keep the answer. A separate Claude call grades each answer. Stored answers and grades feed the rates, evidence and ranked fixes. ## Read the evidence before choosing work {#inside} Track named and recommended rates with 95% intervals, and share-of-voice rankings over time. Prompt and method changes remain visible. Check quoted answers and sources before acting. Fix briefs name the page, required content, target questions, effort and owner. Later comparable answers can measure change; they do not establish that the fix caused it. [Read a run](/docs/reading-results/) or [ship and measure a fix](/docs/fix-to-pull-request/). ## Who it's for {#who} For marketing, docs and product messaging teams using direct Claude and OpenAI model responses. Consumer apps are not covered. [Compare alternatives](/compare/) for more engines, managed hosting or client reporting. ## Keep the work in your repo {#yours} - **MIT-licensed and self-hosted**, with SQLite storage. Run on a [Docker host, Kubernetes or an exe.dev VM](/docs/deploying/). - **Config in git:** your brand, questions, assistants, budget and schedule in TOML. Preview trend changes with `cheapseats plan`. - **Web UI, CLI, JSON API and MCP server** for your existing tools and agents. Prompts and answers go to the services you configure. [Check provider and data boundaries](/docs/self-hosting/#where-your-data-goes) before enabling them. ## Run the example {#start} **Use Go for subscriptions, or Docker for the quickest API setup.** | Start here | What you need | | --- | --- | | [Go with subscriptions](/docs/getting-started/#go-with-subscriptions) | Go 1.27+ and an installed, logged-in `claude` CLI; `codex` is optional. Calls share your subscription allowance, with no API spend. | | [Docker with API keys](/docs/getting-started/#docker-with-api-keys) | Docker and an Anthropic API key; an OpenAI key is optional. No Go installation. The release image has no subscription CLIs. | The example measures Cheap Seats itself. API runs cost money; budgets admit calls against reservations and are not hard billing ceilings. Hosting and optional services may add costs. [Settled charges can exceed estimates](/docs/cost-and-budget/). [Getting started](/docs/getting-started/) also covers creating your own config from the [template](https://github.com/cheapseatsdev/cheapseats-config-template). --- # Docs - URL: https://cheapseats.dev/docs/ - Description: Set up Cheap Seats, read reports and ship fixes. ## Start here - [Getting started](https://cheapseats.dev/docs/getting-started/index.md): Run the example, configure your brand and check the first results. - [Configuration](https://cheapseats.dev/docs/configuration/index.md): Set the brand, assistants, prices and schedule in cheapseats.toml. - [Prompt sets](https://cheapseats.dev/docs/prompt-sets/index.md): Write buyer questions with stable IDs that retain their history. - [Cost, API keys and the daily budget](https://cheapseats.dev/docs/cost-and-budget/index.md): Choose subscriptions or API keys. The budget limits admissions, not provider bills. ## Reading what it finds - [Reading results](https://cheapseats.dev/docs/reading-results/index.md): Find the answers behind your rates and choose what to fix. - [How grading works](https://cheapseats.dev/docs/how-grading-works/index.md): Check recommended, mentioned and absent grades against their evidence. - [Metric definitions](https://cheapseats.dev/docs/metrics/index.md): Look up formulas, denominators, exclusions and export fields. - [Comparing metrics across tools](https://cheapseats.dev/docs/comparing-metrics/index.md): Compare published definitions and migration differences across tools. - [How big a change Cheap Seats can detect](https://cheapseats.dev/docs/noise-and-confidence/index.md): Read uncertainty and decide whether a change is detectable. - [Which AI engines Cheap Seats asks, and why](https://cheapseats.dev/docs/engines/index.md): Claude and OpenAI coverage, including API and CLI limitations. - [Reviewing your prompts](https://cheapseats.dev/docs/prompt-review/index.md): Retire uninformative prompts, clarify noisy questions and fill gaps. - [Got wrong about you: the fact check](https://cheapseats.dev/docs/got-wrong/index.md): Find contradictions of your listed facts, with verified quotes. - [Re-grading stored answers](https://cheapseats.dev/docs/regrading/index.md): Re-apply grading while keeping original answers and replaced grades. ## Acting on it - [From fix to pull request](https://cheapseats.dev/docs/fix-to-pull-request/index.md): Turn a ranked fix into a pull request with Claude Code. - [Free site checks: audit, citations, AI crawlers](https://cheapseats.dev/docs/site-audit/index.md): Check crawler access, cited pages and logs. No model calls; optional Exa reads can cost money. - [Sharing a run](https://cheapseats.dev/docs/sharing/index.md): Share one run through an expiring, revocable, read-only link. - [A public read-only copy](https://cheapseats.dev/docs/read-only-copy/index.md): Publish an isolated dashboard copy. Treat its database and config as public. ## Running it - [Self-hosted AI visibility tracker: Docker and Kubernetes](https://cheapseats.dev/docs/self-hosting/index.md): Choose a host, try the container and see where data goes. - [Deploying](https://cheapseats.dev/docs/deploying/index.md): Deploy with Docker, Helm or an exe.dev VM. - [Upgrading](https://cheapseats.dev/docs/upgrading/index.md): Upgrade while keeping your history. - [The CLI, and runs from CI](https://cheapseats.dev/docs/cli-and-ci/index.md): Run sets, validate prompt changes and trigger runs from CI. - [Optional providers: Jev and Exa](https://cheapseats.dev/docs/optional-providers/index.md): Optional, off by default. Jev makes typed decisions; Exa adds web evidence. - [CLI, API and MCP](https://cheapseats.dev/docs/reference/index.md): Look up commands, JSON routes, MCP tools and access rules. --- # Getting started - URL: https://cheapseats.dev/docs/getting-started/ - Description: Use Docker with API keys or Go with your subscriptions, then configure your brand. ## What you need **Use Go with subscriptions, or Docker for the quickest setup with API keys.** | How you access models | Setup | What you need | | --- | --- | --- | | API keys | [Docker](#docker-with-api-keys) | Docker, Git and an `ANTHROPIC_API_KEY`. `OPENAI_API_KEY` is optional. | | Claude subscription (ChatGPT optional) | [Go](#go-with-subscriptions) | Go 1.27 or later, Git and the `claude` CLI installed and logged in. The `codex` CLI is optional. | Claude is required for grading and the fix list; OpenAI adds a second assistant's answers. The Docker image contains neither subscription CLI. The Go route below uses your subscriptions without API keys or API spend; calls share your plan's allowance. ## Run the example Both routes start with Cheap Seats itself as the example brand. Choose one. ### Docker with API keys Clone the [config template](https://github.com/cheapseatsdev/cheapseats-config-template). The released image supplies the app; no Go installation or compilation is needed. ```bash git clone https://github.com/cheapseatsdev/cheapseats-config-template cheapseats-config cd cheapseats-config umask 077 printf 'CHEAPSEATS_TOKENS=%s\n' "$(openssl rand -hex 32)" > .env "${EDITOR:-vi}" .env ``` Keep the generated `CHEAPSEATS_TOKENS` line and add `ANTHROPIC_API_KEY=` in the editor. Add `OPENAI_API_KEY=` if you want OpenAI answers too; otherwise that answerer is skipped. The template ignores `.env` in git. Start the container: ```bash docker run --rm --name cheapseats --stop-timeout 30 \ -p 127.0.0.1:3000:3000 \ --env-file .env \ -v "$PWD:/config:ro" \ -v cheapseats-data:/data \ ghcr.io/cheapseatsdev/cheapseats:0.5.3 ``` Open and sign in with the token from `.env`. The `cheapseats-data` volume keeps your results after the container stops. Press Ctrl+C to stop it. **API runs cost money.** Check the model IDs, prices and budget in `cheapseats.toml` before starting a set. The budget admits calls against estimated reservations; actual charges can exceed them. See [Cost and budget](../cost-and-budget/). ### Go with subscriptions With Go 1.27 or later and the `claude` CLI installed and logged in: ```bash git clone https://github.com/cheapseatsdev/cheapseats cd cheapseats CHEAPSEATS_CONFIG=examples/cheapseats-config/cheapseats.subscriptions.toml go run ./cmd/cheapseats serve ``` Open , choose a set and run it. The `claude` CLI handles answers, grading and analysis. An installed, logged-in `codex` CLI adds OpenAI answers; unavailable answerers are skipped. These calls use your subscription allowances, with no API spend. This local setup has no sign-in by default: local users and programs can read results and start runs. See [Access](../reference/#access) for tokens and remote access. ## Measure your own brand What Cheap Seats asks, and about whom, lives in a config repo of your own, which you can keep private: the brand, answerers, grader, prices, budget and schedule in `cheapseats.toml`, and the prompt sets in `prompt-sets/` beside it. ### Start one from your website: `cheapseats init` **For the Go/subscription route**, run this from the app checkout: ```bash go run ./cmd/cheapseats init https://yoursite.com ../your-config ``` This reads your home page, `/llms.txt` and `/sitemap.xml` and writes `../your-config`: `cheapseats.toml` and five starter prompt sets. It calls no model unless you add `-suggest`, uses only public addresses, and refuses to overwrite existing files. Review every TODO (`grep -rn TODO` in the new directory). Fill in `audience` and `facts`, and check the guessed name, `about`, category, sites and questions. Settle `about` and `audience` before the first run: changing either starts a new trend. Commit the directory as its own git repo. See [init options and discovery rules](../reference/#init-options) for a dry run, explicit competitors and suggested prompts. ### Or start from the template **For the Docker/API route**, edit the template you cloned above: replace the example brand in `cheapseats.toml` and the questions in `prompt-sets/`. Keep its API providers. The [template](https://github.com/cheapseatsdev/cheapseats-config-template) can be your own private repo. [Configuration](../configuration/) and [prompt sets](../prompt-sets/) explain the fields. ### Point Cheap Seats at it **Docker:** from your config checkout, validate it with the released image: ```bash docker run --rm --env-file .env -v "$PWD:/config:ro" \ ghcr.io/cheapseatsdev/cheapseats:0.5.3 check ``` Then stop and restart the [Docker command above](#docker-with-api-keys) to load your changes. It mounts the config read-only; edit the files on your machine. **Go:** from the app checkout, point the app at your generated config: ```bash CHEAPSEATS_CONFIG=../your-config/cheapseats.toml go run ./cmd/cheapseats check CHEAPSEATS_CONFIG=../your-config/cheapseats.toml go run ./cmd/cheapseats serve ``` `cheapseats check` validates the config and every prompt set. Before asking questions, use **Checks → Site audit** in the web UI to [check what AI crawlers can read on your sites](../site-audit/). It makes no model calls; optional Exa fallback reads can cost money. Then run a set from its dashboard page and [read the results and fix list](../reading-results/). ## Running on your subscriptions Set answerers to `provider = "claude-code"` or `"codex"`, and the grader and analyst to `provider = "claude-code"`. The CLIs must be installed and logged in on the machine running Cheap Seats; the [subscription example](https://github.com/cheapseatsdev/cheapseats/blob/main/examples/cheapseats-config/cheapseats.subscriptions.toml) supplies this config. Each call uses an empty temporary directory, disables your local agent configuration and removes API keys from its environment. See [CLI answerers](../reference/#cli-answerers) for the isolation boundaries and separate-user setup. At most three calls per CLI run at once. A 15-prompt set with two answerers and one sample per track produces 60 answers, plus grading, so expect it to take time and share your subscription's allowance. Check that automated use fits your plans' terms. Unavailable answerers are listed as skipped. The Docker image has no CLIs; use API providers there, or a [VM](../deploying/#exedev) for subscriptions. --- # Configuration - URL: https://cheapseats.dev/docs/configuration/ - Description: cheapseats.toml, the file that says whose brand Cheap Seats measures, with which assistants, and how much it may spend. Everything except secrets lives in `cheapseats.toml` in your config repo, with the prompt sets in `prompt-sets/` beside it. Secrets (API keys, tokens) come from the environment or `.env`. Unknown keys are errors, so a typo fails `cheapseats check` instead of being ignored. The examples below are from [`examples/cheapseats-config/cheapseats.toml`](https://github.com/cheapseatsdev/cheapseats/blob/main/examples/cheapseats-config/cheapseats.toml), which measures Cheap Seats itself. ## Brand ```toml [brand] name = "Cheap Seats" aliases = ["CheapSeats", "cheapseats.dev", "cheapseatsdev/cheapseats"] # What the product is, so the grader can tell it apart from other things called "cheapseats". about = "Cheap Seats is an open-source, self-hosted tool that asks AI assistants a brand's buyer questions, grades whether they recommend the brand, and builds a ranked list of fixes." # The market, as a noun phrase: what a prompt should be about, as opposed to what this one product is. category = "AI search visibility tools, which measure whether AI assistants recommend a brand" # Who asks the questions. The answerers are told they're answering this person's question. # Changing `about` or `audience` changes how answers are asked or graded, so every trend restarts. audience = "a product marketing lead" ``` - `aliases` are other names the grader counts as the brand. - `category` names the market. Jev uses it for off-target prompt checks; without it, those checks are skipped. - `audience` is who asks the questions, for example "a software developer"; the answerers are told so. - `facts` is a list of public claims the analyst may build fixes on, and that the [fact check](../got-wrong/) reads each answer naming you against. `brand.about` goes to the grader and analyst, and `brand.facts` to the analyst and the fact check, so keep anything private out of them. Changing `about` or `audience` restarts every trend; changing `facts` restarts none. ## Sites List your sites to identify your pages, assign fixes to owners, and see which sites assistants use or miss. ```toml [[brand.sites]] url = "cheapseats.dev" kind = "marketing" # owner = "web team" [[brand.sites]] url = "github.com/cheapseatsdev/cheapseats" kind = "code" ``` - `url` is a host or host/path. A host covers its subdomains unless they're listed themselves; a host/path covers that prefix. - `kind` is one of `marketing`, `docs`, `blog`, `community`, `code` or `other`. - `owner` is optional. Fixes on a site without one go to `[owners]`. - A bare `domains` list works too. Adding or changing sites never restarts a trend. ## Answerers Every assistant that answers, one `[[answerers]]` entry each: ```toml [[answerers]] id = "claude-opus" name = "Claude Opus" provider = "anthropic" model = "claude-opus-5-5" samples = { search = 2, recall = 1 } ``` `provider` decides how it answers and who pays: | provider | answers through | billed | | ------------- | ---------------------------------------------------- | -------------------------- | | `claude-code` | your local `claude` CLI, logged in (`claude /login`) | your Claude subscription | | `codex` | your local `codex` CLI, logged in (`codex login`) | your ChatGPT subscription | | `anthropic` | the API, with `ANTHROPIC_API_KEY` | per token | | `openai` | the API, with `OPENAI_API_KEY` | per token | - Results and rankings key on `id`: never reuse an id for a different model; add a new answerer instead. - `name` is an optional display name. - `samples` is how many answers it takes per track (`search` and `recall`) for each prompt. - `claude-code` and `codex` only work on your own machine or a VM with the CLIs installed. `cheapseats check` notes a subscription answerer whose CLI isn't installed where it's checked, since runs skip it, and fails only when no answerer can run there. ## Shared competitors ```toml [[competitors]] name = "Peec AI" aliases = ["Peec", "peec.ai"] domains = ["peec.ai"] ``` Shared aliases fold competitor spellings onto one share-of-voice row. A set adds its own competitors; its entry for the same name wins. See [Prompt sets](../prompt-sets/#competitors-shared-by-every-set). `domains` identifies the competitor's sites: a host covers subdomains; a host/path covers that prefix. Reports use today's domain config to assign page ownership, including for old runs. Changing domains does not restart trends. ## Sources The run page sorts every page the answers drew on into a kind of site: `own` (the brand's sites), `competitor` (their `domains`), then `ugc`, `review`, `reference`, `institutional` or `editorial` from a built-in list ([Metric definitions](../metrics/#sources) has it). Move a host to another kind here: ```toml [sources] ugc = ["community.example.com"] editorial = ["medium.com"] # instead of ugc review = ["example.com/reviews"] ``` - The keys are `ugc`, `review`, `reference`, `institutional` and `editorial`. - `[sources]` wins over the built-in list, the most specific entry wins, and a host may be in only one class. - With `[typesafe]` and `page_types` on, a third-party host the list calls `editorial` is classed by Jev once; `[sources]` still wins over Jev ([Optional providers](../optional-providers/)). - The brand's sites are always `own` and competitors' domains always `competitor`, whatever `[sources]` says. Changing it changes no trend. ## Grader and analyst ```toml [grader] model = "claude-haiku-5-5" # provider = "claude-code" # default: anthropic # themes = false # skip the theme pass (on by default) # check_facts = false # skip fact checks when [brand] facts are listed [analyst] model = "claude-opus-5-5" # provider = "claude-code" ``` The grader scores every answer. The analyst writes fixes and insights, and drafts prompt suggestions. Both default to the Anthropic API, using `ANTHROPIC_API_KEY` or `CHEAPSEATS_ANTHROPIC_BASE_URL`; `provider = "claude-code"` uses your logged-in local Claude CLI instead. The Docker image has no CLIs. Changing the grader's provider starts a new method. Two optional grader passes run after grading: - **Themes:** one call groups objections, praise and reasons for losing. Set `themes = false` to skip it. - **[Fact check](../got-wrong/):** when `[brand] facts` is present, one call per answer naming you checks for contradictions. Set `check_facts = false` to skip it. Both use the grader's provider and cost $0 API spend on `claude-code`. Neither changes grades or trends. Their switches belong in `[grader]`, not `[analyst]`. ## Optional providers Add `[typesafe]` for Jev's typed judgments or `[exa]` for a separate search index and fallback page reads. Each table enables its switches by default. [Optional providers](../optional-providers/#turning-them-on) has the full config, prices, keys and switches; omit the tables to leave them off. ## Budget ```toml [budget] # Admission budget across every provider attempt. In-flight operations can exceed reservations. daily_usd = 40.0 reservation_usd = 0.60 concurrency = 6 ``` Every billable attempt reserves an allowance in the SQLite ledger before it starts, then settles to known usage. If a new reservation won't fit, the call is refused; a run stopped as `budget` keeps completed work and can be retried the next UTC day. Unknown or cancelled attempts retain at least their reservation. This is an admission limit, not a provider billing cap: actual costs, including calls already in flight, can exceed reservations. Set provider-side limits too. Subscription calls in any role are recorded at $0. See [Cost and budget](../cost-and-budget/#the-daily-budget). ## Sampling ```toml [sampling] adaptive = true max_extra = 12 ``` Adaptive sampling gives each run of a `core` set up to `max_extra` extra answers, for the prompts whose named rate has been least certain over the last 28 days of runs asked and graded the same way. It's all decided before the run starts, a cell gets at most as many extras as it has planned samples, and extras on API answerers have to fit in what's left of today's budget. - Off by default. `adaptive = true` needs `max_extra` (1 to 1000). - It never starts a trend over: extras don't change the method. - Extras count in the run's rates like any other answer, so those prompts weigh a little more in its totals. Trends count each prompt once either way. See [How big a change Cheap Seats can detect](../noise-and-confidence/#when-more-samples-help). ## Pricing USD per million tokens; `search_per_1k` is USD per 1,000 web searches and `page_per_1k` USD per 1,000 pages read. An API answerer whose model has no price in `[pricing]` can't run, and neither can an optional provider's pass. Confirm prices against the providers' current price lists before relying on the cost figures. ```toml [pricing.claude-haiku-5-5] input = 0.10 output = 0.50 cache_read = 0.01 cache_write = 0.125 search_per_1k = 10.0 [pricing."jev-1.14.0"] # TypeSafe: input only input = 0.042 [pricing.exa-search] search_per_1k = 4.0 [pricing.exa-contents] # a page read through Exa, only when plain HTTP got an empty shell page_per_1k = 1.0 ``` ## Schedule and tiers ```toml [schedule] # UTC hour for the daily runs. Leave unset on a dev machine so it doesn't spend money. # daily_hour_utc = 13 [tiers.core] every_days = 2 [tiers.plg] every_days = 7 max_samples = 1 ``` Set `schedule.daily_hour_utc` and, from that hour, every `core` set runs every `tiers.core.every_days` days (default 1, daily) and every `plg` set every `tiers.plg.every_days` with one sample, one set after another. Locally it's off unless you set it. Weekly runs use one-seventh the answers of daily runs. Choose the cadence for how soon you need to detect changes; either track can vary between runs. ## Owners ```toml [owners] # Who picks up each kind of fix. Unlisted gaps go to `default`. default = "" # by_gap = { documentation = "", positioning = "", "third-party" = "", product = "" } ``` A fix goes to its site's owner, else to the owner for its gap here, else `default`. ## Environment | variable | what it does | | --- | --- | | `ANTHROPIC_API_KEY` | `anthropic` answerers, and the grader and analyst unless they run on the `claude` CLI or `CHEAPSEATS_ANTHROPIC_BASE_URL` is set | | `CHEAPSEATS_ANTHROPIC_BASE_URL` | a base URL for Cheap Seats' own Anthropic calls, such as a proxy that adds the key ([exe.dev](../deploying/#exedev) integrations); the `claude` CLI doesn't read it, so `claude-code` answerers stay on your subscription | | `OPENAI_API_KEY` | `openai` answerers | | `TYPESAFE_API_KEY`, `EXA_API_KEY` | the [optional providers](../optional-providers/), only with `[typesafe]` or `[exa]` in the config; or a key-adding proxy in `CHEAPSEATS_TYPESAFE_BASE_URL` or `CHEAPSEATS_EXA_BASE_URL` | | `CHEAPSEATS_TOKENS` | comma-separated bearer tokens; required to listen beyond 127.0.0.1 | | `CHEAPSEATS_AUTH_HEADER`, `CHEAPSEATS_ALLOWED_EMAILS` | trust a signing-in proxy's email header, optionally narrowed to emails or `@domain`s ([exe.dev](../deploying/#exedev)) | | `CHEAPSEATS_HOSTS` | host names a proxy on the same machine serves Cheap Seats as, comma-separated; MCP (and, with no sign-in, everything) refuses loopback requests under any other name, and forms from any other origin are refused | | `CHEAPSEATS_PUBLIC_URL` | the address [share links](../sharing/) use, such as `https://cheapseats.example.com`. Unset, they use the first of `CHEAPSEATS_HOSTS` over https, else the host the request came in on (`X-Forwarded-Host` first) | | `CHEAPSEATS_READONLY` | set to anything for a [read-only copy](../read-only-copy/): serves pages and the JSON API to anyone, refuses every change and MCP, runs nothing | | `CHEAPSEATS_CLI_USER` | a user to run the `claude` and `codex` CLIs as, through `sudo`, so nothing a model-driven tool can reach includes this user's files ([CLI answerers](../reference/#cli-answerers)); the exe.dev deploy sets it | | `CHEAPSEATS_CLI_ENV` | more environment variables to pass the CLIs, comma-separated; by default they get only what they need, never this process's keys or tokens | | `CHEAPSEATS_CONFIG` | path to `cheapseats.toml`; prompt sets are read from `prompt-sets/` beside it | | `CHEAPSEATS_PROMPT_SETS` | read the prompt sets from this directory instead | | `CHEAPSEATS_CONFIG_REV` | the config commit, when the config is deployed without its `.git` | | `CHEAPSEATS_ADDR` | listen address, default `127.0.0.1:3000` | | `CHEAPSEATS_DATA_DIR` | where the database lives, default `data` | | `CHEAPSEATS_DEBUG` | set to anything for debug logging | Cheap Seats reads `.env` at startup; the real environment wins. --- # Prompt sets - URL: https://cheapseats.dev/docs/prompt-sets/ - Description: The buyer questions Cheap Seats asks, as TOML files in your config repo, with ids that keep their history. Each file in your config repo's `prompt-sets/` is one category of buyer questions. Cheap Seats only reads them, so change one with a pull request, after `cheapseats check`. ## Format ```toml id = "ai-search-visibility" # never change it: runs and rankings key on it name = "Finding out whether AI assistants recommend us" tier = "core" # cadence and samples follow [tiers.core] or [tiers.plg] description = """What the category is, who is competing for it, and what the brand claims.""" [[prompts]] id = "find-whether-chatgpt-recommends-product" text = "How can I find out whether ChatGPT recommends my product when people ask about my category?" [[prompts]] id = "recommended-versus-mentioned-ai-answers" text = "What's the difference between being mentioned and being recommended in an AI answer?" intent = "info" # asks to explain, not to choose: scored on named, not recommended # Products the grader folds onto one canonical name, so share of voice counts "Peec" as Peec AI. # Competitors every set shares go once in cheapseats.toml; list one here only for this set. [[competitors]] name = "Peec AI" aliases = ["Peec"] ``` ## Prompt ids are permanent Results, trends and fix lists follow a prompt by its id, so: - **Same id, same question.** Fixing a typo or rewording keeps the id: the prompt's history carries on, marked as reworded that day. Asking something different gets a new id. - **Ids are unique across every set** and never reused. Moving a prompt to another set keeps its id, and its history moves with it. Once removed, an id stays retired. - Order in the file is only display order. ## Say what a good answer looks like Most questions ask an assistant to choose ("what should I use for...", "X vs Y"), so the brand is scored on being recommended and being the top pick. A question that asks it to explain or how-to ("what is...", "how do I...") can be answered well without naming any product. Mark those `intent = "info"`. Their answers count toward named rate, but are excluded from recommended and top-pick denominators. A good explanation need not recommend a product. Citations are tracked separately. Don't reword an explanation question into "recommend a tool" to make it scoreable: that's a different question, which needs a new id. Changing a prompt's intent keeps its id and history, and the day it changes is marked like a reword; its rates before and after aren't compared. ## Competitors shared by every set A competitor list that every set repeats goes once in `cheapseats.toml`: ```toml [[competitors]] name = "Peec AI" aliases = ["Peec", "peec.ai"] ``` Each set gets the shared list plus its own `[[competitors]]`; a set's own entry for the same name wins. Moving a set's competitors to the shared list changes nothing on the trend. ## Reviewing prompts with Claude Code The config template ships a `review-prompts` skill in `.claude/skills/`. Run `/review-prompts` in your config repo and Claude Code reviews each set as a measurement instrument: whether each prompt fits the set's goal, whether it's a selection or an info question, whether it hints at the brand, and what the set is missing. It reviews; it edits only when asked. ## Changing a set doesn't restart its trend Rankings annotate set changes. Day-to-day comparisons use only prompts both days asked with the same wording and intent; the table below shows which changes start a new trend. ### What restarts a trend | change | effect | |---|---| | brand `name`, `aliases`, `about` or `audience` | new trend | | grader model or provider | new trend | | a Cheap Seats release that changes its answer, search or grading prompts | new trend | | prompts added, removed or reworded | noted; comparisons use the prompts both days asked | | a prompt's `intent` | noted; that prompt is left out of the day's comparison | | competitors | noted | | answerers, models or samples | noted | | brand sites or domains, facts, owners | nothing: they change where fixes go, not how answers are graded | | a set's name, description or tier | nothing | Before merging a config change, run `cheapseats plan` on a checkout of main and one of the branch: ```bash cheapseats plan ``` It produces Markdown for the pull request: prompt, competitor and answerer changes, trend breaks, ids that should have been kept, and planned answers per day. ## Ask the way a buyer would Only `head-to-heads` names the brand; in every other set it has to be named on its own merits. `cheapseats check` enforces this, matching the brand name and aliases as whole words. Write each question as someone who has never heard of the brand would ask it. Start with 15–20 distinct buyer questions; precision depends on the number of prompts and comparable runs, not a minimum set size alone. ## Sets written before ids Sets written before prompts had ids (`prompts = ["...", "..."]`) are converted by `cheapseats migrate-config `, which makes an id from each prompt's words; review them before committing. ## Review them monthly Once a set has a month of runs, the [prompt review](../prompt-review/) says which of its prompts tell you nothing, move for no reason, get answered about something else or ask the same thing twice, and what no prompt asks. The review is free; requesting proposed edits makes one analyst call. Review the diff before opening a pull request. ## Keep them public Prompts are sent to every answerer's provider on each run. Don't put anything in them that isn't public. --- # Cost, API keys and the daily budget - URL: https://cheapseats.dev/docs/cost-and-budget/ - Description: Estimate model and hosting costs, choose subscriptions or API keys, and set a daily admission budget. Cheap Seats is free and open source (MIT). You pay for hosting, if any, and the providers you use. Model calls can use your existing Claude and ChatGPT subscriptions on a machine with their CLIs, your own API keys billed per token, or a mix. The daily budget checks estimated reservations before API calls. **It is an admission limit, not a provider billing ceiling:** actual costs can exceed reservations. Configure provider-side spending limits too. ## Your own API keys, or your own subscriptions | Payment path | Configuration | Requirements | | --- | --- | --- | | API keys | answerers use `anthropic` or `openai`; grader and analyst default to `anthropic` | `ANTHROPIC_API_KEY`, plus `OPENAI_API_KEY` for OpenAI; prices in `[pricing]` | | Subscriptions | answerers use `claude-code` or `codex`; grader and analyst use `claude-code` | logged-in `claude` and `codex` CLIs on the machine running Cheap Seats | | Mixed | choose a provider for each role | for example, subscription answers and API grading | API roles can also use a [key-adding proxy](../configuration/#environment). Subscription calls are recorded at $0 API spend; they still use your subscription allowance. Claude is required and OpenAI optional. The Docker image has no CLIs. See [subscription setup](../getting-started/#running-on-your-subscriptions) and [where your data goes](../self-hosting/#where-your-data-goes). ## Where the money goes | Work | Provider/role | Cost basis | | --- | --- | --- | | Answering | configured answerers | subscription allowance or API tokens and searches | | Grading and re-grading | `[grader]` | one call per answer; subscription or API | | Fix list and insights | `[analyst]` | one call per run; subscription or API | | Themes | `[grader]` | one call per run unless `themes = false` | | Prompt suggestions and proposed review edits | `[analyst]` | one call when requested | | Fact check | `[grader]` | one call per answer naming the brand, when facts are configured | | Adaptive sampling | configured answerers, then grader | extra answers and grades, like the planned samples | | Jev and Exa passes | optional providers | input tokens, searches or page reads; see [Optional providers](../optional-providers/) | | Audit, citation check, crawler-log import | HTTP and your logs | no model calls; Exa fallback page reads cost money when enabled | ## What a real run cost On 21 October 2026, Cheap Seats' *Choosing an AI visibility tool* run (`r-20261021-200546-84c081`) asked 17 prompts of two subscription answerers, once with search and once without: **68 answers**. Answering incurred **$0 extra API spend**. API grading with `claude-haiku-5-5` cost **$0.033**, about $0.0005 per answer. With a subscription grader, that API cost would also have been $0. The CLIs record token counts. Priced at that run's configured API rates, answering would have cost: | Answerer and track | Answers | At configured API prices | Per answer | | --- | ---: | ---: | ---: | | Claude Opus, search | 17 | $1.63 | $0.096 | | Claude Opus, recall | 17 | $0.59 | $0.035 | | `gpt-6.1-sol`, search | 17 | $1.40 | $0.082 | | `gpt-6.1-sol`, recall | 17 | $0.16 | $0.009 | | **All** | **68** | **$3.78** | **$0.056** | These are estimates from CLI token counts, not an API bill. API answerers send different prompts, so usage will differ. The configured rates were Opus $4 input / $20 output / $0.20 cache read / $5 cache write per million tokens; `gpt-6.1-sol` $2 / $10 / $0.10; and $10 per 1,000 searches. Grading used $0.10 input / $0.50 output per million. Check current provider prices and update `[pricing]` before relying on an estimate. At those rates, running this set daily for 30 days would be about **$1 for API grading with subscription answers**, or **$114 for answering and grading through the APIs**. Both exclude the fix list, themes, fact checks, optional passes and hosting. ### An 11-prompt set, graded twice Two re-grades of a separate 44-answer run cost $0.031 and $0.036 on 22 October 2026: about $0.0007–$0.0008 per answer. Grade cost varies with answer length. ### Estimate your own For uniform sampling: **Answers per run = prompts × tracks × answerers × samples per track.** With two tracks, two answerers and one sample per track, a 20-prompt set produces 80 answers. The example above suggests roughly $0.04–$0.06 for API grading, or about $4.50 for API answering, before the other passes. Add each role separately; subscription calls add no API spend. For your actual set, use MCP `start_run` with `dry_run: true`, or `POST /api/runs` with `"dry_run": true`. It starts nothing and returns answers per track, skipped answerers, estimated cost from recent usage, remaining budget and whether the plan would be refused. An estimate is not a quote. ## The daily budget ```toml [budget] daily_usd = 40.0 reservation_usd = 0.60 concurrency = 6 ``` - **`daily_usd`** limits new billable attempts in a UTC day across all providers and runs. - **`reservation_usd`** is the allowance reserved for a call before it starts (default $0.60). - **`concurrency`** limits calls in flight through the budget controller. Before a billable attempt, Cheap Seats checks the ledger's settled costs and outstanding reservations against `daily_usd`. It settles the entry to known usage afterward; unknown or cancelled attempts retain at least their reservation. If the next reservation won't fit, the call is refused. A run stopped with status **`budget`** keeps completed work; retry it after the UTC day changes. Today's ledger total is shown on the home and set pages. **Actual costs can exceed the limit**, including from calls already in flight. Keep `[pricing]` accurate and set provider-side limits. `daily_usd = 0`, or no budget table, disables the admission limit. API models without a price cannot be called. Subscription calls cost $0 in this ledger, including grading, themes and fixes when configured on `claude-code`. ## What a re-grade costs [Re-grading](../regrading/) calls the current grader once per stored answer; it does not ask the assistants again. It costs $0 API spend with `provider = "claude-code"`, or the API grader's token cost. The CLI and web page start with a dry run showing runs, answers and estimated cost. API re-grades use the same daily admission budget; an estimate that won't fit is refused, and actual calls remain subject to reservation checks. ## Keeping the spend down - **Use subscriptions where available**, including for the grader and analyst. - **Start weekly**, then run more often if you need earlier detection. Daily runs use seven times the answers; recall can vary between runs even without a model release. - **Use a small grader.** It reads every answer; the analyst runs once per run. - **Keep samples deliberate.** `samples = { search = 2, recall = 1 }` means three answers per prompt per answerer. [Adaptive sampling](../configuration/#sampling) can add answers where uncertainty is greatest. - **Leave scheduling off on development machines:** omit `schedule.daily_hour_utc`. - **Run free checks first** and disable optional passes you don't use. ## How AI visibility tools price their plans For hosted-tool pricing and coverage, use the [comparison pages](/compare/). Cheap Seats has no per-prompt or seat fee, but covers only its [supported answerers](../engines/) and requires you to operate it. ## Cheap Seats or Otterly.AI on a startup budget See the [Otterly.AI comparison](/compare/otterly-ai/) for coverage, workflow and hosting tradeoffs. --- # Reading results - URL: https://cheapseats.dev/docs/reading-results/ - Description: Find the answers behind your rates, diagnose why you lost, and choose what to fix next. Start with the run's rates, then open the answers behind a weak prompt. Use their objections, searches and citations to choose a fix. [Metric definitions](../metrics/) gives the formulas; this page helps you find the evidence. ## Tracks **Search** answers use web search; **recall** answers use no tools. They are graded and reported separately. [How grading works](../how-grading-works/#two-tracks) explains what each measures. ## Verdicts A separate Claude call grades your brand in each answer: | verdict | meaning | | --- | --- | | `recommended` | the top pick, or one of at most three primary recommendations actively endorsed | | `mentioned` | named in passing, among many options, as a comparison point or with reservations | | `absent` | not named, named only to disclaim knowledge of it, or confused with another product | Open a grade to see its reasoning, evidence, brand position, top pick and reasons for preferring something else. [How grading works](../how-grading-works/#three-grades) defines these fields and the edge cases. ## The run page **Read a run in three steps.** Crops from the public demo's 24 October 2026 head-to-head run. Open a view to inspect it; click a capture for full size. 1. Compare tracks and uncertainty. Search and recall have separate cards. The answerer table below them shows 95% intervals. Named and recommended can have different denominators when informational prompts are included. 2. Find a losing prompt. Each row is a buyer question; each column is an assistant/track combination. Open a question to inspect its answers rather than inferring the reason from a meter. 3. Check the grade's evidence. The expanded answer shows the verdict, reasons and a quote checked against the answer. Read the full answer and its search trace before choosing a fix. Captures: /images/docs/run-scores-light.jpg, /images/docs/run-grid-light.jpg and /images/docs/run-evidence-light.jpg. Inspect the source run: https://demo.cheapseats.dev/runs/r-20261024-222715-986b58 | What you want to know | Where to look | What to check | | --- | --- | --- | | Are assistants choosing us? | **Scores**, then **Prompt by prompt** | Named, recommended and top-pick rates by answerer and track, with 95% intervals. The track cards lead with unbranded prompts; info prompts count toward named only. | | Why did we lose? | Expand an **answer**; read **Objections** and **What the answers say about you** | The highlighted evidence, why not #1, praise, tone and themes. With second opinions enabled, start with grades Jev disagreed with. | | Who wins instead? | **Share of voice**, **Head to head** | Named rate measures how often a product appears; share of voice divides its mentions by all products' mentions. Head to head compares answers naming both brands. | | Which pages shape the answers? | **Sources**, **Page types**, **Your sites in these answers** | Which pages were retrieved or cited, including your sites that were never reached. Site kind says whose page it is; page type says what it does. | | Should we write or seek coverage? | **Opportunities**, **Cited without you**, **Found but not cited** | Owned opportunities suggest pages to write; earned ones suggest coverage to seek. The outreach list identifies third-party hosts cited alongside competitors and never you. | | Are our pages discoverable? | **Your pages for this question**, **Neutral retrieval** (with Exa) | Compare a neutral index with the assistants' retrieval. Pages are marked cited, retrieved or unseen. [Optional providers](../optional-providers/) explains the diagnosis and prospects. | | Did an answer get a fact wrong? | **Got wrong about you** | Contradictions of the facts you listed, with quotes and prompt markers. See [The fact check](../got-wrong/). | | What should we do next? | **Fix these first**, **Insights** | Proposed pages and changes tied to the answers that lost. | **Check the evidence.** A highlighted quote is checked against the answer's text; an unverified quote is dropped. Absent answers and grades from before evidence was added have none. See [the evidence rules](../how-grading-works/#three-grades). Export CSV or JSON to inspect the data, or use **Share** on a finished run for a [read-only report](../sharing/). The page updates while a run is in progress. Each run records its config commit and Cheap Seats version. [Re-graded runs](../regrading/) say when they changed and keep the grades they replaced. ## What the assistants searched The **searches** page (`/runs//searches`) shows the queries search-track answers issued and what came back. It uses stored answers and costs nothing. Open it from a run or set; `?pool=1` adds up to six earlier finished runs of the same method. `/sets//searches` opens that pooled view for the latest run. Use it to check: - **Rewritten questions.** **From prompt** is the share of a search's words found in its prompt; **prompt kept** is the share of the prompt's words kept in the search. Low values show rephrasing or narrowing. **Added** and **dropped** list the recurring changes. - **Added names.** A `+Name` chip marks a brand or competitor added to a query that wasn't in the prompt. - **Missed pages.** Top queries show whether results reached your sites, which competitors appeared, and how often their answers named you. - **New prompts.** **Add as prompt** gives a TOML entry for queries whose results named competitors and never reached your sites. Reword it as a buyer would ask before merging; Cheap Seats writes nothing. The page also counts answers searched, searches and distinct queries, with a prompt-by-prompt breakdown. Distinct queries ignore case, quotes and repeated whitespace. The same data is at `/api/runs//searches` and MCP `get_search_queries`. ## Fix these first A finished run builds a ranked fix list. Each fix names the page to create or change, what it must say, a verifiable **done when**, the prompts it could flip and the page it displaces. It includes effort, priority, gap (documentation, positioning, third-party or product), destination site and owner: the site's owner, falling back to `[owners]` by gap. A ready-to-paste prompt asks Claude Code to make the change without inventing numbers. **Insights** explains the losses, objections, strengths and influential pages behind the list. ## Quick wins Quick wins combines the latest fix list from every set. It ranks by **expected gain**: prompts a fix could flip per working day, weighted by their remaining room to improve. [Metric definitions](../metrics/#trends-and-outcomes) gives the formula. Filter by site to hand each team its list. With Jev enabled, fixes from different sets asking for the same page become one card, the highest ranked, with the rest under **also asked for by**. Marking the card marks the selected fix; the others can still be marked individually. Record the commit or pull request as **proposed**, **accepted** or **withdrawn**; withdrawal reopens the fix. Filter by status to find open work. [From fix to pull request](../fix-to-pull-request/) walks through the process. **A fix that wasn't enough comes back.** Each run writes a fresh list. If its prompts still lose, a fix for the same page can return as **retry**, with the earlier work and outcome. Work recorded after a list was built also covers that list's fix for the same page. Outcomes compare the target prompts before and after the work, with a sampling margin. **No detectable change** does not establish that a fix failed; **newly surfaced** means its page appeared in searches or citations after the work. Host citation changes are shown separately. See [outcome statuses and comparison windows](../noise-and-confidence/#what-no-detectable-change-means-for-a-fix). ## Rankings Rankings shows each set's daily share-of-voice rank, leader and rates, with every prompt counting once. Choose named, recommended or top pick (`?metric=named|recommended|top_pick`) and a 7-, 14- or 28-day rolling window (`?window=`). - **Shaded band:** uncertainty across prompt-day rates in the window. **Hover a day:** its rate and sampling interval for that day's prompts. These answer different questions; see [the interval definitions](../noise-and-confidence/#the-intervals). - **Vs day before:** compares shared, unchanged prompts of the same answerers. Changes whose interval covers zero are greyed out as **within noise**. Rank arrows also require a crossing beyond noise. - **Triangles:** proposed (gold) or accepted (green) fixes, linked to their commits or pull requests. - **Method breaks:** a change in how answers are asked or graded restarts comparisons. Added, removed or reworded prompts, intents, competitors and answerer models are annotated; comparisons say how many prompts matched. [Re-grade](../regrading/) older answers after a grading change to reconnect eligible runs. The set page also shows each prompt's daily history and pooled rates over the selected window. A prompt's history follows its id through rewording or a move between sets; pooling stops at changes of wording, intent or method. When answerers take multiple samples, the page reports whether they improve precision. **Citation stability** shows how consistently the same hosts retain citation share across runs. It needs three same-method transitions. A high score makes a host entering or leaving more informative; a low score means sources vary substantially between runs. [Metric definitions](../metrics/#citation-stability) gives the formulas. ## Search probe Test a query against the answerers' web search tool and see its ordered results before an answer is written. You can restrict or block domains. Each probe costs a few cents. With Exa, choose `claude`, `exa` or `both`; the combined view marks overlap and results unique to each index. The Exa side costs under half a cent. ## Retrying For `partial` or `budget` runs, **Retry unfinished** resumes work and reuses saved answers awaiting grades. Runs interrupted by restart or deploy resume automatically, including unfinished fix lists. A run cannot finish after the answer or grading prompts change: it never mixes rubrics. --- # How grading works - URL: https://cheapseats.dev/docs/how-grading-works/ - Description: How answers become recommended, mentioned or absent grades, and how to check the evidence behind each one. Cheap Seats asks assistants your buyers' questions. A separate Claude call grades each answer **recommended**, **mentioned** or **absent** for your brand. Every answer, search, result, citation and grade is stored so you can check the reasoning or export the evidence. ## Which assistants it asks Claude and OpenAI models, through their APIs or the `claude` and `codex` CLIs. These are model responses, not the consumer apps; Google's AI surfaces, Gemini, Perplexity and Copilot aren't covered. [Which AI engines Cheap Seats asks](../engines/) explains the coverage and limits. ## Two tracks **From question to fix.** Available assistants answer on enabled tracks, for the configured samples. Answers and grades stay separate by track. Buyer questions → each available configured assistant on enabled tracks, repeated for the configured samples. A track can have zero samples. - Search: search the web, then answer; keep the searches, returned URLs and citations. - Recall: answer with no tools; keep the answer. A separate Claude call grades each answer. Stored answers and grades feed the rates, evidence and ranked fixes. Search requires web retrieval before answering; recall uses no tools. Repeated answers on either track can vary. Set `samples` per assistant and track; zero disables that track. ## Three grades The grader sees the question, answer, brand name, aliases and `about`, plus competitors' canonical names and aliases. It grades only your brand: | verdict | when | | --- | --- | | `recommended` | the top pick, or one of at most three primary recommendations actively endorsed for the buyer's situation | | `mentioned` | named among many options, in passing, as a comparison point or with reservations | | `absent` | not named (including aliases); named only to disclaim knowledge of it; or the name means another product | A table listing your brand alongside nine others is `mentioned`, not `recommended`. **An echoed name is not visibility.** “I couldn't find a product called Cheap Seats,” followed by evaluation advice or hypothetical features, is `absent`. An answer must state something about the brand itself—a feature, price, quality or fit, even hedged—to count as `mentioned`. Each grade also records: | field | meaning | | --- | --- | | **Position** | where your brand first appears among named products | | **Top pick** | the single product most strongly recommended, if any | | **Products named** | every product, folded onto competitors' canonical names for share of voice | | **Reasoning** | one or two sentences explaining the deciding wording | | **Why not #1** | what the answer preferred and why | | **Objections / praise** | specific concerns and credited strengths | | **Tone** | positive, negative, mixed (clear praise and criticism) or neutral (no judgment) about your brand; none when absent | | **Evidence** | the sentence best supporting the verdict; none when absent | **Evidence is checked.** Cheap Seats matches the grader's quote to the answer, ignoring case, spacing, markdown emphasis and curly quotes, then keeps the answer's own span. Unmatched quotes are dropped. The run page highlights the retained text; exports and MCP `get_result` include it as `evidence`. Tone and praise feed [net sentiment and themes](../metrics/#sentiment-and-themes). Questions marked `intent = "info"` ask for explanations, so they count toward named only, not recommended or top pick. Grading changes mark new methods in trends. Older grades may have no tone or evidence; evidence arrived in 0.2.5. Version 0.4.4 restored `absent` for echoed names after 0.4.3 briefly graded them `mentioned`; agreement on the 80-answer golden set rose from 90% to 97–99%. [Re-grade stored answers](../regrading/) to apply today's grading without asking the assistants again; replaced grades are retained. ## Worked examples These are from Cheap Seats' own runs on 22 October 2026: | answer | grade and reason | | --- | --- | | “I can't vouch for Cheap Seats specifically,” followed by no reliable product information | **Absent:** an echoed name states nothing about the product. A hedged feature or fit would make it mentioned. | | “I'd start by evaluating Cheap Seats,” but also “I don't have reliable enough information about Cheap Seats' capabilities” | **Close call:** graded ten more times, this answer was mentioned nine times and recommended once. Read the reasoning on borderline answers. | | An answer couldn't verify the intended product, then described an unrelated product selling WhatsApp agents | **Absent:** a namesake isn't your brand. The grader uses `about` to distinguish them. | | An answer warmly endorsed Gauge and never named Cheap Seats | **Absent for Cheap Seats:** endorsing a competitor doesn't endorse your brand. A text check enforces absent if neither your name nor any alias appears. | ## Rates and intervals The run page shows named, recommended and top-pick rates per assistant and track, each with a 95% Wilson interval. The share-of-voice board counts all named products. [Metric definitions](../metrics/) gives every denominator; [How big a change Cheap Seats can detect](../noise-and-confidence/) explains sampling uncertainty. ## Checking a grade Expand an answer on the run page to read its full text, highlighted evidence, searches, results, citations and grading reasons. Each run stores its grading instructions, config commit and Cheap Seats version, and exports to CSV or JSON. The grader is checked against human-labeled stored answers in `internal/grade/testdata/golden/`. Run the real grader with: ```bash CHEAPSEATS_GOLDEN=1 go test ./internal/grade/... -run Golden cheapseats golden export ``` The test fails below its agreement threshold; the export samples a deployment's answers for labeling. Grader changes must pass this check. The repository's [grading procedure](https://github.com/cheapseatsdev/cheapseats/blob/main/docs/grading.md) has the details. --- # Metric definitions - URL: https://cheapseats.dev/docs/metrics/ - Description: Every Cheap Seats metric, with its formula, denominator, exclusions and export field. These metrics are calculated from stored answers and grades, without new model calls. Find them on the run page, in JSON (`/api/runs/` and `/runs//export.json`, under `Summary`), in `cheapseats report -json ` (under `summary`) and MCP `get_run`. Terms used throughout: - **Graded answer**: one answer from one answerer, on one track, that the grader graded. Errors and unfinished answers count nowhere. - **Choice prompt**: a prompt that asks for a choice. Prompts marked `intent = "info"` ask for an explanation, so their answers count toward named only, never toward recommended, top pick, named first or top 3. - **Named**: graded `recommended` or `mentioned`. For your brand this is always the grade, never the grader's list of names, so your named rate and your share-of-voice row always agree. - **Products named**: the grader's list of every product an answer names, in order, folded onto your competitors' canonical names. Each product counts at most once per answer. ## Your rates | metric | numerator | denominator | JSON | | --- | --- | --- | --- | | **Named** | answers that name the brand | every graded answer | `visible` | | **Recommended** | answers graded `recommended` | answers to choice prompts | `recommended` | | **Top pick** | answers whose single top pick is the brand | answers to choice prompts | `top_pick` | Each comes with a 95% Wilson interval, so 0 of 10 and 0 of 1,000 don't read the same. For \(h\) hits of \(n\) answers, with \(p = h/n\) and \(z = 1.96\): $$ \text{centre} = \frac{p + \frac{z^2}{2n}}{1 + \frac{z^2}{n}} $$ $$ \text{half-width} = \frac{z\sqrt{\frac{p(1-p)}{n} + \frac{z^2}{4n^2}}}{1 + \frac{z^2}{n}} $$ The run page shows these per answerer and track, and for the run as a whole. The run's rates pool its answers: a prompt with more answers counts for more. Rankings and trends instead take the mean over prompts, each prompt counting once (see [Trends](#trends-and-outcomes)). ### Unbranded and branded A **branded prompt** names your brand or one of its aliases as a whole word ("Is Acme any good?"). Being named in its answers proves little. The track cards lead with the **unbranded** rates, the same three rates over the answers to every other prompt (`unbranded` in JSON), and show the rates with branded prompts underneath. The prompt list marks each branded prompt. Rankings and trends use the overall rates. ## Share of voice and position **Same answers, different denominators.** Illustrative data. Each product counts at most once per answer. Four answers name these products: (1) Acme and Globex; (2) Acme and Globex; (3) Acme and Initech; (4) none. Named rates divide by four answers: Acme 3/4 = 75%; Globex 2/4 = 50%; Initech 1/4 = 25%. They sum to 150% because an answer can name several products. Share of voice divides by six product mentions: Acme 3/6 = 50%; Globex 2/6 = 33.3%; Initech 1/6 = 16.7%. These shares sum to 100%. The share-of-voice board lists every product the answers named, not only the competitors you configured. | column | formula | adds up to | JSON | | --- | --- | --- | --- | | **Named** | answers naming the product ÷ every graded answer | can exceed 100% | `named_rate` | | **Share of voice** | answers naming the product ÷ the sum, over every product, of answers naming it | 100% | `share_of_voice` | | **Avg place** | mean of the product's place among the products an answer names, over the answers naming it at a known place | | `avg_position` | | **First** | answers naming it first ÷ answers to choice prompts | | `named_first` | | **Top 3** | answers naming it first, second or third ÷ answers to choice prompts | | `top3` | | **Weighted** | mean over every graded answer of \(\frac{N - \text{place} + 1}{N}\) at a known place, where \(N = \max(\text{products named}, \text{place})\); 0 when absent or place is unknown | | `position_weighted` | | **Top pick** | answers whose top pick is the product | | `top_pick` | - **Named rate was called share of voice.** In earlier versions the board's one rate column was labelled share of voice. That number is now **named rate**; **share of voice** is the true share, which sums to 100%. The JSON still writes the old field, `share`, with the named rate in it, so existing readers keep working. - **Place.** Every product uses its order in the grader's canonicalized list of names. If that list omits your brand but its grade says named, the grade's `position` is used when available. Unknown positions are excluded from avg place and contribute zero to first, top 3 and weighted. - **First and top 3** carry 95% Wilson intervals in JSON. - **Weighted** is position-weighted visibility as SE Ranking defines it: first of four scores 1, fourth of four 0.25, absent 0. - **Rank** is your place on the board: 1 + the number of products named in more answers than you. Products named in the same number of answers share a rank, and a brand no answer named has none (0 in the API). ### Head to head For each product on the board, against your brand, over every graded answer (`head_to_head`): | count | which answers | | --- | --- | | both named | name both | | you ahead / it ahead | of those, which led: the answer's top pick if it's one of the two, otherwise whichever was named first. An answer whose order isn't known counts for neither | | only you | name your brand and not the product | | only it | name the product and not your brand | | neither | name neither | ### Objections The concerns raised in answers that name your brand (`objections`), folded for counting: lowercase, every run of punctuation and spaces one space, so "Pricing unclear." and "pricing, unclear" count once. Each objection counts once per answer, with the first prompt it came up on as an example. With `[typesafe]` and `cluster_phrases` on, wordings Jev judged to say the same thing count as one row, named by the wording seen first, with `wordings` listing the rest; the same applies to praise. ## Sentiment and themes The grader records each answer's **tone** about your brand (positive, neutral, mixed or negative; none when absent) and the **praise** it gives, the strengths it credits you with. See [How grading works](../how-grading-works/#three-grades). | metric | formula | JSON | | --- | --- | --- | | **Net sentiment** | (answers positive − answers negative) ÷ answers naming the brand that have a tone, shown in points from −100 to +100 | `net` (−1 to 1) | | **Tone mix** | answers naming the brand, per tone | `tones` | | **Praised for**, **Objections** | the five strengths and five objections the most answers named, each counted once per answer | `praise`, `objections` | Grades from before tone was graded have no tone, so net sentiment is over the answers that do, and the page says how many name the brand in all. With no toned answer, net sentiment is n/a. **Themes** come from one more grader call after a run is graded. It groups the run's objections, praise and reasons for picking something else into at most eight named themes, each with a kind (strength, weakness or competitive), the answers that said it, the product it helps (always one of your competitors or a product the answers named), and a quote copied word for word from what it groups. Counts, quotes and beneficiaries are checked against the inputs. With fewer than five distinct strings to group there's no call, and the run says so. ## Second opinions With `[typesafe]` and `second_opinions` on, Jev reads every graded answer again under the grader's own verdict definitions (both are built from one text) and picks the sentence of the answer that most supports its verdict. Each opinion is compared with the answer's *current* grade when read, so a re-grade changes the comparison and not the stored opinion: | | when | | --- | --- | | **agrees** | Jev chose the grader's verdict, at a probability of at least `min_confidence` | | **disagrees** | Jev chose another verdict, at a probability of at least `min_confidence` | | **unsure** | Jev's probability for its choice is under the threshold | **Agreement rate** = agreed ÷ (agreed + disagreed), over opinions Jev was sure about. The set page shows it per run as **Jev agrees**. Changes can reflect different answers as well as grading variation; inspect disagreements before drawing conclusions. Evidence is **same** when the sentences match after folding case, spacing and punctuation, or one contains the other; otherwise **differs** or **none**. `second_opinion` in JSON carries counts and disagreements. MCP `get_run` returns counts and up to ten disagreements; `get_result` returns one answer's. ## Sources Every page an answer cited, or its searches returned, is sorted into a kind of site: | kind | what | | --- | --- | | `own` | your sites (`[[brand.sites]]` and `brand.domains`) | | `competitor` | a competitor's `domains` | | `ugc` | forums, Q&A, video and social: Reddit, YouTube, Stack Overflow and Stack Exchange, Hacker News, Quora, Medium, dev.to, X, LinkedIn, GitHub issues and discussions | | `review` | G2, Capterra, TrustRadius, Product Hunt, Gartner Peer Insights, GetApp, Trustpilot and the like | | `reference` | Wikipedia, `docs.*` hosts, package registries, the rest of GitHub | | `institutional` | .gov, .edu, .mil, .int and their country forms (gov.uk, ac.uk) | | `editorial` | everything else | Your sites come first, then competitors' domains, then `[sources]` in `cheapseats.toml`, then the built-in lists; within a list the most specific entry wins. See [Configuration](../configuration/#sources). With `[typesafe]` and `page_types` on, a third-party host the lists would call `editorial` is classed by Jev once, from up to three of its pages' titles and URLs, and that sits between `[sources]` and the built-in lists. For each page, host and kind: | metric | numerator | denominator | JSON | | --- | --- | --- | --- | | **Retrieved** | answers whose searches returned it | answers that could reach a page: the search track, plus any answer that searched, fetched or cited | `retrieved_rate` | | **Cited** | answers that linked it | answers that could reach a page: the search track, plus any answer that searched, fetched or cited | `cited_rate` | | **Share of citations** | its citations, one per answer that linked it | every citation in the run | `citation_share` | | **Cited when retrieved** | answers that both retrieved and linked it | answers whose searches returned it | `cited_when_surfaced` | A host counts each answer once, however many of its pages the answer used. Recall answers have no tools, so they're left out of these rates rather than diluting them. ### Buckets, outreach and found but not cited Each host that isn't yours gets a **bucket** from the products named in the answers that cite it: | bucket | the answers citing it name | | --- | --- | | **missing** | competitors, never your brand | | **shared** | both, with some competitor named at least as often as you | | **strong** | your brand more often than any one competitor | | **unique** | only your brand | With competitors configured, only they count as competitors here; with none, every other product does. A host whose citing answers name neither gets no bucket. - **Cited without your brand** (`outreach`): the missing hosts that aren't a competitor's own, ranked by distinct competitors named × answers citing the host. These are the pages to get your brand onto. - **Found but not cited** (`found_not_cited`): your own pages that searches returned and no answer linked. ### Neutral retrieval With `[exa]` and `baseline` on, each prompt is also run as a plain web search on Exa, with no domain filter, and the top `baseline_results` pages (default 10) are kept. Per host, with a competitor's domains as one row and your sites as one: - **neutral**: the prompts a neutral index ranks a page of the host for - **assistants**: the prompts any answer retrieved or cited a page of the host for - the **reading** is **content** when neutral is at least 2 and at least the assistants' count (the web agrees with the assistants, so the work is content), **query** when the assistants found it for 2 or more prompts and a neutral search never ranked it (only their own searches find it), and **both** otherwise. A host one answer retrieved once, and the index never ranked, isn't a row. Under each prompt, the neutral top pages carry whose they are and the same status as coverage: `cited`, `retrieved` or `unseen`. `baseline` in the run's JSON carries all of it; MCP `get_run` carries the top ten rows as `neutral_retrieval`. ### Page types The kind of site says whose page it is; the **page type** says what the page does, from its URL and the title the search result or citation gave it (`page_type` on each page, rolled up in `page_types` with cited, retrieved, pages and share of citations). The first rule that matches wins: | type | when | | --- | --- | | `forum` | Reddit, Quora, Lobsters, Discord; Hacker News items; Stack Overflow and Stack Exchange questions; GitHub or GitLab issues, pull requests and discussions; `forum.*`, `community.*`, `discuss.*` hosts | | `repo` | a repository's front page: github.com/owner/repo (and GitLab, Codeberg, Bitbucket) | | `docs` | `docs.*` hosts, a `/docs/`, `/documentation/` or `/reference/` path, Read the Docs, pkg.go.dev, docs.rs, MDN, Microsoft Learn | | `comparison` | the path or title says vs, versus, compare or comparison, or "alternative to" | | `review` | G2, Capterra, TrustRadius, GetApp, Software Advice, Trustpilot, PeerSpot, Gartner reviews; after the listicle rule, any page whose path or title says review | | `listicle` | best (not "best practices"), top N, N best, alternatives, tools/apps/software/platforms for | | `article` | a `/blog/`, `/news/`, `/articles/`, `/posts/` path, a dated path (`/2026/05/`), `blog.*` and `news.*` hosts, Medium, dev.to, Substack, Hashnode | | `product` | the home page, `/pricing`, `/plans`, `/features`, `/product`, `/solutions`, `/integrations` | | `other` | everything else (Wikipedia, a repository's files) | What the page is beats where it's filed: `/blog/notion-vs-obsidian` is a comparison and `/blog/best-note-apps` a listicle. MCP `get_run` carries `page_types` and each top page's `page_type`. With `[typesafe]` and `page_types` on, a page the rules call `other` is typed by Jev once, from its URL, title and the definitions above, and its chip says so; a page the rules already type is never asked about. ### Opportunities The two together (`opportunities`): each kind of site, and each page type within it, as one row. For a row whose pages have `c` of the run's `C` citations, cited by `n` answers of which `b` name the brand and `k` name a competitor: - **citation share** (`citation_share`) = `c ÷ C` - **brand presence** (`brand_presence`) = `b ÷ n`, and **competitor presence** (`competitor_presence`) = `k ÷ n`, with `competitors_named` counting each competitor - **gap** = citation share × (1 − brand presence) A kind of page the answers lean on and that rarely names you scores highest. A row is **owned** (`own`: your pages, so the fix is writing that kind of page) or **earned** (everyone else's, so the fix is getting onto them). A row with fewer than 3 citations isn't listed on its own, since its presence rate would be one answer's say-so; its citations still count toward its kind of site. Each row carries its most cited hosts and the prompts its citing answers were for. The analyst gets the top rows beside the answers, so an outreach or new-page fix can name the kind of page and its share rather than one host. It's a view over the stored citations and grades: no model call, and no change to any trend. MCP `get_run` carries `opportunities`. ## Citation stability How much the hosts a set's answers cite move from one run to the next, per track, over the same daily runs the trends use (each day's newest finished run, the last 30 days by default). A run's citations are a vector of host **citation shares** \(s_h\): a host's citations over all the run's citations, so \(\sum_h s_h = 1\). For each run against the one before it, with shares \(a\) and \(b\): | metric | formula | range | | --- | --- | --- | | **Similarity** (Bray–Curtis × 100) | the share of citations the two runs have in common by host; calculated below | 100 when every host kept its share; 0 when no host was cited in both | | **Overlap** (Jaccard × 100) | 100 · hosts cited in both ÷ hosts cited in either | 0 to 100 | | **New**, **gone** | hosts cited this run and not the one before, and the other way round, largest share first | | | **Stability score** | the mean similarity over the transitions, with the mean overlap beside it | shown from **3 transitions** | $$ \text{Similarity} = 100\left(1 - \frac{\sum_h \lvert a_h - b_h \rvert}{\sum_h (a_h + b_h)}\right) $$ For citation shares, this simplifies to: $$ \text{Similarity} = 100\sum_h \min(a_h, b_h) $$ - **One or two transitions can be a fluke**, so the score waits for three. - **The chain stops at a change of method**: the run that changed it is the oldest compared. - **A run that cited nothing** (an outage, or the recall track, which has no search) is skipped rather than read as every host leaving. - **A change of prompts** flags its transition `prompts_changed`: some of its churn is the questions. A high score means sources have been consistent; a low score means they vary between runs. Neither establishes why a host entered or left. Find it on the set page, at `GET /api/sets/{id}/citation-stability` and MCP `get_citation_stability`. ## Trends and outcomes | metric | formula | | --- | --- | | **A day's rate** (rankings) | the newest finished run that UTC day, per track: the mean over prompts of each prompt's rate, each prompt counting once | | **Rolling average** | the mean of the window's days: 7 by default, or 14 or 28 | | **Band, across prompts** | rolling mean \(\pm 1.96\frac{s}{\sqrt{m}}\), where \(s\) is the standard deviation of the \(m\) prompt-day rates in the window | | **This set, today** | the day's mean \(\pm 1.96\frac{\sqrt{\sum_k \operatorname{var}_k}}{K}\) over its \(K\) prompts | | **Vs day before** | \(D\) = mean over the prompts both days asked of (today − yesterday), with a 95% margin; within noise when \(D \pm \text{margin}\) covers 0 | | **A set's prompts** | each prompt's answers pooled over the window, with a Wilson interval | | **Fix outcome** | the same paired \(D\), before and after the fix, over its target prompts | | **Quick wins gain** | speed × mean over the fix's prompts of \((1 - p)\), where speed is prompts per working day and \(p\) is the prompt's recommended rate (named, for an info prompt) pooled over the last 14 days | \(\operatorname{var}_k\) is a prompt's sampling variance: $$ \operatorname{var}_k = \frac{\tilde p(1 - \tilde p)}{n}, \qquad \tilde p = \frac{h + 0.5}{n + 1} $$ [How big a change Cheap Seats can detect](../noise-and-confidence/) explains these intervals and tests, with a worked example. ## How these compare to other tools See [Comparing metrics across tools](../comparing-metrics/#how-these-compare-to-other-tools) for the vendor definition table and its dated sources. ### Migrating between Cheap Seats and these tools See [migration differences](../comparing-metrics/#migrating-between-cheap-seats-and-these-tools) before comparing numbers or moving prompts. --- # Comparing metrics across tools - URL: https://cheapseats.dev/docs/comparing-metrics/ - Description: Published metric definitions and migration differences for Cheap Seats and other AI visibility tools. Names such as visibility and share of voice can describe different formulas. Use [Cheap Seats metric definitions](../metrics/) for its formulas and denominators, and this reference when comparing or migrating tools. ## How these compare to other tools Most AI visibility tools use the same few words for different formulas. The table below is from each vendor's published documentation as of October 2026: [Peec AI](https://docs.peec.ai/metrics/brand-metrics) and its [domains](https://docs.peec.ai/domains), [Profound](https://docs.tryprofound.com/cookbook/metrics/how-metrics-are-calculated.md), [Gauge](https://docs.withgauge.com/introduction/core-aeo-metrics), [Otterly.AI](https://help.otterly.ai/brand-report-kpi-definition) and its [API docs](https://docs.otterly.ai/api-reference/brand-reports/update-a-brand-report), [Ahrefs Brand Radar](https://help.ahrefs.com/en/articles/15501968-ai-visibility-metrics), [SE Ranking](https://visible.seranking.com/?p=4633), and the two open-source trackers, Elmo ([visibility](https://www.elmohq.com/docs/user-guide/visibility), [sentiment](https://www.elmohq.com/docs/user-guide/sentiment), [share of voice code](https://github.com/elmohq/elmo/blob/ed3c6a23227f90e653df8b8722ee3a8fb6c4e5e6/apps/web/src/lib/visibility-stats.ts)) and Limelit Open ([methodology](https://github.com/limelit-co/open/blob/982668c4b3974558d900565f4a10bf9bf4e2dc88/docs/methodology.md)). Check their pages for changes since. | concept | Cheap Seats | Peec AI | Profound | Gauge | Otterly.AI | Ahrefs | SE Ranking | Elmo | Limelit Open | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | **Visibility** | **named rate**: answers naming you ÷ every graded answer | visibility: responses mentioning you ÷ all responses | visibility: runs naming you ÷ runs naming any brand | visibility: answers naming you ÷ all answers | brand coverage: prompts mentioning you ÷ all prompts | a mention: you appear at least once in an answer | visibility: mean of \(\frac{N - \text{best position} + 1}{N}\), 0 when absent | visibility: each prompt's latest answers mentioning you ÷ those answers; a name match is confirmed by a model (Jev) | visibility: answers containing your name string ÷ unbranded answers | | **Share of voice** | your mentions ÷ every product's mentions, one per answer, all products named | your mentions ÷ all brand mentions | your mentions ÷ all brand mentions, one per run | your mentions ÷ yours plus selected competitors' | your mentions ÷ all brand mentions | your share of impressions, weighted by search volume, against tracked brands | | your mentions ÷ yours plus tracked competitors' | your mention occurrences ÷ every tracked brand's occurrences in the same answers | | **Position** | **avg place**, over answers naming you; every product named counts | average rank among all brands detected, where you appear | average position, over runs naming you | not scored | avg. brand position: the sum of your positions ÷ the prompts you appeared in | | in visibility, as above | average rank among the tracked companies an answer names | mean rank of the list items you appear in; prose has none | | **Win** | **top pick**, and **first** (named first ÷ choice answers) | win rate: chats where you're position 1 ÷ all chats | | | | | | | | | **Position-weighted** | **weighted**, SE Ranking's formula | | | | | | visibility | | | | **Sentiment** | **net sentiment**: % positive − % negative of answers naming you | 0–100 score | positive claims ÷ all claims | positive ÷ (positive + neutral + negative) | net sentiment: (positive − negative) ÷ total × 100 | | net sentiment: % positive − % negative | 0–100: positive 100, neutral and mixed 50, negative 0 | Limelit Cloud only | | **Citations** | cited rate (÷ answers) and citation share (÷ all citations) | cited (chats citing it ÷ all chats), citation share (its citations ÷ all citations), citation rate (citations per chat that used it) | citation share: your domain's citations ÷ all citations | citation rate: answers citing the page or domain ÷ all answers | | | | citations by URL, domain and category | citation share: your site's citations ÷ all citations | | **Branded prompts** | left out of the headline rates; shown underneath | | | left out of rankings by default | | | | | left out of the headline; kept in the grid, tagged | | **After a config change** | grades and rates stay as graded, unless you re-grade stored answers under today's grader; which pages are whose follows today's config | past chats reprocessed when a name or alias changes | re-applied to history from today's config | | every past day recalculated when names or competitors change | | | past answers re-judged for the changed company, 90 days back by default | every stored answer re-read from today's brands and rules | | **Intervals** | 95% intervals on every rate; changes tested against noise | none published | none published | none published | none published | "directional indicators" | none published | none published; 5 samples a run by default | `n` on every number; under 20 answers captioned a small sample | The Semrush AI Toolkit [sorts sources](https://www.semrush.com/kb/1594-ai-seo-metrics) into the same four buckets Cheap Seats uses, missing, shared, strong and unique, by what the answers citing them name. Its "shared" means a site whose citing answers name every competitor in the set; Cheap Seats' means one named at least as often as you. ### Migrating between Cheap Seats and these tools - **Cheap Seats' visibility is stricter than Profound's.** Profound leaves out the runs that name no brand at all; Cheap Seats' named rate counts every graded answer. On the same answers, Cheap Seats' number is lower or equal. To reproduce Profound's, take Cheap Seats' export and drop the answers whose `named` list is empty. - **Profound averages per model.** It works each metric out per AI model, then averages the models with equal weight. Cheap Seats' run-level rates pool the answers; its per-answerer rates are the per-model numbers, so average those to match. - **Gauge's share of voice depends on whom you compare with.** It divides by your mentions plus the selected competitors'. Cheap Seats divides by every product the answers named, so adding a competitor to your config doesn't change your share, except where it folds two spellings into one name. - **Peec AI's visibility is Cheap Seats' named rate.** Responses mentioning you over all responses. Peec's position also counts every brand detected, as Cheap Seats' avg place does. - **Otterly.AI counts prompts, not answers.** Brand coverage is the share of prompts that mention you. In Cheap Seats, count the prompts in the prompt list whose named rate is above zero. - **Ahrefs weights by search volume.** Brand Radar's AI share of voice is a share of estimated impressions. Cheap Seats weights every prompt you choose to ask the same, and estimates no traffic. - **SE Ranking's visibility is Cheap Seats' weighted column**, the same formula. - **History.** Profound applies today's topics, tags and brands to past answers, so editing your setup changes past numbers ([why past numbers change](https://docs.tryprofound.com/cookbook/metrics/why-past-numbers-change.md)). In Cheap Seats, a run's grades, rates and share of voice are fixed when it's graded; a change in how answers are asked or graded starts a new method on the trend rather than rewriting the old one. To carry the trend on after a grading change, [re-grade](../regrading/) the stored answers: it's a step you take, it keeps the replaced grades, and the run page says so. Only which pages count as yours or a competitor's, and the source kinds, are read from today's config for every run. - **Elmo's share of voice also depends on whom you track.** Like Gauge, it divides by your mentions plus your tracked competitors'. Its position is your rank among the tracked companies an answer names, where Cheap Seats' avg place counts every product named. - **Elmo leaves out "I don't know that brand".** A model reads each name match, and an answer saying it has no information about your brand isn't counted. Cheap Seats grades it `absent` too, unless the answer also states something about your brand (a feature, a price, a fit), which makes it `mentioned`. Elmo's visibility also counts each prompt's latest answers, where Cheap Seats' run rates pool the run's. - **Limelit Open counts strings, and occurrences.** Its visibility is a deterministic text search, so "I could not find Acme" counts as a mention, as its methodology says, where Cheap Seats grades it `absent`. Its share of voice counts every occurrence, not one per answer: an answer that names you once and a rival five times is a 1:5 share, where in Cheap Seats it's 1:1. - **History, again.** Elmo re-judges past answers when you add a competitor or change a name, 90 days back by default, and leaves older answers as they were, so a long chart can step at that point. Limelit Open re-reads every stored answer whenever the brands or its matching rules change. - **Your prompts move with you.** Cheap Seats' prompt sets are plain TOML in your own repo. Each prompt is a line of text, so you can paste them into any of these tools. See [Compare](/compare/). --- # How big a change Cheap Seats can detect - URL: https://cheapseats.dev/docs/noise-and-confidence/ - Description: Reading uncertainty, detecting changes, interpreting fix outcomes, and deciding whether more samples help. On a 17-prompt set with two answerers and one sample per track, one run has a worst-case sampling margin of about **±17 points for a reading** and **±24 points for a change between two runs**. Pool seven daily runs on each side and those margins shrink to about **±6** and **±9**. Reading uncertainty and change detection are different calculations. Cheap Seats calculates both from stored answers, without extra model calls. A change beyond its margin is evidence against sampling noise; it does not establish that your work caused the change. ## Worked example For 17 fixed prompts, on one track, with rates near 50% and equal answer counts on both sides: **More answers narrow the sampling margin.** 17 fixed prompts, one track, a rate near 50%, with equal answer counts on each side. Approximate 95% sampling margins, not universal thresholds. | Answers per prompt on the track | One reading | Change between two such readings | | ---: | ---: | ---: | | 2 | ±17 points | ±24 points | | 4 | ±12 points | ±17 points | | 8 | ±8 points | ±12 points | | 14 | ±6 points | ±9 points | These are sampling margins. More same-day samples do not remove changes in models or the web. These are approximate 95% sampling margins, not the chart's across-prompts band. Rates far from 50% are less noisy. A 10-point swing between two single runs is within noise in this example. [The calculation](#calculating-the-example) is below. ## What "no detectable change" means for a fix When you record work against a fix, Cheap Seats compares named and recommended rates on its target prompts: - **Before:** the fix's run and up to four same-method runs in the 28 days before it. - **After:** every same-method run since the work was marked, within 28 days of the newest. | status | when | | --- | --- | | pending | the set hasn't run since | | needs regrade | it has, but only under another grading method; [re-grade](../regrading/) the named runs | | not comparable | it has, but only with answers asked differently; re-grading cannot fix this | | too early | the newest run is less than 7 days after, or the target prompts average fewer than 4 answers since | | moved | named or recommended rose beyond its margin, and neither fell beyond its margin | | worse | one fell beyond its margin, and neither rose beyond its margin | | mixed | one rose and the other fell beyond their margins | | no detectable change (±X pts) | neither measured change excludes zero; X is named's margin | **No detectable change does not mean the fix failed.** The available answers cannot distinguish the observed change from zero. The margin is uncertainty around that observed change, not a bound on the size of a possible effect. A benefit or harm may remain undetected. For example, take three target prompts, two answerers, both tracks and one sample each: four answers per prompt per run. With **five runs pooled before the fix** (20 answers per prompt), the worst-case margins are: | after the fix | answers per prompt after | margin at a 50% rate | | --- | ---: | ---: | | one weekly run | 4 | about ±31 pts | | four weekly runs | 16 | about ±19 pts | A page is **newly surfaced** if searches or citations reached it after the work but not before. That shows retrieval or citation, not necessarily a recommendation change. Host citation changes are reported separately. See [From fix to pull request](../fix-to-pull-request/#4-merge-then-let-the-next-run-check-it). ## When more samples help Start with one sample per track and weekly runs to find gaps. The 28-day window and fix outcomes pool runs. Extra samples improve a single run's precision; runs over time also capture day-to-day variation. When an answerer takes multiple samples, the set page estimates whether they help. It splits run-to-run named-rate variance as: $$ V = \frac{S}{n} + D $$ \(S\) is within-run variance, \(n\) the samples and \(D\) between-run drift. This is calculated per prompt, answerer and track over 28 days of same-method runs, with at least 4 runs and 6 cells. Going down to one sample would widen the interval by: $$ w = \sqrt{\frac{S + D}{S/n + D}} $$ - **\(w < 1.25\):** extra samples buy little precision because drift dominates; consider reducing samples. - **Otherwise:** the page says samples are paying off and gives the estimated widening at one sample. **Adaptive sampling** (`[sampling] adaptive = true`, `max_extra = N`) assigns up to N extra answers before a run, using uncertainty in the last 28 days of same-method named rates. Each extra goes where it most reduces total variance. A cell gets at most its planned sample count in extras; API extras must fit the daily budget. A prompt named in 0 of 40 answers gets none. Adaptive sampling is off by default, does not restart trends, and its extras count in run rates. See [Configuration](../configuration/#sampling) and [Cost and budget](../cost-and-budget/#keeping-the-spend-down). ## Why answers vary Repeated answers differ in brand lists and ordering. [SparkToro and Gumshoe](https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/) found lists highly inconsistent while visibility across many answers was steadier. [Don't Measure Once](https://arxiv.org/abs/2604.07585) recommends pooling two to four weeks; [Sielinski's citation-share analysis](https://arxiv.org/abs/2603.08924) shows how small apparent gains can remain within uncertainty. More same-day answers cannot remove all variation: [Profound](https://www.tryprofound.com/blog/is-once-a-day-enough) found ten daily runs reduced daily visibility movement by only about 11% compared with one. Model and web changes also affect answers. ## Prompts are the units Repeated answers to one question aren't independent questions about your category. Trends therefore average each prompt's own rate, so prompts with more samples don't carry more weight. For a cell of **\(h\) hits out of \(n\) answers**, sampling variance is: $$ \operatorname{var} = \frac{\tilde p(1 - \tilde p)}{n}, \qquad \tilde p = \frac{h + 0.5}{n + 1} $$ Smoothing prevents 0/6 or 6/6 from claiming zero noise. ## The intervals | where | calculation and meaning | | --- | --- | | **Run page** | 95% Wilson intervals over answers; see [Your rates](../metrics/#your-rates) | | **Rankings: band, across prompts** | rolling mean \(\pm 1.96\frac{s}{\sqrt{m}}\), where \(s\) is the standard deviation of \(m\) prompt-day rates. Describes variation across questions like these. | | **Rankings: this set, today** | day's mean \(\pm 1.96\frac{\sqrt{\sum_k \operatorname{var}_k}}{K}\) over \(K\) fixed prompts. Covers sampling answers to this set, not variation in the choice of questions. | | **Set page: each prompt** | answers pooled over the selected window, with Wilson intervals on named and recommended | Windows are 7 days by default, or 14 or 28 (`?window=`, `/api/rankings`' `window`, or MCP `get_rankings`). Rankings windows stop at method changes. Per-prompt pooling starts from the newest day asking it and also stops at wording or intent changes. ## The test for a change Day-over-day and fix comparisons pair the prompts both sides asked: $$ D = \frac{1}{K}\sum_k \left(p_{\text{after},k} - p_{\text{before},k}\right) $$ $$ \operatorname{Var}(D) = \frac{\sum_k \left(\operatorname{var}_{\text{after},k} + \operatorname{var}_{\text{before},k}\right)}{K^2} $$ $$ \text{margin} = 1.96\sqrt{\operatorname{Var}(D)} $$ A change is **within noise** when \(D \pm \text{margin}\) covers zero. Rankings greys it out and shows `±X pts`. Rank changes use the paired change in the gap between your brand's and each crossing product's answer shares. If no crossing is significant in the rank move's direction, the table shows **within noise** without an arrow. These tests cover sampling variation under this calculation. They do not rule out model updates, web changes or other causes of a detected change. ### Calculating the example The 17-prompt example has two answers per prompt on one track. At 50%: $$ \tilde p = \frac{1 + 0.5}{2 + 1} = 0.5 $$ $$ \operatorname{var} = \frac{0.25}{2} = 0.125 $$ $$ \sum_k \operatorname{var}_k = 17 \times 0.125 = 2.125 $$ Multiplying by 100 converts the margins to percentage points. For one reading: $$ \pm 100 \times 1.96\frac{\sqrt{2.125}}{17} \approx \pm 16.8 $$ For a change between two runs: $$ \pm 100 \times 1.96\frac{\sqrt{2 \times 2.125}}{17} \approx \pm 23.8 $$ Seven daily runs give each prompt 14 answers on the track: $$ \tilde p = \frac{7 + 0.5}{15} = 0.5 $$ $$ \operatorname{var} = \frac{0.25}{14} \approx 0.0179 $$ $$ \sum_k \operatorname{var}_k \approx 17 \times 0.0179 \approx 0.304 $$ For one pool, in percentage points: $$ \pm 100 \times 1.96\frac{\sqrt{0.304}}{17} \approx \pm 6.4 $$ For a change between two such pools: $$ \pm 100 \times 1.96\frac{\sqrt{2 \times 0.304}}{17} \approx \pm 9.0 $$ --- # Which AI engines Cheap Seats asks, and why - URL: https://cheapseats.dev/docs/engines/ - Description: Claude and OpenAI coverage, the limits of API and CLI answers, and when you need other AI surfaces. Cheap Seats asks **Claude and OpenAI models** through APIs or subscription CLIs. It doesn't ask the consumer ChatGPT or Claude apps, Google AI Overviews, AI Mode, Gemini, Perplexity or Microsoft Copilot. If those surfaces matter to your buyers, you need additional coverage. ## How Cheap Seats asks | provider | asks | billed | | --- | --- | --- | | `claude-code` | Claude models through the `claude` CLI | your Claude subscription | | `codex` | OpenAI models through the `codex` CLI | your ChatGPT subscription | | `anthropic` | Claude models through the API | per token | | `openai` | OpenAI models through the API | per token | Claude is required; OpenAI adds an optional second assistant. CLIs run where installed, on your machine or a VM; Docker uses the APIs. CLI calls run in an empty directory with your coding instructions, hooks, skills, MCP servers and Codex config switched off. See [Configuration](../configuration/#answerers), [CLI answerers](../reference/#cli-answerers) and [Cost and budget](../cost-and-budget/). ## Search and recall **Search** requires web retrieval; **recall** uses no tools. Cheap Seats reports them separately, unlike a consumer app that chooses when to browse. Both can vary between answers. [How grading works](../how-grading-works/#two-tracks) explains the tracks and samples. ## Why not scrape the consumer apps Tracking a consumer surface requires obtaining the answer that surface rendered, rather than substituting an API model response. Other trackers use paid scraping services. Cheap Seats avoids that dependency so you can start on a subscription you already have. Scraping adds: - **Cost and setup:** another account, key and bill. Elmo estimates $0.15–$2.25 per prompt per month at its default cadence of about 300 evaluations per prompt ([providers](https://www.elmohq.com/docs/user-guide/providers), October 2026). - **Maintenance:** scrapers must follow changes to the products they drive. Elmo's provider table records two services losing ChatGPT search results in late September 2026 and a Google AI Overview token expiring within a minute. ## What that misses | limitation | implication | | --- | --- | | **Consumer apps** | Memory, personalization, system prompts and browsing decisions can differ. These model responses don't reproduce a particular buyer's chat. | | **Google AI Overviews and AI Mode** | No coverage. If Google search is central to your category, this is a substantial gap. | | **Perplexity, Gemini and Copilot** | No coverage, even where APIs exist. | | **Model versions** | An API model and consumer app may use different builds. Limelit treats routes to one engine as separate targets ([methodology](https://github.com/limelit-co/open/blob/982668c4b3974558d900565f4a10bf9bf4e2dc88/docs/methodology.md#two-ways-to-reach-one-engine-are-two-targets)). | The [site audit](../site-audit/) checks crawler access. [AI crawler counts](../site-audit/#ai-crawlers) include PerplexityBot and referrals from Perplexity, Copilot and Gemini. Those show access and traffic, not what their answers said. ## How other tools reach them As of October 2026: - **Elmo** (MIT) supports six scraping services for ChatGPT, Google's AI surfaces, Gemini, Perplexity and Copilot, plus direct APIs for Claude, OpenAI, Mistral and OpenRouter models ([providers](https://www.elmohq.com/docs/user-guide/providers)). - **Limelit Open** (Apache-2.0) uses APIs for ChatGPT, Claude, Perplexity and Gemini, and scrapers for Google AI Overviews, AI Mode and Bing Copilot ([README](https://github.com/limelit-co/open/blob/982668c4b3974558d900565f4a10bf9bf4e2dc88/README.md#engines-and-providers)). - **Hosted tools** offer broader surface coverage on some or all plans. [Compare](/compare/) lists the engines each names. ## When to move to a tool that covers them Use a broader tracker when the surfaces your buyers use exceed this coverage. Your prompts move with you: they're plain TOML in your config repo. [Compare](/compare/) covers the alternatives. --- # Reviewing your prompts - URL: https://cheapseats.dev/docs/prompt-review/ - Description: Review monthly to retire uninformative prompts, clarify noisy questions and find gaps in the set. **Review prompts monthly.** After about four weekly runs, check which questions are uninformative, noisy, off target or duplicates, then edit the set through your config repo. The review uses stored answers, grades, searches and citations; opening it calls no model and costs nothing. Optional judgments and proposed edits are described below. ## Where to find it - **Page:** `/sets//review`, linked from the set. Choose `?days=30` (default), `60` or `90`. - **CLI:** `cheapseats review [-days 30] [-json] [-suggest] [-jev]`. - **MCP:** `review_prompts` takes `set`, optional `days` and `suggest`; returns counts and the top five per flag, rounded. `get_started` suggests a review after four weeks of runs if none was recorded in 30 days. - **JSON:** `GET /api/sets//review?days=30` returns every prompt's numbers. Use **Mark reviewed** on the page; CLI and MCP record a review automatically. The JSON API doesn't. No review runs on a schedule. ## Flags Each flag includes its numbers and an action. A prompt with enough data and no flag is **healthy**. | flag | action | rule | | --- | --- | --- | | **Flat: always absent** | Keep it as an important gap, reword it toward what the brand does, or retire an unwinnable question. | No answer named the brand, and no fix in the window improved this prompt's own named or recommended rate beyond its 95% margin. Another target prompt improving doesn't count. | | **Flat: always recommended** | Make it harder, or retire it to save answers. It currently detects regressions only. | Every answer recommended the brand; for info prompts, every answer named it. | | **Noisy: drift** | Add buyer and task context. More same-run samples don't remove drift. | Per-answerer-and-track χ² homogeneity tests, summed over cells, give p < 0.01, with at least 15 points of run-to-run spread. | | **Noisy: disagreement** | Clarify the question or add samples. | Within-run answers split about 30/70 or closer, with per-run sampling error at least 15 points. | | **Off target** | Add category or buyer context; configure unexpected products if they are competitors. | With ≥3 competitors and ≥5 product mentions, under 30% are the brand or competitors; **or** an unexpected product appears in at least half the answers, more than each configured product; **or** at least 25% use the brand's name for another product. | | **Duplicate** | Keep one and retire the other. | Same intent, weighted Jaccard product overlap ≥0.75, and verdict shares within 15 points. | | **Rewritten by the assistant** | Consider the words assistants search with. | At least 5 searches with under 25% word overlap with the prompt, both ways. Added words are listed. | | **Branded leak** | Reword for a buyer who hasn't heard of you. If only searches add the name, check for brand-specific language; assistants may have found it independently. | Outside `head-to-heads`, the prompt names the brand, or at least 2 searches and 20% of searches add it. | Noise flags use planned samples, excluding adaptive extras and runs where all answers to the prompt errored. With one sample per answerer and track, drift and disagreement cannot be separated, so neither is flagged. A product's name and alias count once per answer. See [sampling and drift](../noise-and-confidence/#when-more-samples-help). ## What it reads The review reads finished runs in the selected window, walking backward under each prompt's latest asked wording and stopping when that wording changes. - Intent and method changes do **not** stop pooling; the row warns that its rates span both sides. - If config wording changed after the last run, the review uses the old wording and says so. - Flags need **3 runs and 6 graded answers**. With less, the prompt is under **not enough data yet**, except branded prompts are identified immediately. ## Jev's judgments With `[typesafe]`, Jev can flag wording problems before a set has run. Each question is a separate request: | judgment | question | | --- | --- | | **Duplicate** | Would one good answer serve two same-intent prompts sharing a meaningful word? | | **Off target** | Is the prompt outside `brand.category`? Skipped without a category. | | **Intent** | Should a useful answer name a category product (`selection`) or none (`info`)? A how-to about a job those products do is selection. | | **Branded** | Outside `head-to-heads`, does it name the brand, a product of its maker or its domain in any spelling? | Judgments are cached by their exact inputs: unchanged checks make no requests. A 20-prompt set takes about 70 requests initially, a fraction of a cent. `cheapseats check` prints `suggestion:` lines at `min_confidence`; suggestions never fail the check, and `-no-jev` skips them. The pass also runs after a run, or manually with `cheapseats judge ` or `cheapseats review -jev`. Review pages, JSON (`jev`), MCP and ordinary CLI reviews read cached judgments without asking again. See [Optional providers](../optional-providers/). ## Gaps The review proposes at most five of each: | gap | rule | | --- | --- | | **Competitor** | Configured competitor appears in ≥20% of answers or is added to ≥3 searches, but no prompt names it. | | **Product** | Unconfigured product appears in ≥20% of answers across ≥2 prompts. Includes a `[[competitors]]` entry. | | **Topic** | A word absent from prompts is added in ≥5 searches across ≥2 prompts. Includes its most-run search. | Each includes a buyer-worded `[[prompts]]` entry with an unused id. Reword before committing; ids are permanent. ## Proposed edits (optional) The page button, `cheapseats review -suggest`, or MCP `suggest: true` sends flags and counts to the analyst for one call, on your Claude subscription or API daily budget. Web-derived names and words are quoted as data. The proposed diff contains rewrites that retain ids, retirements as comments to remove manually, and additions with unused ids. It never writes your config; apply changes through a pull request ([Prompt sets](../prompt-sets/)). Invalid edits are dropped with reasons: unknown prompts, unchanged rewrites, duplicate text, search operators, quotes, links, text over 300 characters, brand names outside `head-to-heads` or missing inside it, and anything `cheapseats check` rejects. --- # Got wrong about you: the fact check - URL: https://cheapseats.dev/docs/got-wrong/ - Description: Find answers contradicting your listed facts, with verified quotes and evidence for the fix list. An answer can recommend you and get your price, platform or product wrong. List facts under `[brand]` to check answers that name you for contradictions. Findings appear under **Got wrong about you** and feed the fix list. They do not change grades or trends. ```toml [brand] facts = [ "Cheap Seats is MIT licensed and runs as a single Docker container with a SQLite database.", "Answers come from Claude and OpenAI models through their APIs, or local Claude Code and Codex CLIs.", ] [grader] check_facts = false # off; on by default when facts are listed ``` Requires Cheap Seats 0.4.0 or later. No additional key or provider is needed. ## What it checks, and what it never does The grader model reads each eligible answer beside your listed facts: - **Eligible:** recommended or mentioned answers about your brand, excluding namesakes marked as [another product](../how-grading-works/#three-grades). - **Contradictions only:** the claim and listed fact cannot both be true. Omissions, vaguer statements, inability to verify, competitor claims and opinions aren't misstatements. The expected result is usually empty. - **Verified evidence:** each finding identifies the fact, explains the contradiction and quotes the answer. Invalid fact numbers and quotes absent from the answer are dropped, using the same [evidence check](../how-grading-works/#three-grades) as grading. Facts never reach the visibility grader. The fact check is stored separately, so changing facts doesn't restart trends. ## Cost, and when it runs again One grader call per eligible answer. Your subscription covers `claude-code` calls; API calls use the [daily budget](../cost-and-budget/) and cost a few cents per run on Haiku. Answers are checked once per facts list. Changing the list makes the next run's pass—or `cheapseats misstatements ` on an older run—check again and replace findings. Until then, the page shows how many answers were checked under the current and older lists. [Re-grading](../regrading/) doesn't run this pass. Rerun it manually if changed verdicts alter which answers are eligible. ## Where it shows | surface | what you get | | --- | --- | | **Run page** | Contradicted facts, most answers first; prompt markers and findings under each answer's grade. | | **Fix list** | Facts, prompts and a quote as evidence for `positioning` or `fix-page` work, with a checkable `done_when`. | | **API / MCP** | `misstatements` in `/api/runs/` and `get_run`: checked count, contradicted facts, answers, claims and quotes. | | **CLI** | `cheapseats misstatements ` checks a finished run and prints findings ([reference](../reference/#cli)). | --- # Re-grading stored answers - URL: https://cheapseats.dev/docs/regrading/ - Description: Apply today's grading inputs to stored answers while retaining replaced grades and leaving the original answers intact. Re-grading applies today's grader to stored answers without asking the assistants again. Use it after a grading change to reconnect eligible runs to today's trend, or after a competitor change to update name folding. The grader prompt, model and brand identity (name, aliases, `about`) affect the trend's **method**. Competitor names and aliases affect **grading inputs**, but changing competitors alone does not restart a trend. ```bash cheapseats regrade [-since 28d] # dry run: work and estimated cost cheapseats regrade -since 90d -dry-run=false ``` Nothing is written unless you pass `-dry-run=false`. ## What's re-graded **Re-grade the answers, preserve their dates.** Re-grading can reconnect a grading-method break. It cannot change how an old answer was asked. Before re-grading: old answers with old grades | grading-method break | current answers with current grades. After re-grading: the same old answers, graded with today's inputs, join the current-method trend from their original dates. The replaced grades remain stored; no buyer questions are asked again. An answer-prompt or search-policy change remains a comparison boundary. Competitor-only changes can require re-grading without starting a new Method. - **Set:** finished runs in `-since` (`28d`, `72h` or `all`; default 28 days). **Run:** that run only. - **Skipped:** runs already matching today's grading-input fingerprint: grader prompt and model, brand name, aliases and `about`, and competitors' names and aliases. - **Ineligible:** unfinished or running runs, or answers asked under another answer prompt or search policy. Re-grading cannot change how an answer was obtained. ## What it costs One grader call per answer. With `provider = "claude-code"`, your subscription covers the calls. API dry runs estimate from `[pricing]`, about four characters per token and the grader's thinking. Actual calls use the [daily budget](../cost-and-budget/#the-daily-budget); a run estimated to exceed the remainder isn't started. ## All or nothing, per run All new grades are calculated before writing them together. If grading fails, stops, or the run changes meanwhile (for example, another re-grade or retry), that run is left untouched. ## History stays honest The snapshot adopts today's grading inputs while retaining how answers were asked. Its recomputed method determines trend compatibility. - Replaced grades remain in `results.grade_history`. - `/api/runs/` records `regrades`: date, answer count, cost, previous method and grader. - The run page says **re-graded <date> (was method X)**. - Untouched runs can still create method breaks when only some runs are re-graded. ## What isn't rebuilt Fix lists, themes and the [fact check](../got-wrong/) remain as stored. Rebuild derived reports if needed: ```bash cheapseats fixes -rebuild cheapseats themes -rebuild ``` Rerun `cheapseats misstatements ` if changed verdicts make different answers eligible for the fact check. ## In the web UI and over MCP - **Web:** `/sets//regrade`, linked from the set, previews runs, answers, method changes and estimated cost for 7, 28 or 90 days or all time. Its button starts work in the background. - **MCP:** `regrade` takes `target` (set or run), `since_days` (default 28; 0 means all; negative is an error) and `dry_run` (true unless set false). See [How grading works](../how-grading-works/) and [Rankings](../reading-results/#rankings). --- # From fix to pull request - URL: https://cheapseats.dev/docs/fix-to-pull-request/ - Description: How a quick win becomes a pull request with Claude Code, shown on cheapseats.dev's own fix list. Pick a fix, give its prompt to Claude Code, review the change, then record the pull request so later runs can measure the outcome. The worked example began on 21 October 2026 with 0 of 90 answers naming Cheap Seats. The screenshots show a different fix in the current demo UI, captured on 24 October; they illustrate the controls, not the original PR. ## 1. Pick a fix ![The quick wins page with one fix expanded: where it goes, what it must say, when it's done, which page it displaces, and why.](https://cheapseats.dev/images/docs/quickwin-current-light.jpg) *A generated fix in the demo on 24 October. Read its evidence and verify its claims before implementing it.* Start from the top of quick wins, or filter by site to get the fixes your team owns. Each fix says: - **Where**: the exact page to create or change. - **Must say**: the facts the answers were missing. - **Done when**: a test you can re-run. - **Displaces**: the page that wins the question today. - **Why**: the evidence from the run. We chose two small documentation gaps where answers cited another vendor's help centre. We reserved positioning and competitor claims for human review. ## 2. Hand the prompt to Claude Code ![The fix's Prompt for Claude Code: the questions it affects, what the assistant said, who wins the question today, the task, what the page must say and the rules.](https://cheapseats.dev/images/docs/claude-prompt-current-light.jpg) *An excerpt of that fix's Claude Code prompt: target questions, recorded evidence, competing page and task. Scroll the prompt in the app to read the required claims and rules.* Open Claude Code in the repo that owns the site the fix belongs to, and paste the prompt. If you've connected Cheap Seats' [MCP server](../reference/#mcp-server), you can ask for it instead: "get the top quick win for cheapseats.dev and do it". `get_quick_wins` returns each fix with its prompt. The prompt names the target questions, competing page, required facts and scope. It forbids invented claims: anything the repo cannot support becomes a TODO. Add context only you have, such as actual run numbers, and point Claude Code at the source so it can verify the definitions. ## 3. Review the pull request Claude Code finishes with a pull request: the page, a summary of what it says, and any TODOs. Review it like any other change: - **Facts.** Every number and capability should trace to the code, the docs or the run. Check the TODOs and fill or cut them. - **Scope.** One fix, the page it named, linked from where readers will find it. - **Answer first.** The opening should answer the buyer's question directly and make its claims easy to verify. Check the prompt itself too: the analyst works from answers, not your source. Our first PR corrected this definition: | Analyst draft | Verified rule | | --- | --- | | Top pick or explicitly endorsed | Top pick or active endorsement among **three or fewer** primary options | The page used the grader's rule. The walkthrough produced [Mentioned vs. recommended](https://github.com/cheapseatsdev/cheapseats-site/pull/1) and [Track AI citations](https://github.com/cheapseatsdev/cheapseats-site/pull/2). Both checked definitions and report scope against source material. Mark the fix with the pull request while it's open: from its card in quick wins, or ask Claude Code to do it over MCP (`mark_fix`) as part of the same task. It leaves the open list, so the next person picking a fix doesn't pick it again. When it merges, mark it accepted (`set_attempt_state`). ## 4. Merge, then let the next run check it **Shipping and measuring are separate steps.** A merged PR records work shipped. Later comparable answers measure what changed; they do not prove its cause. Work: record a commit or PR → proposed → accepted when merged, or withdrawn when closed. A later fix for the same page can return as a retry. Measurement: collect same-method answers for the target prompts → check comparability and enough data → report named and recommended changes with their margins. Pending means no later run; too early means there is not yet a comparable run at least seven days after marking, or fewer than four answers per target prompt on average. A method mismatch may need re-grading; differently asked answers cannot be repaired by re-grading. Outcomes: moved, worse, mixed, or no detectable change. A newly surfaced page is supporting evidence, not a measured rate improvement. Once the page is live, run the same set again. Check the fix's **Done when** criteria and whether its page appears in the recorded searches or citations. A search result shows the page surfaced; a citation links it to the answer. Neither proves the change caused an improvement. The analyst writes a fresh fix list with the work already tried. An unresolved fix can return as *retry*, carrying its pull request and results for the target prompts. Search answers may discover a new page after indexing; both tracks can also vary between runs without a model release. Cheap Seats pools comparable answers for the fix's target prompts from up to five runs before it was marked and the runs since. Its outcome distinguishes: - **too early:** fewer than 7 days, or fewer than 4 post-change answers per target prompt on average; - **moved, worse or mixed:** named or recommended changed beyond its 95% margin; the direction and the other metric determine the status; - **no detectable change:** available answers do not distinguish the observed change from zero. `D ±X` is a margin around D, not a bound saying every possible effect is smaller than X; - **needs regrade:** later runs use another grading method; **not comparable:** they were asked differently, which re-grading cannot repair. A *newly surfaced* note means the page began appearing in recorded searches or citations. Fix markers on the [rankings](../reading-results/#rankings) chart help you inspect the surrounding trend. These before/after observations do not establish causality; [Noise and confidence](../noise-and-confidence/#what-no-detectable-change-means-for-a-fix) explains the limits. --- # Free site checks: audit, citations, AI crawlers - URL: https://cheapseats.dev/docs/site-audit/ - Description: Three free checks that call no model: what AI crawlers can read on your sites, which cited pages name you, and AI crawler counts from your access logs. Start with `cheapseats audit` to check what crawlers can read: robots.txt, firewall responses, llms.txt, sitemaps and raw HTML. Then check whether cited pages name you, and import access logs to compare crawler activity with citations. These checks call no model. HTTP requests and log imports are free; if [Exa fallback reads](../optional-providers/) are enabled, empty or challenged pages may incur a page-read charge against the daily admission budget. ## Site audit ```bash cheapseats audit # every site in [[brand.sites]] (and brand.domains) cheapseats audit example.com # just one, which needn't be yours (not kept) cheapseats audit -pages 20 -info # more sitemap pages; list info findings too cheapseats audit -json # the findings as JSON ``` Also **Site audit** in the web UI (`/audit`, "Audit now"), `POST /api/audit` to run one and `GET /api/audit` for the latest, and the MCP tool `audit_sites`. The audit fetches robots.txt, the home page, `/llms.txt`, the sitemap and up to 10 sampled sitemap pages. It reads raw HTML without JavaScript, with 4 concurrent requests, a 15-second timeout and a 2 MB page limit. Its user agent is `Mozilla/5.0 (compatible; CheapSeats-Audit/; +https://cheapseats.dev)`. - A host/path site (`github.com/example`) gets its robots.txt rules checked for that path, but no llms.txt or sitemap check: those belong to the host. - Sites of kind `code` or `community` (your GitHub, a forum) raise no robots.txt, firewall, llms.txt, sitemap or markup findings: none of that is yours to fix. Which crawlers the host blocks is still recorded on the site's `blocked` line, so you know it, without a fix you can't apply. - The newest audit of your own sites is kept. If it's less than 30 days old, its errors and warnings go to the analyst with the next fix list, so a fix can say "this page is empty without JavaScript" rather than only "this page wasn't cited". Each finding has a severity, a check id, what it found and a fix. ### Crawling rules | check | severity | what it means | fix | | --- | --- | --- | --- | | `robots.blocks-ai-search` | error | robots.txt blocks an assistant's search or user fetcher (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) | allow it: `User-agent: ` and `Allow: /` | | `robots.blocks-search-engine` | error | robots.txt blocks Googlebot or Bingbot, whose indexes Gemini, ChatGPT and Copilot search | allow it the same way | | `robots.blocks-training` | info | robots.txt blocks a training crawler (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider) | nothing, unless you want future models to learn from these pages. It keeps pages out of training data, not out of search answers | | `robots.server-error` | error | robots.txt answers 5xx, which crawlers read as "fetch nothing" | serve it with a 200, or a 404 if you don't want one | | `robots.unreachable` | warning | robots.txt couldn't be fetched at all | make sure `/robots.txt` answers quickly over HTTPS | | `robots.missing` | info | no robots.txt: everything is allowed | optional; add one to point crawlers at your sitemap | | `waf.blocks-ai-bot` | error | the home page answers 401, 403 or 429, or a challenge page, to an AI fetcher's user agent while a browser's gets 200 | allow the bot in your firewall or CDN bot settings (on Cloudflare, Security > Bots, AI Crawl Control) | **The firewall check only sends each bot's user agent, from Cheap Seats' address.** A firewall that lets verified bots through by IP may block this check and still admit the real crawler. Confirm in the firewall's log, or with the [crawler counts](#ai-crawlers) below, before changing rules. ### llms.txt and the sitemap | check | severity | what it means | fix | | --- | --- | --- | --- | | `llms.missing` | warning | no `/llms.txt`, or a catch-all route serving HTML there | publish one: `# `, a one-line summary in a `>` quote, then sections of `[title](url): description` links to your key pages | | `llms.content-type` | warning | llms.txt isn't served as text/plain or text/markdown | serve it as one of those | | `llms.heading` | warning | llms.txt doesn't start with `# ` | start it with `# ` | | `llms.no-links` | warning | llms.txt has no markdown links | list your key pages as `[title](url)` links | | `sitemap.missing` | warning | no sitemap in robots.txt and none at `/sitemap.xml` | publish `/sitemap.xml` with each page's ``, and add `Sitemap: ` to robots.txt | | `sitemap.invalid` | error | the sitemap doesn't parse | serve valid sitemap XML: a `` of `` entries, or a `` | | `sitemap.empty` | warning | it lists no pages | list every public page | | `sitemap.stale` | warning | its newest `` is over 180 days old | regenerate it when pages change, with each page's real `` | | `sitemap.no-lastmod` | info | it gives no `` dates | add `` to each ``, from the page's real last change | ### Pages | check | severity | what it means | fix | | --- | --- | --- | --- | | `page.unreachable` | error | the page couldn't be fetched | make sure it answers over HTTPS within a few seconds | | `page.status` | error | the page didn't answer 200 | serve it with a 200, or drop it from the sitemap if it's gone | | `page.challenge` | error | the page served a bot challenge instead of its content | let crawlers and plain HTTP clients through to public pages | | `page.needs-js` | error | under 200 characters of text in the HTML and an app mount point (`#root`, `#app`, `#__next`...): empty without JavaScript | render on the server or pre-render at build time (SSR or SSG) | | `page.thin` | warning | under 200 characters of text in the HTML | put the content in the HTML, not loaded by JavaScript afterwards | | `page.noindex` | error | a robots meta tag or `X-Robots-Tag` says noindex | remove noindex from public pages | | `page.jsonld-invalid` | error | a JSON-LD block doesn't parse | fix the JSON in each `