Configuration

cheapseats.toml, the file that says whose brand Cheap Seats measures, with which assistants, and how much it may spend.

Everything except secrets lives in cheapseats.toml in your config repo, with the prompt sets in prompt-sets/ beside it. Secrets (API keys, tokens) come from the environment or .env. Unknown keys are errors, so a typo fails cheapseats check instead of being ignored.

The examples below are from examples/cheapseats-config/cheapseats.toml, which measures Cheap Seats itself.

Brand

[brand]
name = "Cheap Seats"
aliases = ["cheapseats.dev", "cheapseatsdev/cheapseats"]
# What the product is, so the grader can tell it apart from other things called "cheapseats".
about = "Cheap Seats is an open-source, self-hosted tool that asks AI assistants a brand's buyer questions, grades whether they recommend the brand, and builds a ranked list of fixes."
# Who asks the questions. The answerers are told they're answering this person's question.
# Changing `about` or `audience` changes how answers are asked or graded, so every trend restarts.
audience = "a product marketing lead"

• aliases are other names the grader counts as the brand.
• audience is who asks the questions, for example “a software developer”; the answerers are told so.
• facts is a list of public claims the analyst may build fixes on.

brand.about and brand.facts are sent to the grader and analyst, so keep anything private out of them. Changing about or audience restarts every trend.

Sites

The brand’s websites. Pages on them count as its own; the fix list puts each fix on the site that suits it and assigns it to that site’s owner; each run shows which sites the answers drew on, including ones they never reached.

[[brand.sites]]
url = "cheapseats.dev"
kind = "marketing"
# owner = "web team"

[[brand.sites]]
url = "github.com/cheapseatsdev/cheapseats"
kind = "code"

• url is a host or host/path. A host covers its subdomains unless they’re listed themselves; a host/path covers that prefix.
• kind is one of marketing, docs, blog, community, code or other.
• owner is optional. Fixes on a site without one go to [owners].
• A bare domains list works too. Adding or changing sites never restarts a trend.

Answerers

Every assistant that answers, one [[answerers]] entry each:

[[answerers]]
id = "claude-opus"
name = "Claude Opus"
provider = "anthropic"
model = "claude-opus-5-5"
samples = { search = 2, recall = 1 }

provider decides how it answers and who pays:

• Results and rankings key on id: never reuse an id for a different model; add a new answerer instead.
• name is an optional display name.
• samples is how many answers it takes per track (search and recall) for each prompt.
• claude-code and codex only work on your own machine or a VM with the CLIs installed. cheapseats check fails on a config whose subscription answerers can’t run where it’s checked.

Shared competitors

[[competitors]]
name = "Peec AI"
aliases = ["Peec", "peec.ai"]
domains = ["peec.ai"]

Competitors the grader folds onto one canonical name in every prompt set, so share of voice counts “Peec” as Peec AI. A set’s own [[competitors]] adds to the list, and its entry for the same name wins. See prompt sets.

• domains are the competitor’s own sites, each a host (covering its subdomains) or a host/path prefix (github.com/withgauge). The run page counts pages on them as that competitor’s, in its sources, buckets and outreach list. Like the brand’s sites, domains only decide whose pages are whose: they’re read from today’s config for every run, and adding or changing them changes no trend.

Sources

The run page sorts every page the answers drew on into a kind of site: own (the brand’s sites), competitor (their domains), then ugc, review, reference, institutional or editorial from a built-in list (Metric definitions has it). Move a host to another kind here:

[sources]
ugc = ["community.example.com"]
editorial = ["medium.com"]   # instead of ugc
review = ["example.com/reviews"]

• The keys are ugc, review, reference, institutional and editorial.
• [sources] wins over the built-in list, the most specific entry wins, and a host may be in only one class.
• The brand’s sites are always own and competitors’ domains always competitor, whatever [sources] says. Changing it changes no trend.

Grader and analyst

[grader]
model = "claude-haiku-5-5"
# provider = "claude-code"   # on your Claude subscription, through the claude CLI

[analyst]
model = "claude-opus-5-5"
# provider = "claude-code"

The grader grades each answer; the analyst writes the fix list and insights, and drafts prompts for cheapseats suggest-prompts. By default both go through the Anthropic API, with ANTHROPIC_API_KEY or the proxy in CHEAPSEATS_ANTHROPIC_BASE_URL. With provider = "claude-code" they run on the local claude CLI and your Claude subscription instead; with subscription answerers too, Cheap Seats needs no API key at all (examples/cheapseats-config/cheapseats.subscriptions.toml is set up that way). The Docker image has no CLIs, so keep the API there. Switching the grader’s provider starts a new method on every trend, since answers are graded a different way.

[grader]
model = "claude-haiku-5-5"
themes = false   # skip the theme pass

After each run is graded, one more grader call groups its objections, praise and reasons for losing into a few named themes, shown on the run page. It’s on by default; themes = false turns it off. It changes no rate or trend. themes belongs in [grader], not [analyst].

Budget

[budget]
# Admission budget across every provider attempt. In-flight operations can exceed reservations.
daily_usd = 40.0
reservation_usd = 0.60
concurrency = 6

Every API call (API answers, grading, the theme pass, the fix list, prompt suggestions, search probes) reserves an allowance against budget.daily_usd in a SQLite ledger before it goes out, and settles to the real cost after. A run that would go over stops with status budget; retry it the next day.

This is an admission limit, not a provider billing cap: one call can cost more than its reservation, so set provider-side limits too. Subscription answers are recorded at $0 and don’t count against the budget.

Sampling

[sampling]
adaptive = true
max_extra = 12

Adaptive sampling gives each run of a core set up to max_extra extra answers, for the prompts whose named rate has been least certain over the last 28 days of runs asked and graded the same way. It’s all decided before the run starts, a cell gets at most as many extras as it has planned samples, and extras on API answerers have to fit in what’s left of today’s budget.

• Off by default. adaptive = true needs max_extra (1 to 1000).
• It never starts a trend over: extras don’t change the method.
• Extras count in the run’s rates like any other answer, so those prompts weigh a little more in its totals. Trends count each prompt once either way.

See How big a change Cheap Seats can detect.

Pricing

USD per million tokens; search_per_1k is USD per 1,000 web searches. An API answerer whose model has no price in [pricing] can’t run. Confirm prices against the providers’ current price lists before relying on the cost figures.

[pricing.claude-haiku-5-5]
input = 0.10
output = 0.50
cache_read = 0.01
cache_write = 0.125
search_per_1k = 10.0

Schedule and tiers

[schedule]
# UTC hour for the daily runs. Leave unset on a dev machine so it doesn't spend money.
# daily_hour_utc = 13

[tiers.core]
every_days = 1

[tiers.plg]
every_days = 7
max_samples = 1

Set schedule.daily_hour_utc and, from that hour, every core set runs every tiers.core.every_days days (default 1, daily) and every plg set every tiers.plg.every_days with one sample, one set after another. Locally it’s off unless you set it.

Weekly is usually enough: recall only moves when models change, and search with the web over days to weeks. Daily mostly buys a smoother trend line, at seven times the answers.

Owners

[owners]
# Who picks up each kind of fix. Unlisted gaps go to `default`.
default = ""
# by_gap = { documentation = "", positioning = "", "third-party" = "", product = "" }

A fix goes to its site’s owner, else to the owner for its gap here, else default.

Environment

Cheap Seats reads .env at startup; the real environment wins.