2026-06-24-self-contained-custom-arena

Decision: self-contained custom arena + public stays benchmark-only

Set in the 2026-06-24 build session (Vincent, engineering). Resolves an incoherence that surfaced while building the dual score, and reverses the public half of 2026-06-23-public-score-better-of-two. (Reverses a joint Vincent+Elliot call decision — worth a quick confirm with Elliot.)

Current truth

• Two arenas, never mixed on one view. The benchmark score (shared category, apples-to-apples across all brands, comparable) and the custom score (the brand’s own prompt universe — its evolved target market) measure different things over different denominators. A view shows one arena, never a custom number next to a benchmark comparison.
• Custom = a self-contained private arena. When a brand tracks custom prompts, the operator Score page re-scores the entire custom roster — the focal brand and every observed competitor (the brands that actually show up in that brand’s prompts’ connector pickers) — on the brand’s own prompts. So “how do I rank vs Booking/Expedia on the prompts I care about, over time” is apples-to-apples within the operator’s arena. The headline and the competitor chart are the same arena, so they always agree.
• Custom-arena scores are PRIVATE operator intelligence. A competitor’s custom score is owned by the arena owner (organization_id = the operator, brand_id = the competitor); DB row-level security shows it only inside that operator’s dashboard — the competitor never sees it. The same brand legitimately has different numbers in different arenas (its public benchmark vs its score on someone’s prompts). That’s the insight, not a bug. There is no single “custom score” for a brand — it’s per-arena.
• Public stays benchmark-only. The public marketing site shows the benchmark score for every brand (the one comparable, public scorecard) and never a custom-arena number. This reverts the 2026-06-23 “public = better of (benchmark, custom)” rule.
• The invariant that makes it non-confusing: a custom-arena score never escapes its arena context — only ever shown inside the owning operator’s dashboard, always arena-labelled, never public, never presented as a competitor’s standalone score. Public = the standard test everyone takes; custom arena = re-grading everyone on the questions YOU care about.

Why / alternatives

• Why: the 2026-06-23 “better of two” + a benchmark competitor chart produced an incoherent screen — e.g. direct-booker’s custom headline (64.9) sitting below its own benchmark line (66.8) on the chart right under it. Mixing arenas on one view is the root flaw; labelling can’t fix it. The fix is one-arena-per-view.
• Why public stays benchmark: a custom number is over a private prompt set, not comparable to other brands — so it has no business being the public, comparable scorecard (the same incoherence at the public level).
• Rejected — keep competitors on benchmark while the focal shows custom (the first instinct): that is the mix that caused the incoherence. Rejected.
• Rejected — public = max(benchmark, custom) (the 2026-06-23 rule): reintroduces a private-arena number on the public surface. Reverted.

Consequences / risks

• Score volume grows by roster size: every custom-tracking org re-scores its whole observed roster → ~(orgs with custom categories) × (~10–17 competitors) × integrations × daily. ~millions of rows/year at 1000 orgs. Schema is ready (org-scoped grain + RLS already isolate competitor rows; no ownership constraint; append-only; no migration). The real cost is compute — controlled by scoring only active custom categories, idempotency (skip unchanged), the recommendation bucket already rolled up for all roster brands, and reusing each competitor’s integration-level health/audit; only the retrieval bucket is marginal.
• Honest downside shown, not hidden: in its own arena a brand may rank below the giants (Booking/Expedia are huge) — that’s a real, useful signal (“on your own prompts you’re still behind Booking”).
• GTM/marketing unaffected by the volume — public is still one benchmark number per brand.

Timeline

• [2026-06-24] (build session, Vincent) Decided. Engineering record + the producer/resolver mechanics: repo docs/src/content/docs/build/decisions-log.mdx decision 136 and docs/drafts/ (custom-arena build plan). Mechanics layer: ado-score-model.
• [2026-06-23] (prior, now superseded for the public half) 2026-06-23-public-score-better-of-two set public = better-of-two; the custom-vs-benchmark mixing it implied is what this decision corrects.