Ghost Team Brain

ado-score-model

14 min read

✏️ Edit this page on GitHub

ADO Score Model

The score’s measurement rationale (why it’s defensible despite stochastic agents) is in the repo docs: measurement-methodology. This page is the durable “what the score is and why these buckets” layer.

Current truth

  • External name (working, as of 2026-06-17): “[platform] Discoverability Score” — host-scoped (e.g. a ChatGPT Discoverability Score, a Claude Discoverability Score), not “ADO score” or “AgentRank” in customer-facing copy. Internally still the ADO score / four-bucket model below. Chosen to read as MCP/tool/product-specific and to stay clearly distinct from AEO. Not fully locked — see Open questions: the name must leave room to broaden past discovery (we already track more than ranking, and want to cover discovery and used), and Elliot’s gating question “what are we going to use this score for?” should be answered before a final lock. Ties to the rebrand/brand-name question in ado-platform and 2026-06-16-q2-goals-gtm.

  • ADO score = four buckets, 25 pts each, 100 total (refined framing 2026-06-12): (1) Health endpoint — healthy checks ÷ measured checks; degrades gradually on downtime, re-earns over time; (2) Keyword metadata — keyword coverage across tool names + descriptions vs. the category’s benchmark keywords (Claude reads only the first 8 tools, in listing order, which the developer controls — scope note 2026-07-07: this is about the post-retrieval re-judge, what the model reads; it does not contradict backend retrieval, where the tools field is one bag of words with no length penalty → claude-registry-search-decoded; ChatGPT uses all tools for now and the first-8 rule is unvalidated there); (3) Tool-search retrieval — does the MCP registry actually return the connector on keyword searches for the benchmark prompts?; (4) Benchmark visibility / pick-rate — % of benchmark prompts where the connector appears in the connector picker. Buckets are interdependent, so the full data foundation must be stable before the score is presentable externally. Score is intentionally hard to max (always headroom to optimize).

  • Conversion / pick-when-found data exists but is not yet in the score (a future signal): booking.com picked ~100% when found; Brex found ~48% of the time but picked ~100% when found; Turkish Airlines found 72% but picked only 52%. High retrieval + low visibility = Claude finds the connector but doesn’t pick it. There is also a longer backend candidate list before Claude selects which connectors to surface.

  • Known brand bias: well-known brands score higher from model training data (not optimizable). Counter-signal supporting validity: Monday.com scores poorly despite its brand size.

  • Categories are the prerequisite before scores can run: every connector needs a category + keywords. 113 connectors already in the benchmark; ~200 still uncategorized. V1 plan: categorize without prompts, map to existing families, add keywords (the critical per-category output); V2 adds validated prompts later. Sequence: Claude connectors first, ChatGPT second. Status (2026-07-07): executed for Claude — the V2 taxonomy is live, public benchmark per-integration scores are published, and direct-booker was unblocked (recommendations issued 2026-07-02/03). ChatGPT remains unmeasured (platform flag off). Composites stay withheld for any connector missing a bucket, by design.

  • Category framework: canonical categories must be consistent across Claude and ChatGPT. Two tiers — family categories (e.g. finance) → use-case categories. Direct competitors = companies competing for the SAME prompts (travel splits into hotel-booking vs flight-booking); a connector can sit in multiple categories. Category-creation skill: generates 25 prompts, tests against the backend, iterates until 15 succeed, then creates the category. ChatGPT score needs every ChatGPT app in a category (keyword benchmarks) or it’s uselessly basic; plan = map ChatGPT apps onto the Claude categories (keywords already exist there).

  • Category migration discipline (2026-06-19): categories should be one shared schema across Claude connectors and ChatGPT apps, not two trees. Primary category drives score + competitor set; brands can still appear in secondary contexts. Prefer intuitive, tool-led names like “hotel search and booking” over company-level labels. Migrate prompts across refined categories instead of rebuilding from scratch so history survives. A too-broad category can be diagnosed when rank leaders vary wildly across prompts. Edge cases with connectors doing many unrelated jobs are handled one-on-one for now; assumption is ~90% of brands fit cleanly.

  • Extended / agent-readiness ADO score (coming). Beyond the four in-registry buckets, a broader score covering agent readiness — auth flows (auth.md), micropayments, and an agent-accessible website — for the world where personal agents act/book directly. Customer-facing, the score is narrated as a 6-step funnel: presence (in the registry) → name quality (signals intent) → tool names & descriptions → keyword coverage → tool-search shortlist (“group stage”) → final selection (“knockout”). Tool name weighs more in Claude — [evidenced 2026-07-04] for backend retrieval (name ⋙ tool names > descriptions) → claude-registry-search-decoded; “description weighs more in ChatGPT” is [hypothesis] — ChatGPT is unmeasured today.

  • Refined sales framing (2026-06-17): the score should work as a conversation opener not only for brands that already have a connector/app, but also for brands that lack presence. Practical progression discussed internally: agent-ready web → MCP server → AI app in a trusted registry. That progression is a GTM framing, not yet a locked scoring formula.

  • The four buckets = the public “four R’s” (locked 2026-06-23): Reachability, Relevance, Retrievability, Rank — the external framing of the four buckets, wrapped by the Measure → Score → Fix → Lift loop. Full framework + the product UI structure live in discoverability-framework; this page stays the bucket mechanics.

  • Benchmark vs custom score — two arenas, never mixed; public = benchmark-only (decided 2026-06-24, supersedes the 2026-06-23 “better of two”). Two scores exist: the benchmark (authoritative, apples-to-apples across all apps in a category, comparable, public) and the custom score (computed on the brand’s own defined prompts/topics — its evolved target market, private/in-product only). The 2026-06-23 call had said public = max(benchmark, custom); building it showed that mixes two non-comparable arenas on one surface (e.g. direct-booker’s custom 64.9 sitting below its benchmark 66.8 on the chart right under the headline). Reversed 2026-06-24: (a) public stays benchmark-only — never a custom number; (b) custom is a self-contained PRIVATE arena — the operator’s Score page re-scores the whole observed roster (focal + the competitors that show up in the brand’s prompts) on the brand’s own prompts, so headline and competitor chart are the same arena and always agree; (c) a competitor’s custom score is private to the arena owner (RLS — the competitor never sees it), so the same brand has different numbers in different arenas by design — that’s the insight, never shown publicly. Invariant: a custom-arena number never escapes its arena context. In-product UX labeling (“Benchmark” until prompts added, then the custom/arena score) stands. Full decision: 2026-06-24-self-contained-custom-arena (supersedes 2026-06-23-public-score-better-of-two).

  • Category taxonomy rules (decided 2026-06-23). AI-generated categorization (Codex/Claude Code, off description + website + tool names) is imperfect — apps landed in multiple families wrongly (e.g. restaurant/coffee ordering under both Commerce and Travel); hotel subcategories over-fragmented so direct-booker can’t benchmark against relevant hotels. Rules going forward: each brand has ONE primary category (+ optional secondaries); the score tracks the primary only; no subcategory spans multiple families; the sheet lists primary + subcategory per brand. Fix path: enrich each brand via the Perplexity API (web search + tool descriptions → a full brand + connector-capability summary) before re-proposing categories (~$30 run; Vincent providing the key). Strategy: start with broader categories and dissect over time rather than over-niche from the start (Dun & Bradstreet official mappings considered, rejected — tax categories ≠ prompt-competition categories). This is a live blocker for Direct Booker benchmarking.

Open questions

  • Bucket weighting — is even 25/25/25/25 right? (raised 2026-06-23.) The equal split (with a small 5th component) may inflate the headline relative to a brand’s real appearance rate: a brand can score ~77 while only appearing in the picker for a subsection of its target prompts, which “shouldn’t feel that high.” Lean: weight Rank (picker visibility / pick-rate) more heavily so the number tracks actual discoverability. Tactical for now (revisit weights over time as we reflect the score over time), not yet a locked formula change. Grounded in direct-booker (76.81, strong retrieval, weak picker visibility). Update (2026-06-26, unconfirmed): Vincent noted (imprecisely) the live weights are no longer equal 25s (“30/15-something”) and that Retrievability now has two sub-buckets — connector presence + MCP presence (“do you also have an MCP?”). Checked against the live scoring config (2026-07-07): FALSE as stated — the live model is four buckets, 25 pts each, with the composite withheld unless all four buckets are present. The weighting-lean itself (weight Rank more heavily) stays open as a design question.
  • Per-tool health vs surface health. The Health bucket reads surface “healthy/measured”, but a connector can pass that while individual tools fail (Direct Booker case). Claude’s dashboard exposes per-tool health — should the score consume it? It hits UX/outcome more than discovery directly. Tracked as a signal in signal-set-and-weights.
  • Headline score name — leaning “[platform] Discoverability Score”, not locked. Tensions to resolve: (a) “Discoverability” risks boxing us into discovery-only when we want to track discovery and used (the broader discipline might be framed as “Agent Optimisation” / “agent readiness” — Vincent’s lean); (b) the name should keep the company room to pivot, matching how we name the company going forward; (c) Vincent finds “ADO” under-differentiated and “tool” too small, but it must still read as MCP/tool/product-specific and feel very separate from AEO. Decide what the score is for (Elliot’s question) before final lock. Raised 2026-06-16 (Elliot + Vincent, TG). Note: this page’s slug (product/ado-score-model) is intentionally kept while the external name is unsettled — “ADO” remains the internal/discipline term, so don’t rename the page on sight; revisit only once the name locks.
  • Does the score behave sensibly across full datasets? Gate passed (2026-07-07): public benchmark per-integration scores are live on the platform. Still open from the 2026-06-29 sub-task: define A/B/C letter grades for score bands and build a bulk-score view to gut-check many brands at once before publishing a methodology page.
  • ChatGPT score design: include the keyword element (means every ChatGPT app needs a category) or ship a basic tools-only score first?
  • Should the score penalize brands with no connector / ChatGPT app so a low score reads as an action to take (e.g. Air France ~17) rather than a flat zero baseline? (Raised 2026-06-15 — would make the score a much better sales-conversation opener vs e.g. Turkish ~93, whose score is lifted partly by having a ChatGPT app. Positions the score/audit as the sales-pipeline entry point — every gap = a billable action.)
  • How should the score represent the progression from no presence → MCP server → trusted-registry AI app without collapsing a useful GTM story into a sloppy metric?

Timeline

  • [2026-07-07] (maintenance) Status pass against the live platform: closed the 2026-06-26 unconfirmed weights note (still four equal 25-pt buckets; composite withheld unless all four present — no 30/15 split, no Retrievability sub-buckets in the live score); marked the scale-validation gate passed (public benchmark scores live) and the Claude categorization prerequisite executed (V2 taxonomy; Direct Booker unblocked); scoped the first-8-tools rule to the post-retrieval re-judge (no conflict with the backend’s no-length-penalty finding); tagged the ChatGPT description-weight claim [hypothesis] (ChatGPT unmeasured).
  • [2026-06-29] (internal discussion — Ghost Team stand-up, Vincent + Elliot, dual recording) Custom score calculator is LIVE, including the competitor comparison — each client has its own prompt universe and competitors are scored on the client’s prompts (not their own), visible only to that client (“Booking scores higher on my prompts than me — why?”). Confirms 2026-06-24-self-contained-custom-arena is shipped. Elliot initially pushed back (a competitor’s score on their prompts is irrelevant) → resolved: the only meaningful comparison is everyone re-scored on your prompts. Next = methodology validation: define what an A / B / C score is, and build a way to eyeball many scores at once to gut-check the output before a public methodology page. Elliot to test via the Ghost Team demo account (Direct Booker active brand; no signup flow yet — invites only).
  • [2026-06-26] (internal prep — Vincent + Elliot, recorded by Vincent) Vincent noted (imprecisely) the live score weights are no longer equal 25s (“30/15-something”) and that Retrievability now has two sub-buckets — connector presence + MCP presence. Flagged needs-confirmation against the platform config (logged on the bucket-weighting Open question). Reinforced that the platform / canonical product pages should be the single source of truth for the factors + weightings (vs a hand-written methodology doc that drifts) — see company-memory-infra.
  • [2026-06-24] (build session, Vincent) Reversed the 2026-06-23 “public = better of two” → public stays benchmark-only; the custom score is a self-contained private arena (whole observed roster re-scored on the brand’s own prompts; competitor custom scores private to the arena owner via RLS). Resolves the incoherence of a custom headline next to a benchmark competitor chart. Full decision: 2026-06-24-self-contained-custom-arena (supersedes 2026-06-23-public-score-better-of-two). Engineering record: repo decisions-log #136.
  • [2026-06-23] (call — “Directbooker next steps”, Vincent + Elliot, recorded by Vincent) Raised two refinements off the Direct Booker diagnostic (76.81 vs Booking.com 92): (1) bucket weighting may inflate the headline vs real picker-appearance rate → lean toward weighting Rank more heavily (open, tactical); (2) per-tool health can fail while the surface check passes → candidate refinement to the Health bucket. Both logged as Open questions above.
  • [2026-06-23] (call — Vincent + Elliot, recorded by Vincent) Locked the four R’s as the public framing of the four buckets + the Measure → Score → Fix → Lift loop (full framework → discoverability-framework). Decided the benchmark vs live “better of the two” public score (default benchmark on signup → switch to live once it beats benchmark; no paying-customer signal; “Benchmark” → “Live” labeling). Set category taxonomy rules (one primary category, score tracks primary, no subcategory spans families) + the Perplexity-API enrichment re-categorization path (~$30). Flagged the hotel-category fragmentation as a Direct Booker blocker.
  • [2026-06-19] (internal discussion — Direct Booker/category sync) Category migration rules sharpened before the later re-categorization decision: use a single Claude+ChatGPT category schema, keep one primary score-driving category per brand, allow secondary contexts, move prompts across refined buckets rather than rebuilding from scratch, diagnose over-broad buckets by wildly varying rank leaders, and prefer intuitive tool-led category names. Financial-institution category was flagged as the next one to get live.
  • [2026-06-17] (internal discussion — merged Vincent+Elliot “Service offering” notes) Reaffirmed that the score should help sell to brands without a connector/app yet, not just optimize existing owners. Discussed the sales progression agent-ready web → MCP server → AI app in a trusted registry as a useful framing; left open whether and how that belongs in the scoring formula itself.
  • [2026-06-16] (internal discussion — Elliot + Vincent, TG) Naming the headline score. Decision-in-progress: name it the “[platform] Discoverability Score” (host-scoped) rather than “ADO score”/“AgentRank” externally. Reasoning: must (a) stay clearly separate from AEO (AEO optimises text to be cited; ours optimises tools/products to be invoked), (b) leave room to pivot/broaden — we already track more than ranking and want discovery and used, so the name shouldn’t over-commit to discovery, and (c) read as MCP/tool/product-specific. Open tensions: Vincent finds “ADO” under-differentiated and “tool” too small, floats “Agent Readiness Score” / “Agent Optimisation” as the broader discipline framing; worry that “Discoverability” boxes us into discovery-only. Elliot’s gating question before locking: what are we going to use this score for? Ties to the open rebrand question in ado-platform and brand/name in 2026-06-16-q2-goals-gtm.
  • [2026-06-16] (call) Walked the score as a 6-step funnel for awaze; flagged the extended agent-readiness score (auth.md, micropayments, agent-accessible site) as coming.
  • [2026-06-15] (internal discussion — Vincent + Elliot retro) Product idea: make the score penalize a missing connector/ChatGPT app so it feels like an action, not a baseline (Air France ~17 vs Turkish ~93, the latter lifted partly by a ChatGPT app). Frames the score/audit as the sales-pipeline entry point. See 2026-06-15-offering-pricing-fundraise.
  • [2026-06-12] (internal discussion — company-brain session) Score framing refined to the four 25-pt buckets. Surfaced: Claude reads only the first 8 tools in dev-controlled listing order (ChatGPT uses all, first-8 unvalidated); conversion/pick-rate data tracked but not yet scored; brand bias acknowledged (Monday.com low despite size → supports validity). Categories confirmed as the gating prerequisite: 113 benchmarked, ~200 uncategorized; v1 = categorize + keywords, no prompts; Claude first then ChatGPT.
  • [2026-06-10] (internal discussion) Score components defined; agreed scale-validation precedes any external presentation.
  • Older product/build history pre-split lives in ado-platform Timeline.

Graph View

Backlinks