Signal set & weights (Critical · in motion)
Sub-bet of product-thesis. Priority: Critical. Stage: Findability/Selection.
The question
What is the complete set of signals an agent uses to select a tool, and how are they weighted? SEO eventually mapped its signal set and approximate weights even without platform disclosure; ADO has not.
Current working answer
- Method: hold the tool fixed, vary one signal at a time (AEO/citability, registry presence, training-data awareness, system-prompt configuration, web-retrieval behaviour, user memory) across clients and models.
- Early read: weighting appears category-dependent and model-version-sensitive — i.e. stable global weights may not exist. Confidence is low precisely because the answer may be “there is no stable answer.”
- We should track the signal set as a working exhaustive inventory, even before we can prove weights. The near-term asset may be the taxonomy itself plus a repeatable way to test each signal family.
- Organize the inventory under the four R’s (2026-06-23): the signals group beneath discoverability-framework’s Reachability / Relevance / Retrievability / Rank. Registry presence belongs under Retrievability (it’s about being retrievable), not as a separate R. The page also wants a practical flat view (rank, memory, tooling, registry, …) alongside the R-grouped one.
Evidence-status table
The scannable index of every candidate signal, tagged per the epistemic-status standard (2026-07-06-epistemic-status-standard): [fact] = reproducible on demand · [evidenced] = run-level proof, linked · [hypothesis] = plausible, untested · [refuted] = tested negative. Every [hypothesis] row’s settling test is the page-wide method (§How we test these signals: one-signal-at-a-time controlled variation). “Operational prerequisite” is a scope note (a gate that may not be a selection signal), not a maturity state.
Data source marks whether a signal is directly exposed by a host/platform. Official Claude Observability metric means Anthropic exposes the datapoint for at least some published connectors; it does not mean we have evidence that Claude uses it for surfacing or ranking.
| Signal | Primary R | Data source | Evidence status |
|---|---|---|---|
| Keyword coverage (names + descriptions vs benchmark prompt terms) | Relevance + Retrievability | Our run artifacts + keyword analysis | [evidenced 2026-06-23] — improves both; and we now know the target: Claude compresses prompts to ~3–4 capability nouns (~half inferred/synonyms), strips dates/prices/actions, amplifies brand terms — see claude-connector-routing. Causal for retrieval (2026-07-04): adding a searched keyword to indexed text ≈ 3 ranks better (n=30 diff-in-diff), and the edit-lift advisor validated 88% directional on 118 real edits → claude-registry-search-decoded |
| Registry presence / eligibility (host returns you on tool-search) | Retrievability | Host registry / search behavior | [fact 2026-07-04] (Claude) — retrieval = literal text match on exactly 4 fields (name ⋙ tool names > shortDescription ≈ description), no stemming, cap 10 with a relevance threshold → claude-registry-search-decoded |
| Tool-search retrieval (registry actually returns the connector) | Retrievability | Our run artifacts | [evidenced 2026-06-24] — backend rank predicts picker survival (rank-1 shown 74% / picker-1 44%, decaying monotonically to ~16%/0.8% at rank 10); the backend’s own match/rank mechanics now decoded → claude-registry-search-decoded |
| Tool name (phrasing, specificity, intent match) | Relevance | Tool schema + our run artifacts | [evidenced 2026-06-24] — the model inspects each candidate’s tools first; specific task-matched tool names drive the post-retrieval re-rank, generic (“search”) tools fall back to the description → claude-connector-routing |
| Tool description (synonyms, “use this when…”, use-case coverage) | Relevance | Tool schema + our run artifacts | [evidenced 2026-06-24] — the model re-judges from content and discards backend rank order; description is the fallback when tool names are generic → claude-connector-routing |
| Category assignment | Relevance | Internal taxonomy / host registry metadata | [refuted 2026-07-04] for backend retrieval (Claude) — category/familyName is not indexed; isolation probes of literal category names return zero members → claude-registry-search-decoded. (Still [hypothesis] for other surfaces, e.g. directory browsing.) |
| Prominence (per-connector latent in backend ranking) | Retrievability + Rank | Inferred from probe corpora | [evidenced 2026-07-04] (Claude) — backend score ≈ TextScore + a stable per-connector constant (~97.7% order-consistent, account-independent); earned not writable; NOT explained by directory rank/trending/active users/tool calls (~0 corr); gates top-10 membership on crowded keywords → claude-registry-search-decoded |
| Tool descriptions as a retrieval signal | Retrievability | Tool schema | [refuted 2026-07-04] for backend retrieval (Claude) — tool descriptions and params are NOT searched by search_mcp_registry; they matter at the post-retrieval re-judge (Lever 2), not retrieval. Precision matters: keyword work in tool descriptions buys Rank, not Retrievability → claude-registry-search-decoded |
| Tool ordering / position bias (hosts read a subset of tools) | Rank | Tool schema + our run artifacts | [evidenced 2026-06-24] — backend rank is a strong prior but NOT final order; the model overrides it by re-judging tool content → claude-connector-routing |
| Picker visibility / shortlist inclusion | Rank | Our run artifacts | [evidenced 2026-06-24] — quantified shown-rate by backend rank (claude-connector-routing) |
| Pick-rate when visible | Rank | Our run artifacts / host analytics where available | [evidenced 2026-06-24] — quantified by backend rank on the ~4.3k-run corpus: given shown, a backend-#1 converts to picker-#1 ~59% (44.0/74.3), decaying monotonically to ~5% at backend rank 10 (0.8/15.7) → claude-connector-routing; picked-when-found extremes tracked in ado-score-model (Booking ~100%, Brex ~100%, Turkish 52%) |
| Competitive crowdedness (how many connectors contest the same prompt / keyword / category) | Rank + Retrievability | Our run artifacts + category roster | [hypothesis] (candidate weight) — not an additive signal but a normalizer: it scales how hard retrievability (making the returned set) and rank (surviving the picker re-rank) are, by field size; looks easy now only because categories are sparse (~5 travel apps), scales hard as the registry fills. The fill rate itself is now measured: [evidenced 2026-07-02] 96 connectors added / 13 removed / 62 metadata edits in one 30-day window (~3 adds/day; deliberate registry-SEO already observed in the wild) → claude-registry-search-decoded |
| Training-data awareness / brand recall (“boost” signal) | Rank | Our run artifacts + model behavior inference + direct probes | [evidenced 2026-06-24] (confounded) — strong brands systematically promoted above their backend rank (HubSpot #3→picker-1 ×90, Asana #3→#1 ×89, Expedia #5→#1 ×77) → claude-connector-routing; not yet isolated from “better content”. Direct self-report probe (2026-06-26) adds anecdotal support — see note below |
| Prompt phrasing / search-trigger sensitivity | cross-cutting (Retrievability gate) | Controlled prompt variants / run artifacts | [hypothesis] — Vincent’s (2026-06-26): single-word nuances can swing whether a connector search even fires; contested by AEO wording-invariance research Elliot shared (to review) |
| User memory / prior usage | Rank | Controlled test setup | [refuted 2026-07-01] for backend retrieval (Claude) — results are byte-identical across accounts and install/connection states (a needs_reconnect connector still ranks #1); the retrieval stage is unpersonalized → claude-registry-search-decoded. Still [hypothesis] for the picker/invocation stages (we test on free / no-memory accounts to neutralize) |
| System-prompt configuration / host rules | Rank | Inferred host behavior | [hypothesis] |
| Health / uptime | Reachability | Health checks + host dashboard where available | [hypothesis] — operational prerequisite rather than a proven selection signal (see per-tool-health note below) |
| Authentication friction | Reachability | Host metadata + our run artifacts | [evidenced 2026-06-24] — authless connectors survive into the picker more (shown 43% vs 33% auth-required); a selection signal, not just a gate → claude-connector-routing |
| Required tool inputs / argument gating | Rank (invocation gate) | Tool schema + run/interrogation observations | [hypothesis] — a tool whose schema requires args (e.g. dates) can’t be called until the user supplies them, so it’s skipped earlier in the funnel; fewer required inputs = earlier invocation. Observed 2026-06-26 (see note) |
| Freshness / update responsiveness | Reachability | Host registry / review timing | Split by surface: [evidenced 2026-07-04] for backend retrieval (Claude) — metadata edits are re-indexed fast and move rank (Supermetrics absent→#1 overnight; ≈3 ranks/keyword causal, n=30) → claude-registry-search-decoded; [refuted 2026-07-06] as a directory-rank rebound lever (edit diff-in-diff CI crosses zero) → claude-directory-rank-decoded |
| Directory rank / trending status | Rank + Retrievability | Official Claude Observability metric | [refuted 2026-07-04] for backend rank (Claude) — ≈0 correlation with search_mcp_registry ordering (it is not the prominence latent) → claude-registry-search-decoded. On the directory-browse surface itself, trending is also weak: the trending boolean is dead (0 true) and trending_score only weakly, confoundedly precedes rank → [refuted 2026-07-06] as a clean lever → claude-directory-rank-decoded. |
| Active users (30d) | Rank / outcome | Official Claude Observability metric | [refuted 2026-07-04] for backend rank (≈0 corr with registry-search ordering → claude-registry-search-decoded); still [hypothesis] for other surfaces |
| Tool-call volume (30d) | Rank / outcome | Official Claude Observability metric | [refuted 2026-07-04] for backend rank (≈0 corr with registry-search ordering → claude-registry-search-decoded); still [hypothesis] for other surfaces |
| Connector age / time-on-market | Rank | Catalog snapshots (listing/launch date) | [evidenced 2026-07-06] (Claude directory-browse surface) — the novelty boost is real and large: new connectors debut top ~4%, decay with a ~2–3 day half-life to a “new-connector shelf”, a fixed 14-day “New” badge holds rank up (decay ~4× faster once it drops), and net of popularity a <7-day connector ranks ~275 positions above an equally-popular mature one → claude-directory-rank-decoded. The durable reputation boost half stays [hypothesis]. (This is the directory listing rank, not backend search_mcp_registry.) |
| Disconnect rate | Reachability | Official Claude Observability metric | [hypothesis] / operational prerequisite — richer than binary health; no proven surfacing effect yet |
| Error rate | Reachability | Official Claude Observability metric | [hypothesis] / operational prerequisite — plausible trust/reliability penalty; no proven surfacing effect yet |
| Latency distribution (p50 / p95 / p99) | Reachability / outcome | Official Claude Observability metric | [hypothesis] — slow tools degrade UX and could become a selection penalty; unproven |
| Product-specific usage/reliability (Claude / Claude Code / Claude Cowork) | host-specific Rank + Reachability | Official Claude Observability metric | [hypothesis] — signal weights may differ by client surface; Direct Booker already suggests Cowork behavior may differ from Claude |
| Observability eligibility / enriched-dashboard access | cross-cutting | Anthropic product access state | [hypothesis] (meta-signal) — if only some connectors receive these metrics, eligibility may imply a trust/scale tier, but that is not evidence of ranking impact |
| Off-registry web evidence / AEO signals | cross-cutting | Web / external evidence | [hypothesis] |
| Price / commercial fit | cross-cutting | Inferred from product/market context | [hypothesis] |
| Invocation success / reliability | (outcome) | Host analytics / run outcomes where available | [hypothesis] |
| Result-quality feedback loops | (outcome) | Host analytics / run outcomes where available | [hypothesis] |
Signal detail (where it adds beyond the table):
- Keyword coverage — match against benchmark prompt terms and host retrieval behaviour. Now evidenced (2026-06-23): improves both retrievability and relevance.
- Health / uptime — whether the tool is healthy enough to stay trusted/eligible. Surface “healthy” can mask tool-level problems (2026-06-23): a connector can pass the surface health check while individual tools are failing (Direct Booker had API problems on specific tools). The Claude dashboard exposes per-tool health — a useful diagnostic signal. Per-tool failure technically shouldn’t lower discoverability directly, but it degrades UX/outcome (whether the tool actually gets used and re-used), which is the thing the score ultimately cares about.
- Training-data awareness / brand recall — prior model familiarity with the brand/tool; worth measuring as a “boost” (e.g. how training-data-aware is Direct Booker in Claude?). Direct probe (2026-06-26): Claude, asked with no tools / training-data only, knows essentially nothing about Direct Booker vs a rich Booking.com picture (history, business model, market position), and self-reports a “strong familiarity bias towards booking.com purely because I’ve seen it before” (adding that familiarity “mostly isn’t” how selection should work). Self-reported and not corpus-grade, but consistent with the measured brand “boost” — still confounded with “better content”.
- Competitive crowdedness — how many connectors compete for the same prompt / keyword / category. Best modeled as a candidate weight (normalizer), not an additive signal: it scales the difficulty of Retrievability (a crowded keyword returns more candidates, so making the returned set is harder) and Rank (more candidates to beat in the post-retrieval re-judge / picker). Low-stakes today because categories are sparse — Direct Booker wins direct booking partly because only ~5 travel apps compete — but becomes a dominant factor as the registry scales to thousands of rivals per category. A score that ignores it will look easy to max now and get harder as the field fills, so the same raw rank should be worth more in a crowded category than an empty one.
- Required tool inputs / argument gating — a connector whose tool requires arguments can’t be invoked until the user provides them, so it loses the earliest, lowest-information moment of the funnel. Observed via direct-booker (2026-06-26): most OTA hotel connectors require dates, so on a dateless first prompt the model skips them — Theresa saw Claude/ChatGPT try a dates-required connector (Last Minute), fail, then fall back to Booking.com (no dates required). Implication: fewer required inputs = earlier invocation / top-of-funnel entry — a possible discovery lever (DB is weighing a dateless search tool). Distinct from selection: this gates whether you can be called at all yet, not your rank once eligible.
- User memory / prior usage — whether the agent/host has remembered a tool/user pattern. We currently test on free, no-memory accounts to hold this constant.
- Freshness / update responsiveness — whether metadata changes, content digests, timestamps, or change notifications cause the host to re-read/re-index/re-surface. Now split by surface (see table row): backend re-indexing is proven fast (overnight) and edit effects are causal; the directory-rank “edit rebound” is refuted. Residual open: whether updating ever penalizes an app (no penalty observed so far — 280 tool-lengthening events, mean ≈ 0).
- Claude Observability metrics — Anthropic now exposes connector-level observability for at least some published connectors: directory rank/trending, active users, tool calls, disconnect rate, error rate, latency percentiles, and product-level performance across Claude / Claude Code / Claude Cowork. These are official Anthropic datapoints, but currently candidate only as selection signals. The working hypothesis is that host-observed demand and reliability may become a trust layer for surfacing, especially if Claude prioritizes tools that are popular, fast, low-error, and product-appropriate.
How we test these signals
The investigative method that generates evidence for this inventory — run artifacts, the model’s stated reasoning, and direct model interrogation (“why did you choose what you did”, per prompt, on a neutral no-memory account), plus goal-prompt pattern-mining — lives in discovery-analysis-method. The single highest-insight probe is interrogating the model on why a brand was or wasn’t shown; that page is the home for the method and its manual→automated ladder.
What would settle it
- A reproducible signal→weight map that holds across model rotations for a given category → strong answer.
- Conversely, if per-category/per-version variance dominates, the deliverable becomes a method for measuring weights per context, not a fixed table.
Open
- Do any signals dominate across all categories (a universal prior)?
- Is training-data presence a pre-ranking gate (see persona/brand-recall probes in the thesis)?
- Which of the candidate signal families above are true selection signals vs merely operational prerequisites?
- Does freshness/update responsiveness matter only for local sync/cache invalidation, or does it ever change user-visible discovery/shortlisting in real clients?
- Which Claude Observability metrics are exposed to which connectors, and is observability access itself tied to a scale/trust/review tier?
- Do host-observed behavioral metrics (usage, growth, reliability, latency) ever feed pre-connection surfacing, or are they only developer-facing diagnostics?
- Which signals are host-specific (ChatGPT / Claude / Codex) vs general across harnesses?
- Does prompt wording materially change connector selection (single-word search-trigger sensitivity), or is selection wording-invariant as AEO research suggests? (Vincent vs the AEO paper — to test.)
- Does a pre-connected connector still surface the connection picker, or does the model call it directly without one? (Vincent: model directly calls the connected tool, no picker — says we validated this; Elliot: the picker still shows the connected tool used + others to connect.) Likely category-specific; matters for whether already-connected brands keep getting discovery exposure (platform-ecosystem incentive).
Timeline
- [2026-07-06] (maintenance) Retrofitted to the epistemic-status standard (2026-07-06-epistemic-status-standard): Evidenced→
[evidenced <date>], Decoded→[fact 2026-07-04], Candidate→[hypothesis], Tested negative→[refuted 2026-07-04]; evidence links added where rows lacked them;confidence:frontmatter retired. - [2026-07-04] (research — Vincent, registry-search project) Backend
search_mcp_registrydecoded and live-validated → claude-registry-search-decoded. Table updates: registry retrieval Decoded (4 indexed fields, no stemming, cap-10 + threshold); category assignment REFUTED for backend retrieval; tool descriptions REFUTED for retrieval (they act at the Lever-2 re-judge only); new prominence row (stable per-connector latent, evidenced); directory-rank / active-users / tool-calls tested negative for backend rank. Causal edit coefficient ≈ 3 ranks/keyword (n=30); lift advisor 88% directional on 118 real edits. - [2026-06-30] (internal discussion — Vincent + Elliot, recorded by Vincent) Framed two single-source-of-truth docs for the Direct Booker work: (1) the discovery factor set = this page; (2) the discovery flow in Claude = claude-connector-routing + the routing workflow-chart image (needs a clearly-linked home here) — with the live hypothesis backlog as a working Google Doc, not the wiki. Added connector age / time-on-market as a candidate (usage velocity, active users, and directory rank are captured by the Claude Observability rows added the same day). Confirmed the data method: ping Anthropic’s API and backwards-engineer signals (same as Alpic / “Outpick”). Reaffirmed training-data familiarity as strong-but-non-actionable (Booking picked over Direct Booker despite both recalled). Planned analyses: directory-rank↔visibility in stable categories (Elliot), connector-update→visibility-lift on benchmark data (Vincent).
- [2026-06-26] (internal prep — Vincent + Elliot, recorded by Vincent) Direct brand-recall probe (Claude, training-data only): ~nothing on Direct Booker vs a rich Booking.com picture; Claude self-reports a familiarity bias toward booking.com — anecdotal support that training-data recall feeds the brand “boost” (still confounded). Added prompt phrasing / search-trigger sensitivity as a candidate signal (does a single-word change decide whether a connector search even fires). Logged two open debates: prompt-wording invariance (vs AEO research) and whether a pre-connected connector still surfaces the picker.
- [2026-06-24] (research) Corpus-scale evidence from the Claude connector-routing report (~4.3k runs) → claude-connector-routing. Upgraded to evidenced: tool-search retrieval, tool name, tool description, tool ordering/position bias, picker visibility, training-data/brand recall (confounded), and authentication friction (authless survives 43% vs 33% — moved off “operational prerequisite”). Established that keyword coverage’s target is the compressed 3–4-noun array, not the verbatim prompt. Confidence 0.35→0.45.
- [2026-06-30] (research note — Elliot screenshot) Added a Data source column and marked Claude Observability metrics as official Anthropic datapoints but candidate-only selection signals: directory rank/trending, active users, tool-call volume, disconnect rate, error rate, latency percentiles, product-specific usage/reliability, and observability eligibility. No evidence yet that these affect surfacing; track as plausible host-observed behavioral/reliability signals.
- [2026-06-14] (setup) Seeded from the paper’s Section H research agenda (Critical / in motion).
- [2026-06-17] (research pass) Expanded the page from a pure question into a working signal inventory. Added freshness / update responsiveness as a candidate ADO signal family, with the explicit caveat that it is unproven as a discoverability driver and may turn out to be sync-only.
- [2026-06-23] (call — Vincent + Elliot, recorded by Vincent) Restructured the inventory under the four R’s + an evidence-status table; moved registry under Retrievability. Marked keyword coverage as evidenced (improves retrievability + relevance). Added the per-tool-health nuance (surface-healthy can mask failing tools; Claude dashboard shows per-tool health; hits UX/outcome more than discovery directly). Recorded the model-reasoning interrogation method (per-prompt “why shown/not shown” on a neutral no-memory account; reasoning not yet a clean DB attribute; goal-prompt-over-full-dataset as the scaled endgame) as the primary near-term driver of new hypotheses. Target: clean R-structured wiki by Friday (2026-06-26). Grounded in direct-booker (76.81 vs Booking.com 92; strong retrieval, weak picker visibility).