Tool-description optimisation (Critical · in motion) — MCP-specific
Sub-bet of product-thesis. Priority: Critical. Stage: Selection.
The question
What is the best way to optimise tool descriptions for discoverability/selection? Most descriptions today read like marketing copy or API docs — both human-facing genres. Agents parse differently, so agent-optimal descriptions can look strange to humans, creating an internal-review problem operators haven’t noticed yet.
Current working answer
[hypothesis]Tool descriptions are SEO metadata for the agent era — not static artefacts; they degrade as models update, competitors improve, and prompt patterns shift. Iterate continuously and measure each change.- [evidenced 2026-06-14] Patterns that outperform: “Use this when…” framing with synonyms + context signals beats generic descriptions in the same category (ADO paper v3 eval corpus → product-thesis §Source).
- [evidenced 2026-06-14] Proof point: one clause (“REQUIRED: No strong match fallback”) → +5.85% task-success lift (Statista A/B; ADO paper v3 → product-thesis §Source).
- Scope boundary from claude-registry-search-decoded: on Claude, tool descriptions buy Rank (the post-retrieval re-judge), not Retrievability — the backend doesn’t index them.
- The open craft question: is the right output a description a PM would approve, or one that performs measurably better and needs a separate review process? (SEO analogue: meta-descriptions diverged from human copy over time.)
What would settle it
- A measured playbook of description patterns with per-category lift, stable across model rotations.
Running test: 3,000 MCPs continuously measured for description impact on selection + performance.(Corrected 2026-07-07: no such continuous description-impact test exists. The live continuous instrument is the daily registry-probe loop (claude-registry-search-decoded) — which measures retrieval, where tool descriptions are refuted as a signal. Description impact on selection — the post-retrieval re-judge — still needs its own harness; a proposed test, not a running one.)
Open
- How much does agent-optimal diverge from human-readable, and does that force a separate review workflow?
- Which description gains are durable vs model-version artefacts (ties to signal-set-and-weights)?
Timeline
- [2026-07-07] (maintenance) Corrected the phantom “3,000 MCPs continuously measured” running-test claim — no such harness exists; scoped what the live daily probe loop does and does not measure (retrieval yes, description-driven selection no).
- [2026-07-06] (maintenance) Retrofitted to the epistemic-status standard (2026-07-06-epistemic-status-standard): working-answer claims tagged; paper-derived proof points linked via the thesis §Source; added the Claude Rank-not-Retrievability scope boundary;
confidence:frontmatter retired. - [2026-06-14] (setup) Seeded from the paper’s Section H research agenda (Critical / in motion).