research-priorities

Research Priorities (ADO)

The live research agenda. Relentlessly focused on the 10-week proof-of-value goal: a measurable, attributable discovery lift for direct-booker. Belief layer = product-thesis; market landscape = agent-discovery-market. Full agenda + assumptions live in the Quarterly Goals · June–August 2026 doc (Drive). Rule: each question must end the quarter as a published asset or a product input — “research that ends as a Google Doc is a cost.”

Current truth

Status update (2026-07-07) — what this agenda has already produced

• “Reverse-engineer Claude connector selection” — answered, three layers deep: routing measured on ~4.3k runs (claude-connector-routing, 2026-06-24); the backend registry search decoded to fact grade with a validated 88%-directional edit advisor (claude-registry-search-decoded, 2026-07-04); the browsable directory rank decoded (claude-directory-rank-decoded, 2026-07-06). Remaining from that item: weights across model rotations; the ChatGPT port.
• “Controlled before/after interventions” — answered with a split verdict: metadata edits causally move backend retrieval (≈3 ranks/keyword, n=30 diff-in-diff) but are refuted as a browsable-directory-rank lever (edit diff-in-diff CI crosses zero) — the two decoded pages carry the numbers. Memory/store-popularity/web interventions remain untested.
• “Simulating native clients at scale” — now a live instrument, not a plan: a daily frozen-prediction probe battery (~100/day ramping to 300) with next-day scoring runs on the platform since 2026-07-05; every registry edit becomes a scored natural experiment.
• Still open from this agenda: agent loyalty / memory dynamics (P1 #2 — the “smoking gun,” untested), connector segmentation (P1 #3), the four-routes mapping, the real-user prompt corpus, blind novel-keyword prediction (F1 ≥0.90 identification program), and the reasoning-taxonomy mining (approved, not built).

Scope discipline (the recurring founder tension): research is locked to Claude connectors + ChatGPT apps as the serviceable “gold standard.” Vincent pushes hard against scope creep (“don’t map every ChatGPT tool”); Elliot wants the broader surface map so they’re not out-positioned by Ora. Resolution: connector/app focus is Priority 1; broad surface mapping is Priority 2. Adding another harness (Codex/Cowork) is a large T-shirt size, especially if automated.

Priority 1 — directly informs product + client conversations (next 10 weeks)

1. Which prompts trigger tool/connector/app calls vs web vs internal answer. Action-oriented prompts → tool calls (where you must dominate); knowledge/discovery prompts → web (AEO territory). Map, across categories, what counts as “action” vs “knowledge.”
   • Discovered mechanic — Claude’s no-checkout rule: Claude has a hard-coded rule that connectors cannot directly sell products (no checkout / ecommerce connector). A real constraint on what a connector can do.
2. Agent loyalty / memory dynamics (Vincent’s “smoking gun”). Hypothesis: the single most valuable action for a brand is getting a user to connect once. Early evidence: after connecting, re-running prompts shows the picker says “use” rather than “connect.” Experiments: run hotel-category prompts with direct-booker connected (does it win all / do others vanish?); “I always want to use Direct Booker” → does it commit to long-term memory?; add a second brand the next day → does the shortlist collapse to just those two? Elliot’s counter-view: pure connect-once-wins is bad for consumers/competition; he expects rotation/randomization — agreed to test 100%. Spin-off product idea: a deep-link campaign + coupon toolkit to drive users to connect (even “pay a user $10 to connect once”).
3. Connector/app segmentation. Which connector types over/under-perform — platform vs individual brand, geography, etc. (Seeded by the Marriott / direct-booker note.)

Priority 2 — broader surface mapping (stay curious, non-blocking)

• Map what surfaces (web, internal tools, connectors) agents actually use to discover/choose/act, and how it varies by harness × company × persona × task. This is the “is there a product beyond AEO?” question (Elliot’s). Ora frames everything beyond AEO as “agent-ready web,” but Ora skews enterprise / coding-agent, not consumer — “their market, not ours.” (Lucid-type enterprise B2B is where multi-harness mapping would matter — “an Ora for the major clients.“)

Sharper framing from the Q2 goals doc (canonical):

• P1 #1 — which elements actually move appearance & ranking? Run controlled before/after interventions (descriptions, schema, metadata) on real connectors across tools, memory (<<< critical — connected + prompted-into-memory), store popularity, web. Output = the optimisation methodology + before/after case studies + score weighting + the repeatable service playbook. “The product’s reason to exist.”
• P1 — reverse-engineer Claude connector selection and roughly how its factors are weighted, from at-scale testing.
• P2 additions: the four routes any prompt can take (training-data / native tool = web-search = the AEO surface / connector call / fail), tracked per category with the prompt features that predict each; an agent-readable value endpoint (/.well-known/agent-discovery.json, llms.txt) tested as a trackable discovery surface + possible owned moat; a real-user prompt corpus from direct-booker first-party data (the “keyword research” primitive ADO lacks); and simulating native clients at scale (extends the Playwright/API-delta work) so measurement cost collapses and volume becomes a moat.

Open questions

• Is there a meaningful optimization surface beyond AEO per harness, or is AEO ~95% of it (connector = the other 5%)? (Elliot’s core unresolved question.)
• Does connect-once → always-picked actually happen, or does Claude rotate options?
• Does a web search help a ChatGPT app get recommended? (Prior research inconclusive.)
• Tighten the “lever vs surface” wording in the research questions (a lever = something you control; a surface may or may not be controllable).

Timeline

• [2026-07-07] (maintenance) Status pass — the page was three weeks stale and didn’t know its own P1 had been answered: recorded the three decoded pages, the split intervention verdict (backend yes / directory no), and that at-scale simulation is now the live daily probe loop. Remaining items restated in the status block.
• [2026-06-16] (internal discussion — post-Awaze debrief) Agenda set: P1 = tool-call triggers + agent-loyalty/memory + segmentation; P2 = cross-harness surface mapping. Locked focus to Claude + ChatGPT. Discovered Claude’s no-checkout connector rule. Kiwi.com’s ~500k Claude sessions/mo cited as market validation (see agent-discovery-market). To be organized into Linear after ingest.