discovery-analysis-method

Discovery analysis method (how we decode a selection)

The investigative method, distinct from its neighbours: signal-set-and-weights is what signals matter; ado-score-model is how we grade; this page is how we find out WHY a given selection happened — the evidence engine that feeds both. It is the internal Decode step of the Measure → Score → Fix → Lift loop (discoverability-framework); over time it becomes the brain of the Fixes / recommendation engine.

Current truth

• The core bet: observation tells us what correlates; interrogation tells us why. Run data shows that a brand was/wasn’t picked; asking the model directly — “what in this prompt made you not surface X?” — is the highest-insight probe we have. It’s the one thing that turns a correlation into a candidate cause. Manual and per-prompt today; automatable over time.
• Three layers of analysis (the WHAT):
  1. Run artifacts — observational (“what happened”). Tool calls + their responses, connection-picker / selector state, raw response, and (UI-tier) DOM snapshot, network log, console log, screenshots. This is the ground truth of a run — see architecture (intelligence: runs/turns/responses/artifacts).
  2. Model reasoning — stated (“why it says it did it, in-band”). The model’s own thinking/reasoning text emitted during the run. [evidenced 2026-07-05] Now captured as a clean, queryable DB attribute: ~76% of succeeded Claude runs carry visible reasoning (~430 runs/day, median ~250 chars; ~2,400 codeable runs and growing) — capture pipeline in the product repo. Corpus starts 2026-06-22; earlier runs are unrecoverable (the artifacts were never captured).
  3. Model interrogation — elicited (“why, asked after the fact”). A post-hoc, multi-step Q&A back to the model about a specific prompt’s choice. The most insightful and the hardest to automate (it’s prompt-specific, not templatable). Run it on a neutral / fresh, no-memory account so prior usage/memory doesn’t contaminate the answer.
• Aggregate analysis — the goal-prompt. Feed the model the full dataset of runs and ask it to loop until it surfaces the strongest patterns driving discovery vs non-discovery → a prioritized hypothesis view. Complements the per-prompt interrogation with cross-run pattern mining.
• Maturity ladder (the HOW — manual → automated):
  1. Manual, per-prompt interrogation on a neutral account.
  2. Extract model reasoning as a clean DB attribute so it’s queryable, not re-derived each time. ✅ Done 2026-07-05 (76% coverage — see Layer 2 above).
  3. Goal-prompt pattern-mining across all runs → ranked hypotheses. ← Where we are now: an LLM-coded reasoning taxonomy (versioned coder prompt, codes promoted/killed by correlation with observed behavior) is approved and budgeted, not yet built.
  4. Folds into the automated Decode step and the Fixes recommendation engine (prioritize changes across organic / connected / tool-level signals).
• Why this is a moat, not just a method. Organic / pre-connection selection is only observable by running live client sessions (invisible in host logs) — see agent-discovery-market. Interrogating why a live selection happened compounds that: it’s evidence rivals working off logs cannot produce.

Open questions

• Faithfulness. Is the model’s stated/elicited reasoning a true account of its selection mechanism, or a plausible post-hoc rationalization? Models can confabulate reasons — interrogation output is a hypothesis source, not proof; cross-check against held-one-signal-at-a-time experiments before treating a reason as a driver.
• Automating per-prompt interrogation when each probe is prompt-specific and conversational.
• Neutral accounts at scale — how to provision/rotate fresh no-memory accounts cleanly.
• Where the line sits between this (Decode / explain) and the Fixes recommendation engine that consumes it.

Timeline

• [2026-07-07] (maintenance) Layer-2 status flipped: model reasoning is now a clean DB attribute (76% of succeeded Claude runs, corpus from 2026-06-22 — the ⚠️ was fixed by the reasoning-trace capture work). Maturity-ladder marker moved from step 1 to step 3 (goal-prompt mining / coded taxonomy: approved, not yet built).
• [2026-06-23] (call — “Directbooker next steps”, Vincent + Elliot + Vincent’s direction to capture it) Named and homed the analysis method as its own concept. Distinguished the three layers (run artifacts / stated reasoning / elicited interrogation) + the goal-prompt aggregate, and the manual→automated ladder. Triggered by the direct-booker diagnostic (strong retrieval, weak picker visibility → interrogate Claude prompt-by-prompt on a neutral account). Method detail moved here from signal-set-and-weights, which now points to this page.