Research Framework

Status

Implemented. The orchestrator, the 8 phases, the 6 schemas, the 5 search adapters, and the 7 domain tools are operational. The framework is exercised by the Research Worker; the user typically interacts with it through natural-language requests forwarded by the Coordinator.

Purpose

The Research Framework answers multi-step research questions ("find hotels in Barcelona for next September under 200 EUR with good reviews") by:

1. Parsing the request into a structured Task.
2. Running a deterministic phase DAG that produces typed artifacts.
3. Aggregating the artifacts into a final report.
4. Persisting the run so it can be resumed, audited, or reused.

The framework is deterministic at the orchestration level (the DAG does not branch on the model's output) and probabilistic at the content level (adapters and synthesis can return different results for the same input). This split is the design principle that makes the framework auditable.

Architecture

The framework has four layers, each of which is testable in isolation:

Source layout

The framework lives in a single Python package with the following structure:

{framework-root}/
├── pyproject.toml
├── uv.lock
├── .agents/
│   └── skills/
│       └── scout-orchestrate/
│           └── SKILL.md
├── scout/
│   ├── __init__.py
│   ├── orchestrator/
│   │   ├── __init__.py
│   │   ├── dag.py
│   │   ├── runner.py
│   │   ├── state_store.py
│   │   ├── artifact_store.py
│   │   └── mission_runner.py
│   ├── phases/
│   │   ├── __init__.py
│   │   ├── f0_init.py
│   │   ├── f1_candidates.py
│   │   ├── f2_reviews.py
│   │   ├── f4_price.py
│   │   ├── p3_perplexity_synthesis.py
│   │   ├── y1_youtube.py
│   │   ├── y2_extract.py
│   │   └── f6_report.py
│   ├── schemas/
│   │   ├── __init__.py
│   │   ├── task.py
│   │   ├── candidates.py
│   │   ├── evidence.py
│   │   ├── sources.py
│   │   ├── reviews.py
│   │   ├── prices.py
│   │   ├── transcripts.py
│   │   └── phase_result.py
│   └── tools/
│       ├── __init__.py
│       ├── research_cli.py
│       ├── run_research.py
│       ├── run_research.sh
│       ├── perplexity_adapter.py
│       └── adapters/
│           ├── __init__.py
│           ├── tavily_adapter.py
│           ├── duckduckgo_adapter.py
│           ├── google_search_adapter.py
│           ├── youtube_transcript_api.py
│           └── yt_dlp_adapter.py
└── docs/
    ├── README.md
    ├── code-reference.md
    ├── architecture.md
    └── orchestrator/
        └── tools/
            ├── tool-a-research-init/
            ├── tool-b-hotel-research/
            ├── tool-c-review-research/
            ├── tool-d-photo-research/
            ├── tool-e-price-analysis/
            ├── tool-f-video-scrape/
            ├── tool-g-email-sender/
            └── tool-h-report-generator/

The directory names in {framework-root} are placeholders; the real location is a path known to the agent. The framework itself is portable.

Orchestrator

The orchestrator is in scout/orchestrator/. It has five components:

dag.py — Phase DAG

Defines the dependency graph between phases. Each PhaseConfig declares:

• needs: artifacts the phase requires from previous phases.
• produces: artifacts the phase writes.
• parallel_group: optional integer for grouping parallel phases.
• retry: how many times a phase can be re-executed.
• allow_partial: whether succeeded_partial is an acceptable terminal state.

PhaseDAG is the data structure; PhaseKind distinguishes sequential and parallel phases.

runner.py — PhaseRunner

Executes a PhaseDAG for a mission:

• Reads the current RunState from StateStore.
• Resolves which phases are ready to run.
• Launches sequential phases one at a time, parallel ones with a ThreadPoolExecutor.
• Captures each PhaseResult and writes it back to state.
• Re-throws PhaseExecutionError when a phase is failed_terminal.

state_store.py — StateStore + RunState

The checkpoint layer. The orchestrator persists RunState to <run_dir>/phase.json after every phase. RunState is loaded on startup so a crashed run can be resumed.

artifact_store.py — ArtifactStore

Read/write typed artifacts to disk with schema validation. Uses a per-path RLock to allow safe concurrent writes from parallel phases.

mission_runner.py — MissionRunner

The high-level entrypoint. run_mission(mission_id, query, runs_dir) is what the CLI calls. It:

• Runs f0_init to create the run directory and write task.json.
• Loads the DAG and runs phases in order.
• Returns the path to the final report.

The module also exports helpers: get_mission_state, get_mission_artifacts, and register_phase_handler (for custom phases).

Phases

A phase is a single Python function that takes (mission_id, runs_dir) and returns a PhaseResult. The framework ships with 8 phases.

The full DAG is in Mission DAG. The phase lifecycle (input, output, retry, failure) is in Mission Lifecycle.

Schemas

Schemas are Pydantic v2 models in scout/schemas/. They are the contract between layers and the validation layer for artifacts. Eight files:

PhaseStatus is the only enum and has 9 values:

class PhaseStatus(str, Enum):
    pending              = "pending"
    running              = "running"
    succeeded            = "succeeded"
    succeeded_partial    = "succeeded_partial"
    failed_retryable     = "failed_retryable"
    failed_terminal      = "failed_terminal"
    skipped              = "skipped"
    blocked_missing_input = "blocked_missing_input"
    stale                = "stale"

The full schema reference is in Data Dictionary.

Adapters

Adapters translate phase calls into provider-specific HTTP requests. They are in scout/tools/adapters/.

The Perplexity adapter lives one level up (scout/tools/) because it is more than a search adapter — it is the synthesis engine for P3. It has its own configuration, token pool, and request journal documented in External Providers.

All other adapters share a common contract:

def search(query: str, *, limit: int = 10, **opts) -> list[Source]:
    """Return a list of Source objects matching the query."""

Adapters return Source objects from schemas/sources.py so the rest of the framework is provider-agnostic.

Domain tools

The framework includes 7 domain-specific tools that wrap end-to-end workflows (e.g., "research a hotel", "research a video"). Each is a self-contained package with its own src/, tests/, skills/, and docs/.

All tools share the directory pattern from Tool Structure → Directory Layout.

Entry points

The framework has two CLI entry points:

research_cli.py — Orchestrator CLI

python -m scout.tools.research_cli run \
  --mission-id hotel-barcelona-2026-09 \
  --query "find hotels in Barcelona for September 2026 under 200 EUR with good reviews" \
  --runs-dir /tmp/scoute/runs

Returns the path to the final report.md.

run_research.py — Phase-by-phase runner

python -m scout.tools.run_research \
  hotel-barcelona-2026-09 \
  "find hotels in Barcelona for September 2026 under 200 EUR" \
  --runs-dir /tmp/scoute/runs

Runs phases one at a time with full logging per phase. Used for debugging and for the agent's step-by-step mode.

run_research.sh — Wrapper script

Bash wrapper around run_research.py that sets environment variables (SCOUTE_RESEARCH_ROOT, PYTHONPATH) and runs inside the framework's virtual environment.

Configuration

The framework is configured through environment variables and a per-tool config file.

Tool-specific configuration (e.g., Tavily key, Google cx) is loaded by each adapter from a JSON file under {framework-root}/config/.

Failure model

The framework distinguishes between retryable and terminal failures. The classification is encoded in PhaseStatus:

The orchestrator's retry policy is configurable per phase (PhaseConfig.retry). When retries are exhausted, the phase moves to failed_terminal and the mission halts.

Auditability

Every run produces a complete audit trail:

• task.json: the original query and the parsed structure.
• phase.json: the state of every phase at every checkpoint.
• All artifacts (typed by schema).
• report.md: the final aggregated output.
• The framework's own log (one per mission).

The Coordinator can be asked to show the audit trail of any mission at any time. See Audit Model.

Performance

A typical research mission (1 phase F0, 1 F1 with 20 candidates, 3 parallel phases F2/F4/Y1, 1 Y2, 1 P3, 1 F6) completes in 2-5 minutes wall-clock, depending on adapter latency and parallel worker count. The bottleneck is usually the review and price phases; the YouTube phase can take longer when many candidates are involved.

Extensibility

The framework is designed to be extended in three ways:

1. New adapter. Implement the search contract in scout/tools/adapters/ and register it in tools/research_cli.py's adapter factory.
2. New phase. Implement a run(mission_id, runs_dir) function in scout/phases/, export it from phases/__init__.py, and add a PhaseConfig in dag.py.
3. New domain tool. Create a new package under docs/orchestrator/tools/ with the standard src/, tests/, skills/, docs/ structure.

See Tool Spec and Tool Structure for the templates.

See also

• Mission DAG
• Mission Lifecycle
• Artifacts
• External Providers
• Research Worker
• Tooling Layer → Media Tools
• Audit Model
• Failure Catalog