Mission DAG

Purpose

This page is the canonical reference for the phase graph. The DAG is the structure that the orchestrator executes. The high-level framework is in Framework; the phase lifecycle (input, output, retry, failure) is in Mission Lifecycle.

Topology

Three things to notice:

• F1 is the fan-out point. All other phases depend on the candidates produced by F1. This is deliberate: it forces the framework to operate on real, normalized candidate names rather than on the raw query.
• F2, F4, Y1 run in parallel (parallel_group: 1). They share no state and can be executed concurrently.
• P3 is optional. It runs only if the enable_perplexity_synthesis flag is set in the run configuration.

Phase Reference

F0 — Init

F0 parses the raw query into a ParsedQuery (topic, location, period, budget, constraints) using a keyword-based parser. The parsed structure is what F1 uses to build candidate search queries.

F1 — Candidates

F1 is the most important phase. Its output is what every downstream phase operates on. The phase calls the configured search adapters in sequence (Tavily → DuckDuckGo → Google), merges results, deduplicates by URL and normalized title, validates each candidate against the Candidate schema, and writes the typed candidates.json to the run directory.

F2 — Reviews

F4 — Price

Y1 — YouTube Search

Y2 — YouTube Extract

P3 — Perplexity Synthesis (Optional)

P3 is the only phase that uses a paid LLM service (Perplexity) directly inside the DAG. It is opt-in because it costs API credits. The output is narrative prose that the final report (F6) can include verbatim or paraphrase. P3 is documented separately in External Providers because the adapter pattern, the token pool, and the request journal all live there.

F6 — Report

F6 aggregates all artifacts into a single markdown report with a stable structure: executive summary, candidates, reviews, pricing, video coverage, source synthesis (if P3 ran), and a deduplicated source list. The full format is in Artifacts → report.md.

Failure Status

Parallel Execution

The parallel phases (F2, F4, Y1) are launched with a ThreadPoolExecutor. The default worker count is SCOUTE_PARALLEL_WORKERS (default 3). The runner coordinates so that all parallel phases start when F1 finishes, each parallel phase writes its own artifact independently, and the runner waits for all parallel phases to reach a terminal state before starting Y2 (if Y1 was parallel) and F6.

The allow_partial flag on each phase controls whether a partial completion is acceptable:

Checkpointing

The DAG is fully resumable. After every phase, the StateStore writes the current state to <run_dir>/phase.json. If the orchestrator crashes, the next run_mission call will load the state, identify the next phase to run based on the DAG and the state, and continue from there. The full checkpointing model is in Mission Lifecycle → Checkpointing.

DAG Construction

The DAG is built in scout/orchestrator/dag.py. The default DAG is constructed at import time. Custom DAGs can be constructed by passing a list of PhaseConfig to PhaseDAG(...).

from scout.orchestrator.dag import PhaseConfig, PhaseDAG, PhaseKind
 
dag = PhaseDAG([
    PhaseConfig(name="F0", needs=[], produces=["task.json"]),
    PhaseConfig(name="F1", needs=["task.json"], produces=["candidates.json"]),
    PhaseConfig(name="F2", needs=["candidates.json"], produces=["reviews.json"],
                parallel_group=1, retry=2, allow_partial=True),
    PhaseConfig(name="F4", needs=["candidates.json"], produces=["prices.json"],
                parallel_group=1, retry=2, allow_partial=True),
    PhaseConfig(name="Y1", needs=["candidates.json"], produces=["youtube_search.json"],
                parallel_group=1, retry=1, allow_partial=True),
    PhaseConfig(name="Y2", needs=["youtube_search.json"], produces=["transcripts.json"],
                retry=1, allow_partial=True),
    PhaseConfig(name="P3", needs=["candidates.json"], produces=["synthesis.json"],
                parallel_group=1, retry=2, allow_partial=True),
    PhaseConfig(name="F6", needs=["reviews.json", "prices.json", "transcripts.json",
                                  "synthesis.json"], produces=["report.md"]),
])

Custom DAGs are not used in production but are useful for testing individual phases or for research workflows that need a different shape.

See Also

• Framework
• Mission Lifecycle
• Artifacts
• External Providers
• Research Worker