Research Worker

Status

Implemented. The Research Worker drives the framework's 8 phases against real search adapters and produces typed artifacts that the final report aggregates. The worker is the lab's primary entry point for non-trivial research questions.

Role

The Research Worker's job is to turn a research question into a structured, auditable report. Given a query, the worker:

1. Initializes a mission in the framework (tool-a-research-init).
2. Runs the candidate-gathering phase (tool-b-hotel-research for hotel-style missions, or the appropriate domain tool).
3. Runs the parallel review, price, and YouTube phases.
4. Runs the Perplexity synthesis phase (P3) when enabled.
5. Aggregates everything through the report generator (tool-h-report-generator).
6. Optionally delivers the report by email (tool-g-email-sender).

The worker does not reason about the framework's internals. It delegates every step to a domain tool. The tools themselves do the work; the worker coordinates them and handles the lifecycle (checkpointing, retry, recovery).

Architecture

The Research Worker is a thin runtime that:

• Receives a research request from the Coordinator.
• Wraps the request in a mission (mission_id + query).
• Calls the framework's CLI (research_cli.py or run_research.py) to execute the mission.
• Watches the mission's state and the artifact store.
• Surfaces the result (a report.md file) to the Coordinator.

Loading diagram…

The worker is intentionally small. All the complexity lives in the framework and the domain tools. The worker's value is its lifecycle management: it knows when to start, when to wait, when to retry, when to pause, and when to give up.

Domain tools

The worker invokes 8 domain tools from the framework. Each tool is a self-contained Python package with its own src/, tests/, skills/, and docs/.

tool-a-research-init

Initialize a research run, write task.json, manage checkpoint state.

The tool exposes three functions:

• init_research(research_id, mission): create the run directory and write task.json.
• save_checkpoint(research_id, phase, progress): persist the current state to checkpoint.json.
• load_checkpoint(research_id): read the current state.

The checkpoint format is CheckpointState (Pydantic) with fields: research_id, current_phase, completed_phases, next_action, progress, errors, updated_at.

The tool's skill (.agents/skills/research-init/SKILL.md) documents the workflow: initialize, write task, write candidates, create hotels/, create sources/, create notes/. Each step writes a checkpoint; failure writes an error checkpoint with the next recovery action.

tool-b-hotel-research

End-to-end hotel research: gather, normalize, rank, write.

The tool's SKILL describes a 3-phase workflow:

1. Search → candidates. Use web_search with terms like "[destination] 5 star hotel spa booking.com" or "hoteles [destination] 5 estrellas spa". Use web_fetch on aggregator sites. Save to candidates.json.
2. Normalize → hotels/. For each candidate, build a normalized hotel record with name, address, rating, price range, amenities, and source URLs.
3. Rank → report. Rank the candidates by score (rating, price, amenities match) and write a comparison report.

The tool supports multi-language search (English and Spanish at minimum) and is designed for travel-planning queries.

tool-c-review-research

Pull and rank reviews for a list of candidates.

The tool fetches reviews from configured sources, aggregates sentiment and topic distributions, and ranks candidates by review quality.

tool-d-photo-research

Download and organize photos for candidates.

The tool downloads photos, caches them locally, and organizes them into a stable directory structure keyed by candidate ID.

tool-e-price-analysis

Compare prices across providers and discount historical data.

The tool pulls current prices, computes historical statistics (min, max, median, trend), and produces a comparison matrix that the report can render as a table.

tool-f-video-scrape

Scrape and extract video metadata and transcripts.

The tool wraps youtube_transcript_api and yt-dlp. It finds relevant videos, extracts transcripts, and chunks them for the report.

tool-g-email-sender

Send a templated email (report delivery, alerts).

The tool's SKILL describes the workflow: confirm recipients, confirm report metadata, render the body from a template, attach the report files, send through the queue. The queue supports retries and rate limiting.

tool-h-report-generator

Format the final report from artifacts.

The tool's main class is ReportGenerator. It:

1. Loads the compiled candidate data from <research_root>/<research_id>/candidates.json.
2. Calls compile_report_data to merge all artifacts.
3. Renders the report in the requested format (html, md, or json).
4. Writes the report to the output directory.

The supported formats are exposed through the SUPPORTED_FORMATS constant.

Mission lifecycle from the worker's perspective

The worker is responsible for the mission's lifecycle. Internally, the worker follows this state machine:

Loading diagram…

Each transition writes a checkpoint. The worker can be asked at any time "what's the state of mission X?" and will read phase.json to answer.

Checkpointing

Every domain tool writes a checkpoint after each step. The checkpointing layer is tool-a-research-init's checkpoint.py:

• save_checkpoint(research_id, phase, progress): append the current step to checkpoint.json.
• load_checkpoint(research_id): read the current state.
• On failure, write an error checkpoint with the next_action field populated with the recovery step.

The worker uses the checkpoint to:

• Resume a paused mission (read state, pick up where it left off).
• Debug a failure (read the last successful step and the error message).
• Report progress to the Coordinator.

The checkpoint format is documented in Mission Lifecycle → Checkpointing.

Retry and recovery

The worker applies a default retry policy to every tool invocation. The policy is:

• 3 attempts total (1 original + 2 retries).
• Exponential backoff with full jitter: 1s, 2s, 4s, capped at 30s.
• Retry on 429 and 5xx; do not retry on 4xx (except 429).
• Do not retry on tool-internal errors (a Pydantic validation failure is not retryable; it is a bug).

If all retries are exhausted, the worker marks the mission as failed_retryable and surfaces the error to the Coordinator. The Coordinator can ask the worker to retry the mission with different parameters or accept the failure.

Parallelism

The worker runs independent domain tools in parallel when the mission allows it. The parallel set is:

• F2 (reviews) + F4 (price) + Y1 (YouTube search) — these are all in the parallel_group=1.

The worker uses a ThreadPoolExecutor with a configurable worker count (SCOUTE_PARALLEL_WORKERS, default 3). The runner coordinates so that Y2 waits for Y1, and F6 waits for everything.

The worker does not start a new mission until the current one is in a terminal state. This is by design: a single mission at a time keeps the worker's state machine simple.

Configuration

The worker is configured through environment variables. The relevant ones are:

The full list is in Environment Variables.

Performance

A typical mission with all 8 phases completes in 2-5 minutes wall-clock. The bottleneck is usually the review-and-price parallel phase (depends on adapter latency and rate limits). The YouTube phase can take longer when many candidates are involved.

The worker reports a progress percentage every 30 seconds to the Coordinator. The Coordinator surfaces the progress to the user.

Failure modes

The failure catalog is in Failure Catalog.

Relationship with the Coordinator

The Coordinator forwards research requests to the worker and receives the result (a report.md path or a summary). The worker does not initiate work independently.

The handoff is one-way: Coordinator → Worker → result. The worker does not ask the user clarifying questions; the Coordinator clarifies intent before sending the request.

The worker also reports the mission state to the Coordinator on demand. The Coordinator can ask "what's the state of mission X?" and the worker will return the contents of phase.json plus a short summary.

Relationship with the Media Agent

The Media Agent and the Research Worker were initially prototyped in a shared runtime. The architecture models them as separate logical agents because their responsibilities, tools, and risk profiles differ. See ADR-001.

The two agents do not share memory. The Media Agent has no persistent memory layer; the Research Worker has its own workspace memory for mission artifacts.

Future work

• Multi-mission parallelism. Allow multiple missions to run concurrently, isolated by research_id.
• Streaming reports. Stream the report as it is generated, instead of waiting for F6 to complete.
• Cross-mission artifacts. Allow one mission to use another mission's artifacts as inputs.
• Per-tool rate limits. Make rate limits configurable per tool, not just per adapter.
• Result caching. Cache results across missions with the same query.

See also

• Agents and Roles
• Coordinator
• Coding Assistant
• Media Agent
• Research Framework
• Mission DAG
• Mission Lifecycle
• External Providers
• Failure Catalog