External Providers

Status

Implemented. All adapters are operational. The token pool is in production use for the Perplexity integration. The request journal is enabled for all paid adapters.

Purpose

The research framework delegates all external communication to adapters. The goal is:

• Keep the framework's core (orchestrator, phases, schemas) provider-agnostic.
• Make the cost, the rate limits, and the failure modes explicit.
• Provide a complete audit trail of every external call.

This page is the canonical reference for the adapter layer and the cross-cutting policies (token pool, request journal, retry, failover). The framework as a whole is in Framework. The audit and security models are in Audit Model and Security Principles.

Adapter Contract

Every adapter implements (or wraps) a small, stable interface. The minimal contract:

def search(query: str, *, limit: int = 10, **opts) -> list[Source]:
    """Return a list of Source objects matching the query."""

For specialized adapters (YouTube, Perplexity), the contract is extended but always returns objects from scout/schemas/sources.py (or a typed equivalent).

The Source schema:

Returning typed Source objects is the contract that makes the rest of the framework provider-agnostic.

Adapter Inventory

The Perplexity adapter is documented in detail below. The search adapters share the same retry and rate-limit policies; the only thing that differs is the auth and the cost model.

Token Pool

The Perplexity integration is the only one that uses a token pool. The pool exists because the lab has multiple Perplexity accounts (for capacity and redundancy) and because the integration was originally built around a sk-cp-... token plan that has per-account rate limits.

Pool Configuration

The pool is configured by token_pool_config.json (or token_pool_config.example.json for the template):

{
  "providers": {
    "perplexity": {
      "tokens": [
        {"name": "primary",   "value": "pplx-XXXX", "weight": 3},
        {"name": "secondary", "value": "pplx-YYYY", "weight": 2},
        {"name": "fallback",  "value": "pplx-ZZZZ", "weight": 1}
      ],
      "strategy": "weighted-round-robin"
    }
  }
}

The strategy can be round-robin, weighted-round-robin, least-used, or failover. The default is weighted-round-robin.

Pool Runtime

The token pool is held in memory. On every request:

1. The pool selects a token based on the strategy.
2. The adapter makes the request with the selected token.
3. The pool records the request and the response.
4. If the request fails with a 401 or 403, the pool marks the token as quota_exhausted and selects the next one. The request is retried with the new token.
5. If all tokens are quota_exhausted, the pool returns an error and the phase is marked as failed_retryable.

The pool also records:

• The total requests per token.
• The total errors per token.
• The last error per token.
• The last successful request per token.

The pool's state is exposed as a JSON file under {framework-root}/runtime/token_pool_state.json. The Coordinator can be asked to show the current state.

Pool Security

The pool file is the most sensitive file in the framework. It is:

• Stored with permissions 0600 (owner read/write only).
• Excluded from any backup that is uploaded off-machine.
• Re-loaded on every process restart (the file is the source of truth, the in-memory state is a cache).
• Audited via the request journal (every request records which token was used, by name, never by value).

The value of a token is never written to any log, journal, or artifact. The token's name is what the journal records.

Request Journal

Every adapter call is recorded in a single append-only journal. The journal is {framework-root}/runtime/request_journal.jsonl. Each line is a JSON object:

{
  "ts": "2026-06-10T20:15:34.123Z",
  "mission_id": "hotel-barcelona-2026-09",
  "phase": "F1",
  "adapter": "tavily",
  "action": "search",
  "token_name": null,
  "request": {
    "query": "boutique hotel Barcelona Eixample",
    "limit": 10
  },
  "response_summary": {
    "status": 200,
    "items": 10,
    "duration_ms": 412
  },
  "result": "succeeded",
  "error": null
}

The journal is append-only. Entries are never updated or deleted. Rotation is by file: when the journal exceeds a configured size (SCOUTE_JOURNAL_MAX_BYTES, default 50 MB), it is renamed to request_journal.<timestamp>.jsonl and a new file is started.

The journal is the foundation of the audit model. The Coordinator can be asked to:

• Show the last N requests for a given mission.
• Show the requests for a given token.
• Show the requests that failed.
• Replay a request (debug aid).

The journal does not store request bodies that might contain personal data. If an adapter accepts a free-form input, the journal stores a redacted version (e.g., query keywords but not the full text).

Retry Policy

All adapters share a default retry policy. The policy is applied by a shared retry helper that wraps the adapter call.

The retry helper uses exponential backoff with full jitter. It does not retry on 4xx errors other than 429.

Failover

Failover is the act of switching from one provider to another when the primary is unavailable. The framework implements failover at two levels.

Adapter-Level Failover

Each phase declares an ordered list of adapters. The phase calls the first adapter; if it returns failed_retryable after exhausting its retries, the phase calls the next adapter.

Example for F1:

Adapters: tavily → duckduckgo → google

If Tavily returns a 5xx, F1 retries Tavily twice, then falls back to DuckDuckGo, then to Google. If all three fail, the phase is failed_retryable.

Provider-Level Failover (Perplexity Only)

The token pool implements provider-level failover. When a Perplexity request fails because of a token issue (401, 403, 429 from a quota perspective), the pool switches to the next token. The request is retried transparently.

This is in addition to the regular HTTP retry. The full flow for a Perplexity request is:

1. Pick a token from the pool.
2. Make the request.
3. If the response is 200, return.
4. If the response is 429 or 5xx, retry with the same token (exponential backoff).
5. If the retries are exhausted, mark the token as quota_exhausted (for 429) or error (for 5xx) and pick the next token.
6. If all tokens are quota_exhausted, fail the phase.

Rate Limits

Each adapter has a per-minute request cap. The cap is enforced by a per-adapter token bucket:

The cap is configurable per deployment. The cap is enforced before the request, not after, so the framework does not waste requests on rate-limited providers.

Cost Tracking

For paid adapters (Tavily, Perplexity), the framework records the cost of every request in the journal. The cost is the product of the response's reported usage and the provider's published rate.

The journal's cost field is populated when the adapter returns usage information. The framework exposes a /api/cost-summary endpoint that aggregates costs by mission, by adapter, and by token.

See Also

• Framework
• Mission DAG
• Mission Lifecycle
• Audit Model
• Security Principles
• Failure Catalog
• Environment Variables