Data Flow

The data moving through the lab — what crosses each boundary, in what format, and with what guarantees.

Purpose

This page documents the data flow through the lab's architecture. It complements the System Overview and System Architecture with concrete details on the data that crosses each boundary, in what format, and with what guarantees.

Boundaries and data

The lab has 8 layers (see System Architecture → The 8-layer model). Data crosses the boundaries between these layers. Each crossing has a format, a direction, and a set of guarantees.

Boundary	Direction	Format	Guarantee
User ↔ Coordinator	Both	Natural language	Best effort
Coordinator ↔ Agent	Down	Request (JSON)	At-most-once
Agent ↔ Coordinator	Up	Response (JSON)	At-least-once
Agent ↔ Framework	Down	Method call	Synchronous
Framework ↔ Tool	Down	Function call	Synchronous
Tool ↔ Adapter	Down	Function call	Synchronous
Adapter ↔ External	Down	HTTP/HTTPS	Per adapter
Coordinator ↔ State	Both	File I/O	Atomic write
Framework ↔ State	Both	File I/O	Atomic write

The guarantees are documented in detail below.

User ↔ Coordinator

The user communicates with the Coordinator in natural language. The Coordinator parses the request, resolves the intent, and returns a response in natural language.

Aspect	Detail
Format	Natural language (UTF-8 text).
Direction	Both.
Encoding	UTF-8.
Size limit	None enforced (the Coordinator truncates very long inputs).
Guarantee	Best effort. The Coordinator may misunderstand ambiguous requests.
Logging	The Coordinator logs every request and response.

The Coordinator may return artifacts (file paths, images, etc.) along with the natural language response.

Coordinator ↔ Agent

The Coordinator dispatches a request to an agent through its public interface. The format is JSON.

Aspect	Detail
Format	JSON.
Direction	Down (request) and up (response).
Encoding	UTF-8.
Size limit	1 MB per message (configurable).
Guarantee	At-most-once (the Coordinator does not retry).
Timeout	60s (configurable per agent).
Logging	The Coordinator logs the request and the response.

The request schema is:

{
  "request_id": "string",
  "agent": "string",
  "task": "string",
  "params": "object",
  "context": "object"
}

The response schema is:

{
  "request_id": "string",
  "status": "ok | failed | partial",
  "result": "object",
  "error": "string | null",
  "duration_ms": "integer"
}

Agent ↔ Framework

The agent invokes the framework through method calls. The calls are synchronous.

Aspect	Detail
Format	In-process method call.
Direction	Down.
Encoding	Native Python objects.
Size limit	No hard limit (large artifacts are written to disk).
Guarantee	Synchronous; the agent waits for the framework's result.
Logging	The framework logs every call.

The Research Framework's public API is documented in Research Framework → Source layout.

Framework ↔ Tool

The framework invokes tools through function calls. The calls are synchronous and return typed results.

Aspect	Detail
Format	Function call with typed arguments.
Direction	Down.
Encoding	Native Python objects.
Size limit	No hard limit.
Guarantee	Synchronous; the framework waits for the tool's result.
Logging	The tool logs every invocation.

The tool contract is documented in Tooling Layer.

Tool ↔ Adapter

A tool that needs to talk to an external service invokes an adapter. The adapter translates the tool's generic request into the provider's specific format.

Aspect	Detail
Format	Function call with typed arguments.
Direction	Down.
Encoding	Native Python objects.
Size limit	No hard limit.
Guarantee	Synchronous; the tool waits for the adapter's result.
Logging	The adapter logs every call to the request journal.

The adapter contract is documented in External Providers → Adapter contract.

Adapter ↔ External

The adapter talks to the external service through HTTP or HTTPS. The format and the guarantees depend on the provider.

Aspect	Detail
Format	HTTP / HTTPS.
Direction	Down.
Encoding	JSON (most providers) or form-encoded.
Size limit	Per provider (typically 1-10 MB per request).
Guarantee	Per adapter (the adapter handles retries).
Timeout	30s per request (configurable per adapter).
Logging	The adapter logs every call to the request journal.

The request journal is documented in External Providers → Request journal.

Coordinator ↔ State

The Coordinator reads and writes the state layer. The state layer includes MEMORY.md, daily logs, session state, and skill artifacts.

Aspect	Detail
Format	Markdown (for `MEMORY.md` and daily logs) or JSON (for session state and skill artifacts).
Direction	Both.
Encoding	UTF-8.
Size limit	1 MB per file (the Coordinator splits larger files).
Guarantee	Atomic write (the Coordinator writes to a temp file and renames).
Logging	The Coordinator logs every read and every write.

The state layer is documented in Memory and Context.

Framework ↔ State

The framework reads and writes the state layer for research artifacts. The artifacts are JSON files validated by Pydantic schemas.

Aspect	Detail
Format	JSON (validated by Pydantic schemas).
Direction	Both.
Encoding	UTF-8.
Size limit	No hard limit.
Guarantee	Atomic write; validation on read and write.
Logging	The framework logs every read and every write.

The artifact schemas are documented in Artifacts and the Data Dictionary.

End-to-end data flow

The end-to-end data flow for a research request is:

Loading diagram…

End-to-end research flow — every data crossing is auditable.

The flow shows every data crossing. Each crossing is auditable.

Data persistence

The lab persists data in the following locations:

Data	Location	Format
Long-term memory	`{workspace-root}/MEMORY.md`	Markdown
Daily logs	`{workspace-root}/memory/YYYY-MM-DD.md`	Markdown
Session state	`{workspace-root}/sessions/<id>/session.json`	JSON
Skill artifacts	`{workspace-root}/skills/...`	Markdown
Research artifacts	`{research-root}/runs/<mission_id>/*`	JSON
Request journal	`{framework-root}/runtime/request_journal.jsonl`	JSONL
Token pool state	`{framework-root}/runtime/token_pool_state.json`	JSON
Mission state	`{research-root}/runs/<mission_id>/phase.json`	JSON
Audit reports	`{workspace-root}/security/audits/...`	Markdown

The retention policy is documented in Mission Lifecycle → Retention and archival.