Telemetry
What the lab observes. The metrics, the logs, the traces, and the dashboards.
Purpose
This page documents the lab's telemetry: the metrics the lab collects, the logs the lab writes, the traces the lab produces, and the dashboards the operator can build from them.
The telemetry is the foundation of the audit model (Audit Model) and the health checks (Health Checks).
Metrics
The lab collects the following metrics:
Coordinator metrics
| Metric | Type | Description |
|---|---|---|
coordinator.sessions.active | gauge | The number of active sessions. |
coordinator.requests.total | counter | The total number of requests received. |
coordinator.requests.failed | counter | The total number of failed requests. |
coordinator.duration_ms.p50 | gauge | The p50 request duration. |
coordinator.duration_ms.p95 | gauge | The p95 request duration. |
coordinator.duration_ms.p99 | gauge | The p99 request duration. |
coordinator.tokens.used | counter | The total tokens used by the Coordinator. |
coordinator.tokens.remaining | gauge | The tokens remaining in the quota. |
Research Worker metrics
| Metric | Type | Description |
|---|---|---|
research.missions.active | gauge | The number of active missions. |
research.missions.total | counter | The total number of missions started. |
research.missions.succeeded | counter | The total number of missions that succeeded. |
research.missions.failed | counter | The total number of missions that failed. |
research.missions.partial | counter | The total number of partial missions. |
research.phases.duration_ms | histogram | The duration of each phase, by phase ID. |
research.adapters.requests | counter | The number of requests per adapter. |
research.adapters.errors | counter | The number of errors per adapter. |
research.tokens.used | counter | The total tokens used by the Perplexity adapter. |
Media Agent metrics
| Metric | Type | Description |
|---|---|---|
media.requests.total | counter | The total number of media requests. |
media.playbacks.started | counter | The total number of playbacks started. |
media.playbacks.failed | counter | The total number of playbacks that failed. |
media.backends.used | counter | The number of times each backend was used. |
media.devices.discovered | gauge | The number of devices currently discovered. |
Web Agent metrics
| Metric | Type | Description |
|---|---|---|
web.actions.total | counter | The total number of browser actions. |
web.actions.failed | counter | The total number of failed actions. |
web.snapshots.total | counter | The total number of snapshots taken. |
web.duration_ms.p50 | gauge | The p50 action duration. |
web.duration_ms.p95 | gauge | The p95 action duration. |
Coding Assistant metrics
| Metric | Type | Description |
|---|---|---|
coding.invocations.total | counter | The total number of sub-agent invocations. |
coding.invocations.failed | counter | The total number of invocations that failed. |
coding.duration_ms.p50 | gauge | The p50 invocation duration. |
coding.iterations.p50 | gauge | The p50 iterations per invocation. |
coding.tokens.used | counter | The total tokens used by the sub-agent. |
Logs
The lab writes the following logs:
Coordinator log
The Coordinator's log is at
{workspace-root}/logs/coordinator.log. It records:
- Every request received (with session ID and request ID).
- Every response sent (with status, duration, and result summary).
- Every state change (session start/end, memory write, etc.).
- Every error (with stack trace).
The log format is JSON Lines (one JSON object per line). The log is rotated daily.
Scout log
The Scout log is at
~/.openclaw-scout/logs/gateway.log. It records the
same kind of events for the Research Worker and the
Media Agent.
Coding sub-agent log
The Coding sub-agent's log is at
~/Documents/Codex/YYYY-MM-DD/<session>/log.jsonl.
The sub-agent writes its own log format.
Mission log
Each mission has a log at
{research-root}/runs/<mission_id>/logs/mission.log.
The log records every phase transition and every
artifact write.
Request journal
The request journal is at
{framework-root}/runtime/request_journal.jsonl. It
records every external call. The journal is the most
detailed log; it is documented in
External Providers → Request journal.
Traces
The lab produces traces for research missions. A trace
is a directed acyclic graph of phase executions. The
trace is reconstructed from phase.json and the
mission log.
The trace format is OpenTelemetry-compatible. The Coordinator can export the trace to a tracing backend if one is configured.
Dashboards
The lab has a small set of dashboards built from the telemetry. The dashboards are:
Coordinator dashboard
- Active sessions (gauge).
- Request rate (counter rate).
- Error rate (counter rate).
- Request duration percentiles (gauges).
- Token usage (counter rate).
Research Worker dashboard
- Active missions (gauge).
- Mission rate (counter rate).
- Mission success rate (gauge).
- Phase duration percentiles (gauges, by phase).
- Adapter error rate (counter rate, by adapter).
Media Agent dashboard
- Playback rate (counter rate).
- Playback failure rate (counter rate).
- Backend usage (counter rate, by backend).
- Device count (gauge).
The dashboards are built by the operator from the metrics. The lab does not have a built-in dashboard server; the operator exports the metrics to a backend of their choice (e.g., Prometheus + Grafana, or a simple text-based dashboard).
Retention
| Data | Retention | Location |
|---|---|---|
| Coordinator log | 30 days | {workspace-root}/logs/ |
| Scout log | 30 days | ~/.openclaw-scout/logs/ |
| Mission log | Mission lifetime + 30 days | {research-root}/runs/<id>/logs/ |
| Request journal | Indefinite (rotated) | {framework-root}/runtime/ |
| Metrics | 90 days | (exported) |
The retention is enforced by the daily cleanup task.
Sampling
Some metrics are sampled to reduce overhead:
*_duration_ms.p50/p95/p99is computed from a 10% sample of requests.- The request journal records every request, but the body of large requests is truncated.
The sampling rate is configurable. The default is 10% for percentiles, 0% truncation for journal bodies.