Health Checks

The health verification system. The liveness probes, the correctness checks, the performance checks, and the alerts.

Purpose

This page documents the lab's health checks: the liveness probes, the correctness checks, the performance checks, and the alerting policy.

The health checks are part of the operational cadence (Operations) and the runbook (Runbook).

Category	Question it answers
Liveness	Is the component up?
Correctness	Is the component behaving correctly?
Performance	Is the component meeting its performance targets?

Correctness checks

Correctness checks verify that the component is behaving correctly, not just that it is up.

Coordinator correctness

Check	Expected
Memory layer is readable	Reading `MEMORY.md` returns the expected content.
Skill loader activates a known skill	Activating `browser-automation` returns the skill.
Session store accepts a new session	A new session is created and listed.
Routing to the Research Worker works	A research request is dispatched and returns a result.

Research Worker correctness

Check	Expected
Mission initialization works	A new mission is created in the research root.
Phase F0 runs and produces `task.json`	The artifact is written and validates.
Phase F1 runs and produces `candidates.json`	The artifact is written and validates.
The orchestrator can resume a paused mission	The state is loaded and the next phase runs.

Media Agent correctness

Check	Expected
Device discovery works (when devices exist)	A non-empty device list is returned.
A playback request is accepted	The agent returns a result without error.
The media backend selector picks the right backend	The selected backend matches the rules.

Web Agent correctness

Check	Expected
A URL can be opened	The page loads and a snapshot is returned.
A fillable form can be filled	The form fields are filled and the form is submittable.
A snapshot returns actionable elements	The snapshot has at least one clickable element.

Coding Assistant correctness

Check	Expected
A simple task produces code	The sub-agent returns code that passes the tests.
The review rejects bad code	A code with a clear bug is rejected.
The integration places files in the right location	The files are in the project's expected paths.

Performance checks

Performance checks verify that the component meets its performance targets.

Coordinator performance

Check	Target
`duration_ms.p50`	< 5,000 ms
`duration_ms.p95`	< 15,000 ms
`duration_ms.p99`	< 30,000 ms
Tokens per request (avg)	< 5,000
Memory usage	< 2 GB
CPU usage (per request)	< 50% of one core (avg)

Research Worker performance

Check	Target
Mission duration (typical)	< 5 minutes
`F1_candidates.duration_ms.p50`	< 30,000 ms
`F2_reviews.duration_ms.p50`	< 30,000 ms
`F4_price.duration_ms.p50`	< 30,000 ms
`F6_report.duration_ms.p50`	< 5,000 ms
Memory usage	< 1 GB

Media Agent performance

Check	Target
Playback initiation	< 5,000 ms
Backend selection	< 100 ms
Device discovery (cached)	< 50 ms
Device discovery (cold)	< 5,000 ms

Web Agent performance

Check	Target
`act.duration_ms.p50`	< 1,000 ms
`snapshot.duration_ms.p50`	< 2,000 ms

The health checks can be run individually (as in the probes above) or as a script. The lab has a script at {workspace-root}/ops/cheatsheets/health-check.sh that runs all the checks in order and produces a report.

The script's output is a JSON document with one entry per check:

{
  "timestamp": "2026-06-10T20:00:00Z",
  "checks": [
    {"name": "coordinator.liveness", "status": "ok", "duration_ms": 12},
    {"name": "scout.liveness", "status": "ok", "duration_ms": 8},
    {"name": "browser.liveness", "status": "ok", "duration_ms": 5},
    {"name": "media-lab.liveness", "status": "ok", "duration_ms": 23},
    {"name": "research.correctness.f1", "status": "ok", "duration_ms": 4123}
  ]
}

The report is written to {workspace-root}/logs/health-check-<timestamp>.json.

Alerting

The lab does not have an alerting system. The operator is the alerting system.

The operator checks the health checks:

Daily. During the daily checks.
On heartbeat. If the heartbeat probe detects an issue.
On user request. When the user asks "is everything OK?".

When a check fails, the operator follows the relevant runbook procedure. The runbook is in Runbook.

Failure modes

Failure	Severity	Action
Coordinator gateway is down	Critical	Apply Worker recovery.
Scout gateway is down	High	Apply Worker recovery.
Browser CDP is detached	High	Apply Browser CDP target detached.
Media Lab Server is down	Medium	Apply Media Lab Server not responding.
Model quota exhausted	High	Apply Model quota exhausted.
Token pool exhausted	High	Apply Token pool: all tokens exhausted.
Mission stuck in a non-terminal state	High	Resume or cancel the mission.
Memory layer not writable	Critical	Investigate filesystem permissions.
Audit log growing too fast	Low	Apply retention policy.

The full failure catalog is in Failure Catalog.

Purpose

Categories

Liveness probes

Coordinator gateway

Scout gateway

Browser CDP

Media Lab Server

Coding sub-agent