Lab Notes
Operations

Operations Runbook

Step-by-step procedures for the recurring operations. Each procedure is a self-contained section that can be followed without reading the rest of the documentation.

Purpose

This runbook is the operator's reference. It is written for the human who needs to fix something, restart something, or verify that something is working. Each procedure is a self-contained section with the prerequisites, the steps, the expected output, and the recovery if the procedure fails.

The cheatsheet companion is in Cheatsheet; the health checks are in Health Checks.

Conventions

  • All paths use the placeholder {workspace-root}. The real path is the operator's workspace root.
  • All commands assume a POSIX shell. On macOS and Linux, this is the default.
  • Commands that are destructive are marked with [DESTRUCTIVE]. They require explicit confirmation.
  • Procedures assume the operator has shell access to the host.

State review

Verify the health of all gateways, sessions, and tools.

Procedure

  1. Check the Coordinator gateway:

    curl -s http://localhost:{coordinator-port}/health

    Expected output: a JSON document with "status": "ok".

  2. Check the Research Worker gateway:

    curl -s http://localhost:{worker-port}/health

    Expected output: a JSON document with "status": "ok".

  3. Check the Media Agent gateway:

    curl -s http://localhost:{media-port}/health

    Expected output: a JSON document with "status": "ok".

  4. List active sessions:

    {workspace-root}/ops/cheatsheets/list-sessions.sh

    Expected output: a list of session IDs and their states.

  5. List in-flight research missions:

    ls {workspace-root}/research/runs/

    Expected output: a list of mission folders. Each folder has a phase.json with a non-terminal state, or is archived (terminal state).

Recovery

If a gateway is not responding, see the recovery procedure for the specific gateway below.

Session resume

Continue previous work without losing context.

Procedure

  1. Identify the session to resume:

    {workspace-root}/ops/cheatsheets/list-sessions.sh

  2. Resume the session:

    {framework-cli} session resume {session-id}

  3. Verify the session is active:

    {framework-cli} session status {session-id}

Recovery

If the session cannot be resumed, the most common cause is that the underlying process has crashed. Restart the gateway and try again.

Worker recovery

Restart the Research Worker after a crash or after a configuration change.

Procedure

  1. Identify the supervisor:

    • On macOS: launchd with the LaunchAgent ~/Library/LaunchAgents/{worker-launchagent-label}.plist.
    • On Linux: systemd with the unit {worker-systemd-unit}.service.

  2. On macOS:

    launchctl kickstart -k gui/$(id -u)/{worker-launchagent-label}

    The -k flag kills the current process (if any) and restarts it. KeepAlive=true ensures the supervisor restarts the process if it crashes.

  3. On Linux:

    sudo systemctl restart {worker-systemd-unit}.service

  4. Verify the worker is up:

    curl -s http://localhost:{worker-port}/health

    Expected output: a JSON document with "status": "ok".

  5. Verify no missions are stuck:

    for mission in {workspace-root}/research/runs/*/; do
  phase=$(jq -r .current_phase "$mission/phase.json")
  if [ "$phase" != "done" ]; then
    echo "In-flight: $mission ($phase)"
  fi
done

    Expected output: a list of in-flight missions. If the list is empty, all missions are in a terminal state.

Recovery

If the worker does not start:

  • Check the log file:
    • macOS: tail -f /Users/{user}/.openclaw-scout/logs/gateway.log
    • Linux: sudo journalctl -u {worker-systemd-unit}.service -n 100
  • Common issues:
    • Port already in use. Identify and kill the conflicting process.
    • Configuration error. Validate the configuration file.
    • Missing dependencies. Install them.

If a mission is stuck in a non-terminal state, see Research Mission Lifecycle → Recovery.

Media Agent recovery

Restart the Media Agent after a crash or after a configuration change.

Procedure

  1. Identify the supervisor (same as the worker).
  2. Restart the process:
    • On macOS: 
      launchctl kickstart -k gui/$(id -u)/{media-launchagent-label}
    • On Linux: 
      sudo systemctl restart {media-systemd-unit}.service
  3. Verify the agent is up: 
    curl -s http://localhost:{media-port}/health

Coding Assistant invocation

Run the Coding Assistant non-interactively to delegate a specific implementation or analysis task.

Procedure

The Coding Assistant is invoked through a scoped prompt. The exact invocation is framework-specific; the documented requirements are:

  1. The prompt describes the goal and the boundaries, not the implementation.
  2. The prompt references the relevant AGENTS.md, SKILL.md, and TOOL.md files.
  3. The invocation has the trust level required for the task:
    • read-only for read-only inspection.
    • workspace-write for code changes within the workspace.
    • Full trust for tasks that require production access (rare; only with explicit user authorization).

Example invocation (illustrative):

{framework-cli} coding-assistant run \
  --prompt-file {workspace-root}/ops/prompts/{task-name}.md \
  --workspace {workspace-root}/{project-name} \
  --trust-level workspace-write

The prompt file is the implementation prompt described in Coding Assistant → Workflow.

Test suite

Run the test suite for a project.

Procedure

  1. Change to the project directory:

    cd {workspace-root}/{project-name}

  2. Install dependencies (if not already installed):

    npm install   # for Node projects
# or
pip install -r requirements.txt   # for Python projects

  3. Run the tests:

    npm test      # for Vitest
# or
pytest        # for pytest

  4. Verify the test output:

    • All tests pass.
    • The coverage report is at or above the project's minimum (typically 70%).

Audit review

Review the audit log for the past period.

Procedure

  1. List the audit reports for the period:

    ls -lt {workspace-root}/security/audits/ | head -20

  2. Open the most recent report:

    cat {workspace-root}/security/audits/{latest-report}.md

  3. For each finding, check:

    • Is the finding still valid?
    • Has it been resolved?
    • Is the severity still accurate?

  4. Update the report with the current state.

  5. If new findings are identified, add them to Failure Catalog.

Backup

Back up the workspace.

Procedure

  1. Verify the backup destination is available:

    ls {backup-destination}

  2. Run the backup script:

    {workspace-root}/ops/cheatsheets/backup.sh

  3. Verify the backup:

    ls -lt {backup-destination}/ | head -5

    The most recent entry should be the backup just created.

  4. Verify the backup integrity:

    {workspace-root}/ops/cheatsheets/verify-backup.sh {backup-name}

Recovery

If the backup fails:

  • Check the disk space on the backup destination.
  • Check the permissions on the backup destination.
  • Check the script's log for the specific error.

If the backup is corrupted, restore from the previous backup and document the incident in the audit log.

Browser recovery

Recover the browser instance when it crashes or stops responding.

Procedure

  1. Close all browser instances:

    {framework-cli} browser close --all

  2. Start a new browser instance:

    {framework-cli} browser start

  3. Verify the browser is responsive:

    {framework-cli} browser status

Retention

Archive old research missions to cold storage.

Procedure

  1. Identify missions older than the retention period (default 90 days):

    find {workspace-root}/research/runs/ -maxdepth 1 -mindepth 1 \
  -mtime +{retention-days}

  2. Verify the cold storage is available:

    ls {cold-storage-destination}

  3. Archive the missions:

    {workspace-root}/ops/cheatsheets/archive-missions.sh {retention-days}

  4. Verify the archive:

    ls {cold-storage-destination}/research/{year}/{month}/

  5. Remove the archived missions from the workspace:

    [DESTRUCTIVE] rm -rf {archived-missions}

    Confirm with the user before running this command.

Coding Assistant killed by timeout

The Coding Assistant process is killed by the OS (SIGKILL) when its task or prompt is too large. The typical symptom is a log line containing killed and a return code of 137.

Symptoms

  • The Coding Assistant exits with code 137.
  • The log shows Killed near the end of the run.
  • No artifacts or partial output in the workspace.

Procedure

  1. Do not write the code by hand. The Coding Assistant recovery rule is absolute. Stop and recover the assistant before doing anything else.
  2. Identify the task that killed the assistant. It is usually the largest file or the most complex phase.
  3. Split the task into smaller prompts. The recommended rule: one file per prompt. Smaller prompts have lower memory and lower latency.
  4. Re-run with a longer timeout (recommended: 120-180s for small tasks).
  5. If the assistant still dies, consider a more aggressive split: header-only file first, then implementation, then tests.

Recovery

If the Coding Assistant cannot be recovered after three attempts with progressively smaller prompts, stop and ask the operator for help. The operator may need to adjust the environment (memory, swap) or use a different execution path.

Model quota exhausted

The model's text quota is exhausted. The symptom is a 429 or a quota_exceeded error from the model provider. The Text quota is on a 5-hour rolling window; voice, image, and music quotas are daily.

Symptoms

  • The model returns a 429 or a quota_exceeded error.
  • The Coordinator's health probe reports quota_exhausted: true.
  • The quota tracking file shows a non-zero usage percentage.

Procedure

  1. Identify which quota is exhausted (text, voice, image, music) by reading the model provider's response.
  2. For the text quota, wait for the 5-hour rolling window to reset. The reset time is reported by the provider. During the wait:
    • The Coordinator should not initiate new text requests.
    • Background tasks should pause.
    • The user can be informed that the assistant is in a "quota cooldown" state.
  3. For the voice, image, or music quota, wait for the daily reset.
  4. If a critical task is in flight, consider switching to an alternative model (if configured) or to a fallback execution path (e.g., run a CLI tool directly).

Recovery

When the quota resets:

  1. Verify the reset by running a minimal text request.
  2. Resume the in-flight tasks.
  3. Update the quota tracking file with the new usage.
  4. If the quota reset time is not honored, log the incident and consider rotating the API key.

Token pool: all tokens exhausted

All tokens in the research framework's Perplexity token pool are marked quota_exhausted. The symptom is a failed_terminal on the P3 phase with the error no_tokens_available.

Procedure

  1. Read the token pool state file: 
    cat {framework-root}/runtime/token_pool_state.json
  2. Identify which tokens are exhausted and which (if any) are still active.
  3. Wait for the exhausted tokens to reset. Perplexity quotas reset daily.
  4. If the wait is unacceptable, add a new token to the pool:
    • Edit {framework-root}/config/token_pool_config.json.
    • Add the new token entry.
    • Reload the pool (the framework reloads on every request; no restart needed).
  5. Verify the new token is in use by reading the journal and looking for the new token's name field.

Recovery

If the operator cannot add a new token, disable P3 for the affected missions. The DAG handles P3 being absent (phases that need synthesis.json will be marked blocked_missing_input).

Browser CDP target detached

The browser automation loses its connection to the Chromium DevTools Protocol. The symptom is an error containing Target.detached or connection refused on the CDP port (default 18800).

Symptoms

  • Browser actions return connection refused or Target.detached.
  • The Coordinator cannot take a snapshot.
  • The browser tab is still open but unreachable.

Procedure

  1. Check if the browser process is still running: 
    lsof -nP -iTCP:18800 -sTCP:LISTEN
  2. If the process is not running, restart the browser: 
    {framework-cli} browser start
  3. If the process is running but not responding, kill it and restart: 
    kill -9 $(lsof -nP -iTCP:18800 -sTCP:LISTEN -t)
{framework-cli} browser start
  4. Verify the new browser is responsive by taking a snapshot of a known URL.

Recovery

If the browser repeatedly detaches:

  • Check the host's resources (memory, CPU).
  • Check for conflicting browser instances.
  • Consider switching to a fresh profile.
  • Document the incident in the audit log.

Media Lab Server not responding

The Media Lab Server (image and music generation) is not responding on 127.0.0.1:8765. The symptom is a connection refused or a timeout on the API.

Procedure

  1. Check if the server process is running: 
    lsof -nP -iTCP:8765 -sTCP:LISTEN
  2. If not running, start it: 
    cd {workspace-root}/devclaw-tools/audio-server
python3 server.py &
  3. Verify the server is up: 
    curl -s http://127.0.0.1:8765/api/status
  4. If the server is running but not responding, check the log for the most recent error.
  5. If the server repeatedly fails, check the output directory's disk space and permissions.

See also

Previous

Operations

Next

Operations Cheatsheet

On this page