Appendix
Appendix D: Failure Catalog
The lab's known failure modes. Each entry has an ID, a description, a detection rule, a recovery procedure, and a severity.
Purpose
This appendix is the canonical reference for the lab's known failure modes. Each failure mode has an ID (F-NNN), a description, a detection rule, a recovery procedure, and a severity.
The catalog is the source for the runbook (Runbook) and the audit findings (Audit Model).
How to use this catalog
When an incident occurs:
- Find the matching failure mode in this catalog.
- Follow the recovery procedure.
- If the failure is not in the catalog, document it and add it to the catalog.
The catalog is a living document. New failure modes are added as they are discovered.
Failure modes
F-001: Coordinator gateway is unreachable
| Field | Value |
|---|---|
| ID | F-001 |
| Severity | Critical |
| Detection | curl -s http://localhost:18789/health returns a connection error. |
| Description | The Coordinator gateway is not responding. |
| Recovery | Apply Runbook → Worker recovery. |
| Prevention | Monitor the heartbeat; set up a watchdog. |
F-002: Scout gateway is unreachable
| Field | Value |
|---|---|
| ID | F-002 |
| Severity | High |
| Detection | curl -s http://localhost:18790/health returns a connection error. |
| Description | The Scout gateway is not responding. |
| Recovery | Apply Runbook → Worker recovery. |
| Prevention | Monitor the heartbeat; set up a watchdog. |
F-003: Coding sub-agent is killed by the OS
| Field | Value |
|---|---|
| ID | F-003 |
| Severity | High |
| Detection | The sub-agent exits with code 137; log shows Killed. |
| Description | The sub-agent was killed by the OS (typically SIGKILL) because its task was too large. |
| Recovery | Apply Runbook → Coding Assistant killed by timeout. |
| Prevention | Split tasks into smaller prompts; use progressive prompts. |
F-004: Model quota is exhausted
| Field | Value |
|---|---|
| ID | F-004 |
| Severity | High |
| Detection | The model returns 429 or quota_exceeded; the Coordinator's health probe reports quota_exhausted: true. |
| Description | The model's quota is exhausted. |
| Recovery | Apply Runbook → Model quota exhausted. |
| Prevention | Track the quota; warn before the quota is exhausted. |
F-005: Token pool is exhausted
| Field | Value |
|---|---|
| ID | F-005 |
| Severity | High |
| Detection | All tokens in token_pool_state.json are quota_exhausted; the P3 phase is failed_terminal. |
| Description | All tokens in the Perplexity token pool are exhausted. |
| Recovery | Apply Runbook → Token pool exhausted. |
| Prevention | Track the per-token quota; rotate tokens before they are exhausted. |
F-006: Browser CDP target is detached
| Field | Value |
|---|---|
| ID | F-006 |
| Severity | High |
| Detection | Browser actions return connection refused or Target.detached; the CDP port is not listening. |
| Description | The browser's DevTools Protocol connection is lost. |
| Recovery | Apply Runbook → Browser CDP target detached. |
| Prevention | Monitor the browser; restart on repeated detach. |
F-007: Media Lab Server is not responding
| Field | Value |
|---|---|
| ID | F-007 |
| Severity | Medium |
| Detection | curl -s http://127.0.0.1:8765/api/status returns a connection error. |
| Description | The Media Lab Server is not responding. |
| Recovery | Apply Runbook → Media Lab Server not responding. |
| Prevention | Monitor the server; restart on repeated failures. |
F-008: Mission is stuck in a non-terminal state
| Field | Value |
|---|---|
| ID | F-008 |
| Severity | High |
| Detection | A mission's phase.json has been in a non-terminal state for more than the timeout. |
| Description | A mission is stuck; the orchestrator is not progressing. |
| Recovery | Inspect the state; identify the stuck phase; resume or cancel the mission. |
| Prevention | Set a maximum mission duration; alert on overrun. |
F-009: Phase fails with failed_terminal
| Field | Value |
|---|---|
| ID | F-009 |
| Severity | High |
| Detection | The phase's status in phase.json is failed_terminal. |
| Description | A phase failed in a way that cannot be retried. |
| Recovery | Inspect the error; identify the cause; retry the mission or accept the failure. |
| Prevention | Add unit tests for the phase; review the error message before declaring it terminal. |
F-010: External API returns 4xx
| Field | Value |
|---|---|
| ID | F-010 |
| Severity | Medium |
| Detection | The adapter returns 4xx. |
| Description | An external API returned a 4xx error (other than 429). |
| Recovery | Inspect the error; fix the request; retry. |
| Prevention | Validate the request before sending; use a schema validator. |
F-011: External API returns 5xx
| Field | Value |
|---|---|
| ID | F-011 |
| Severity | Medium |
| Detection | The adapter returns 5xx. |
| Description | An external API returned a 5xx error. |
| Recovery | Retry with exponential backoff; if all retries fail, fall back to another adapter. |
| Prevention | Use the request journal to detect patterns; alert on sustained 5xx. |
F-012: External API rate limit
| Field | Value |
|---|---|
| ID | F-012 |
| Severity | Medium |
| Detection | The adapter returns 429. |
| Description | The external API rate limit was hit. |
| Recovery | Wait for the rate limit to reset; reduce the request rate. |
| Prevention | Track the per-adapter rate limit; use the per-minute cap. |
F-013: Artifact validation failure
| Field | Value |
|---|---|
| ID | F-013 |
| Severity | High |
| Detection | The ArtifactStore raises a Pydantic validation error. |
| Description | An artifact does not match its schema. |
| Recovery | Inspect the artifact; fix the producer; regenerate the artifact. |
| Prevention | Add unit tests for the schema; use the schema in the producer's tests. |
F-014: File permission is too permissive
| Field | Value |
|---|---|
| ID | F-014 |
| Severity | Low |
| Detection | ls -l shows permissions wider than 0600 on a sensitive file. |
| Description | A sensitive file is readable or writable by a broader group than necessary. |
| Recovery | chmod 600 <file>; fix the producer to set the correct permissions. |
| Prevention | Add a check in the CI that fails when permissions are too permissive. |
F-015: Outdated documentation
| Field | Value |
|---|---|
| ID | F-015 |
| Severity | Low |
| Detection | The documentation describes a system that has changed. |
| Description | The documentation is out of sync with the system. |
| Recovery | Update the documentation to match the system. |
| Prevention | Update the documentation with every commit that changes the system. |
F-016: Missing dependency
| Field | Value |
|---|---|
| ID | F-016 |
| Severity | Medium |
| Detection | An import fails; a tool reports a missing binary. |
| Description | A required package or binary is not installed. |
| Recovery | Install the dependency; document the install in the README. |
| Prevention | Add a dependency check in the CI; document dependencies in the README. |
F-017: Configuration drift
| Field | Value |
|---|---|
| ID | F-017 |
| Severity | Medium |
| Detection | The actual configuration differs from the documented one. |
| Description | The configuration has drifted from the documentation. |
| Recovery | Update the documentation to match the configuration, or update the configuration to match the documentation. |
| Prevention | Add a CI check that compares the live configuration to the documented one. |
F-018: Memory layer is full
| Field | Value |
|---|---|
| ID | F-018 |
| Severity | Medium |
| Detection | The memory layer's disk usage exceeds the threshold. |
| Description | The memory layer is using too much disk space. |
| Recovery | Apply the retention policy; archive old daily logs. |
| Prevention | Add a disk usage check to the daily tasks. |
F-019: Audit log is growing too fast
| Field | Value |
|---|---|
| ID | F-019 |
| Severity | Low |
| Detection | The audit log's growth rate exceeds the expected rate. |
| Description | The audit log is growing faster than expected, which may indicate a logging bug. |
| Recovery | Inspect the log; identify the source of the growth; fix the bug. |
| Prevention | Add a growth-rate check to the weekly tasks. |
F-020: Provider returns unexpected response
| Field | Value |
|---|---|
| ID | F-020 |
| Severity | Medium |
| Detection | The adapter returns a response that does not match the expected schema. |
| Description | An external provider returned a response in an unexpected format. |
| Recovery | Log the unexpected response; fall back to another adapter; report to the provider. |
| Prevention | Add a schema check in the adapter; alert on schema violations. |
Severity summary
| Severity | Count |
|---|---|
| Critical | 1 |
| High | 6 |
| Medium | 9 |
| Low | 4 |
| Total | 20 |
Adding a new failure mode
When a new failure mode is discovered:
- Add an entry to this catalog with the next ID.
- Update the runbook if the failure needs a new procedure.
- Update the audit report template if the failure affects the audit.
- Update the telemetry if the failure has a metric.
The new entry must have:
- A unique ID (F-NNN).
- A clear description.
- A reliable detection rule.
- A tested recovery procedure.
- A severity.