Operations

Purpose

This page documents the lab's operational model. It is the companion to the Runbook (which is the step-by-step reference) and the Health Checks (which is the verification reference).

Cadence

The lab's operational cadence is:

The cadence is enforced by the framework's scheduler and by the user's discipline.

Roles

The lab's operations involve a single role: the operator. The operator is the user.

The operator's responsibilities:

• Monitor the health checks.
• Review the audit log.
• Apply retention policy.
• Update configuration.
• Restart agents when needed.
• Escalate issues to the framework's maintainers (the user, in the lab's case).

The operator is the only role. The lab does not have multiple users or multiple operators.

Daily checks

The daily checks are:

1. Token pool review. Read the token pool state file. Identify exhausted or error tokens. Add tokens if needed.
2. Mission state cleanup. Identify missions that have been in a non-terminal state for more than the timeout. Resume or cancel them.
3. Log rotation. Check the request journal's size. Rotate if needed.
4. Disk space. Verify the output directory and the research root have enough free space.

The daily checks take 5-10 minutes. The output of the daily checks is a short note in the daily log.

Weekly checks

The weekly checks add:

1. Audit review. Read the most recent audit reports. Identify new findings.
2. Health check report. Read the health check results. Identify degraded services.
3. Retention policy review. Check the retention counters. Adjust if needed.
4. Documentation review. Check that the documentation reflects the current system state.

The weekly checks take 30-60 minutes. The output is a weekly audit report.

Monthly checks

The monthly checks add:

1. Comprehensive audit. Review all components, the security boundary, and the threat model.
2. Security review. Review the security principles and the audit model. Update if needed.
3. Documentation review. Update the documentation to reflect the current system state. Identify gaps.
4. Roadmap review. Update the roadmap with the findings from the audit.

The monthly checks take 2-3 hours. The output is a monthly audit report and an updated roadmap.

Incident response

The lab's incident response is:

1. Detect. The heartbeat, the health checks, or the user detects the incident.
2. Triage. Identify the affected component and the severity.
3. Contain. Stop the affected component if necessary. Prevent the incident from spreading.
4. Mitigate. Apply the relevant runbook procedure.
5. Recover. Verify the system is back to a healthy state.
6. Post-mortem. Write a post-mortem document. Add the findings to the audit log.

The post-mortem document is stored under {workspace-root}/security/incidents/YYYY-MM-DD-{name}.md. The format is:

# Incident — {name}
 
## Date
{YYYY-MM-DD HH:MM UTC}
 
## Summary
{One paragraph.}
 
## Detection
{How the incident was detected.}
 
## Impact
{What was affected.}
 
## Root cause
{What caused the incident.}
 
## Resolution
{How the incident was resolved.}
 
## Follow-up actions
{What needs to change to prevent recurrence.}

Configuration management

The lab's configuration is managed through:

• Files. Configuration files in the locations documented in System Architecture → Configuration files.
• Tools. The Coordinator's session management and the framework's config validation.
• Reviews. Configuration changes are reviewed by the operator before being applied.

The operator is the only one who can change configuration. The Coordinator surfaces the change request and the operator approves it.

Backup

The lab's backup is:

• What is backed up. MEMORY.md, daily logs, skill artifacts, project notes, research artifacts, audit reports.
• What is NOT backed up. Generated outputs (Media Lab outputs, downloaded media), runtime state (session state, in-memory caches).
• Where. External drive or cloud (user's choice).
• When. Daily at midnight (user-configurable).
• How. The backup.sh script in {workspace-root}/ops/cheatsheets/.

The backup retention is 30 days (user-configurable). Older backups are deleted.

Disaster recovery

The lab's disaster recovery is:

1. Detect. The user detects the disaster (e.g., the host is wiped).
2. Restore. Restore from the most recent backup.
3. Reconfigure. Reinstall the tools, the frameworks, the configuration files.
4. Verify. Run the health checks. Verify that the system is back to a healthy state.
5. Document. Document the disaster and the recovery in the audit log.

The recovery time objective (RTO) is 4 hours. The recovery point objective (RPO) is 24 hours (the daily backup).

Change management

The lab's change management is:

1. Plan. The user decides to make a change.
2. Test. The change is tested in isolation.
3. Apply. The change is applied.
4. Verify. The health checks pass.
5. Document. The change is documented in the daily log.

The change is recorded in the audit log if it affects the security boundary or the production behavior.

See also

• Runbook
• Cheatsheet
• Health Checks
• Telemetry
• Audit Model
• Failure Catalog