Audit Model

The audit layer — what is reviewed, what is recorded, the severity model, the typical findings, and the runtime infrastructure.

Purpose

The audit layer is the lab's systematic review of its own state. Audits answer four questions:

What was reviewed?
What works?
What fails?
What is degraded?
Which actions are priorities?

The output of an audit is a report. The reports are stored in {workspace-root}/security/audits/ and are referenced by the Failure Catalog and the Roadmap.

The audit layer also includes runtime audit infrastructure — the mechanisms that record every relevant event as it happens, so the audit reports can be constructed from real data rather than from memory. The two runtime mechanisms are the request journal and the token pool state, both maintained by the research framework. They are documented here because they are part of the lab's audit story, not part of the research framework's own internals.

Audit Cadence

Cadence	Scope	Report location
Weekly	State of gateways, sessions, in-flight missions	`audits/YYYY-MM-DD-weekly.md`
Monthly	All components, security boundary, threat model	`audits/YYYY-MM-DD-monthly.md`
Quarterly	External review of the security model	`audits/YYYY-MM-DD-quarterly.md`
Ad-hoc	Post-incident review	`audits/YYYY-MM-DD-incident-{name}.md`

The weekly audit is the operator's minimum. The monthly audit is the comprehensive review. The quarterly audit is performed by an external reviewer (when available). The ad-hoc audit is triggered by an incident.

Audit Report Structure

Every audit report has the following structure:

# Audit Report — {date}
 
## Scope
What was reviewed in this audit.
 
## Summary
A short summary of the findings. One paragraph.
 
## Findings
A numbered list of findings, each with:
- ID: A-{cadence}-{date}-{n}
- Severity: Critical | High | Medium | Low
- Status: New | Open | Resolved
- Description: What the finding is.
- Impact: What the impact is.
- Recommendation: What to do about it.
- Owner: Who is responsible for the resolution.
 
## Resolutions
A list of findings that were resolved since the last audit.
 
## Actions
A prioritized list of actions to take before the next audit.
 
## Notes
Any additional context.

Severity Model

Severity	Meaning	Response time
Critical	Blocks functionality or creates unsafe data exposure.	Immediate
High	Breaks an important flow or prevents verification.	Within 24h
Medium	Degrades quality, coverage, or reliability.	Within 1 week
Low	Pending improvement, cleanup, or documentation.	Next sprint

The severity is assigned by the auditor. The response time is the maximum time to begin work on the resolution; the actual time to resolution depends on the work required.

Typical Findings

The audit regularly surfaces the following finding types:

Unavailable external dependencies. A service that the lab depends on is down or rate-limited.
Missing packages. A Python package or Node module is missing from the environment.
Inaccessible web sources. A web source is behind a paywall or has changed its structure.
Partially functional flows. A flow that worked in the last audit is now partially working.
File permissions that should be hardened. A file or directory is readable or writable by a broader group than necessary.
Mocks still present where real data is expected. A test or integration is using mock data instead of real data.
Differences between documented state and real state. The documentation describes the system differently from how it actually works.
Stale memory entries. An entry in MEMORY.md that is no longer accurate.
Stale tool registry entries. An entry in tools.json that points to a tool that no longer exists.
Outdated documentation. A page that has not been updated to reflect a recent change.
Untested code paths. A code path that is not covered by tests.
Inconsistent error handling. Different parts of the code handle the same error differently.
Unused code. A module or function that is no longer used.
Inconsistent naming. A naming convention that is not followed everywhere.

Resolution Tracking

Findings are tracked from creation to resolution. The status of a finding transitions as:

New → Open → In Progress → Resolved
                  ↓
                Deferred (with reason)

A finding is Resolved when the recommendation has been implemented and verified. A finding is Deferred when the work is postponed to a later date; the reason is recorded.

The status is updated in the audit report where the finding was first created, and in the consolidated findings log at {workspace-root}/security/findings.jsonl.

Action Prioritization

At the end of each audit, the auditor produces a prioritized list of actions. The priority is based on:

Severity (Critical first).
Effort (low-effort, high-impact first).
Dependencies (unblocking other actions first).

The action list is the input to the next iteration of the roadmap. The roadmap is updated to reflect the actions that have been prioritized for the next period.

Auditor

The auditor is the operator. The weekly and monthly audits are performed by the operator. The quarterly audit is performed by an external reviewer when available; in the absence of an external reviewer, the operator performs a self-review with the explicit acknowledgment that it is a self-review.

The auditor's role is to ask the questions the operator might not ask of their own work. The auditor's authority is limited to making recommendations; the operator decides which recommendations to implement.

Runtime Audit: Request Journal

The research framework writes an append-only journal of every external call to {framework-root}/runtime/request_journal.jsonl. The journal is the foundation of the lab's runtime audit trail.

The journal is documented in detail in External Providers → Request Journal. The summary, from an audit perspective:

One JSON object per line.
One line per external request (search, transcript, Perplexity, etc.).
Records the mission, phase, adapter, action, request summary, response summary, result, and any error.
The token used is recorded by name, never by value.
The file is rotated when it exceeds the configured size (SCOUTE_JOURNAL_MAX_BYTES, default 50 MB).

The audit model uses the journal to answer questions like:

How many requests did mission X make to provider Y?
Which tokens were exhausted during the last week?
Which adapter failed the most in the last 30 days?
What is the average response time per provider?

The journal is rotated but never deleted. Old rotated files are kept in {framework-root}/runtime/journal-archive/.

Runtime Audit: Token Pool State

The research framework's token pool is configured in token_pool_config.json. The pool writes its runtime state to {framework-root}/runtime/token_pool_state.json.

The runtime state records per-token:

Total requests.
Total errors.
Last error (sanitized).
Last successful request timestamp.
Current status (active, quota_exhausted, error).

The audit model uses the pool state to answer:

Which tokens are currently in use?
Which tokens are exhausted and need attention?
Are there tokens that have been error for too long?

The state file is regenerated on every request. It is not rotated. It is the live view; the journal is the historical view.

What the Audit Reports Cover

The audit reports combine the runtime data (journal, pool state, mission states) with the static review (config files, code, documentation) to produce the findings.

For each audit:

The auditor exports the relevant runtime data into a snapshot.
The auditor reviews the snapshot, the code, and the documentation.
The auditor writes the audit report.
The findings are added to the consolidated log.
The actions are added to the next iteration of the roadmap.

The runtime data is what makes the audits reproducible: the same journal data + the same code = the same findings (modulo the model provider's non-determinism).

Audit Tooling

The lab has a small set of audit tools.

`audit-collect.sh`

Collects the runtime data into a snapshot.

{workspace-root}/ops/cheatsheets/audit-collect.sh

The output is a directory under {workspace-root}/security/snapshots/YYYY-MM-DD/ with:

request_journal.jsonl (a copy of the live journal).
token_pool_state.json (a copy of the live state).
mission_states.jsonl (a snapshot of all active mission states).
config_snapshot.json (a copy of the framework's configuration).

`audit-report.sh`

Generates an audit report from a snapshot.

{workspace-root}/ops/cheatsheets/audit-report.sh YYYY-MM-DD

The output is a markdown file at {workspace-root}/security/audits/YYYY-MM-DD.md.

`audit-review.sh`

Reviews the most recent audit report and lists the open findings.

{workspace-root}/ops/cheatsheets/audit-review.sh

The output is a list of open findings with their severity, recommendation, and status.

Real Audit Examples

The lab has performed several real audits. The reports are at:

security/audits/2026-05-11-full-hotel-cleanse.md
security/audits/2026-05-11-scout-ecosystem.md

These are sanitized versions of the actual audits. The reports follow the structure above and contain real findings, resolutions, and actions.

The reports are the most concrete examples of the audit model in action. They are worth reading to understand what a real audit looks like.