Web Agent

Status

Implemented. The Web Agent is a thin layer over the OpenClaw browser control server. It exposes a small action vocabulary that the Coordinator and the user can invoke through natural language.

Role

The Web Agent is responsible for:

• Opening URLs in a controlled browser instance.
• Navigating the page through the accessibility tree.
• Filling forms, clicking buttons, and reading content.
• Taking snapshots and screenshots (for the user).
• Maintaining the browser state across requests.

The Web Agent does not reason about the page. It is an action executor, not a planner. The Coordinator (or the user) decides what to do; the Web Agent does it.

Architecture

The Web Agent runs in the main gateway process. It controls a Chromium-based browser through the OpenClaw browser control server.

Browser control

The Web Agent uses the OpenClaw browser tool. The tool is documented in the bundled browser-automation skill. The summary:

• action="open" opens a URL in a new tab.
• action="snapshot" returns the accessibility tree of the current tab.
• action="act" performs a UI action (click, type, hover, drag, etc.).
• action="screenshot" returns a screenshot (for the user; the Web Agent itself uses snapshots).
• action="tabs" lists the open tabs.
• action="close" closes a tab.

The Web Agent uses snapshot for almost everything. The accessibility tree is more reliable than screenshots (the model can read it directly, no vision needed) and more stable than CSS selectors (the page structure can change; the accessibility tree is more semantic).

Refs and snapshot mode

The snapshot has two ref modes:

• role: refs are role + name based (e.g., e12 is a link with name "Sign in").
• aria: refs are Playwright aria-ref ids (stable across calls within the same tab).

The Web Agent prefers aria for multi-step flows (because the refs are stable) and role for one-off actions (because it's the default).

The snapshot is taken after every navigation and after every action that changes the page. The Web Agent maintains the most recent snapshot in memory.

Common flows

The Web Agent supports a small library of named flows. Each flow is a sequence of actions that the Web Agent can execute in one request.

Open and read

The simplest flow. Open a URL, return the text content of the page.

open(url) → snapshot → extract(text)

The text is what the Coordinator surfaces to the user.

Search a site

Open a site's search page, fill the search box, submit, return the results.

open(search_url) →
  act(fill, selector="input[name=q]", text=query) →
  act(press, key=Enter) →
  snapshot →
  extract(results)

The selector is taken from the snapshot.

Fill a form

Open a page with a form, fill the fields, submit, return the confirmation.

open(form_url) →
  act(fill, field1, value1) →
  act(fill, field2, value2) →
  act(click, submit_button) →
  snapshot →
  extract(confirmation)

Each act(fill, ...) uses a ref from the snapshot.

Apply filters

Open a search results page, apply filters, return the filtered results.

open(search_url) →
  act(click, filter_button) →
  act(click, filter_option) →
  snapshot →
  extract(filtered_results)

The filter interactions are taken from the snapshot.

Capture content for archival

Open a series of URLs, snapshot each, save the snapshots to a file. Used for capturing documentation (e.g., the research page on the lab's public site).

for url in urls:
  open(url) →
  snapshot →
  save_to({workspace-root}/captures/<name>.md)

This is the flow the lab uses to capture the public site's pages for analysis.

Real flows in the lab

The Web Agent has been used for:

• Booking.com. Search for houses in a city for a specific date range, apply filters, return the results. The flow uses open + fill (destination, dates, guests) + click (search) + snapshot + extract.
• Stremio catalog. Open the Stremio catalog, find a title, return the streaming options.
• Public site archival. Capture the lab's public site for offline analysis. The flow opens each URL, snapshots, and saves to disk.
• Form filling. Fill in research-related forms (subscriptions, alerts).

The flow implementations are not stored in the framework; they are composed from the action vocabulary at request time. This keeps the Web Agent small and flexible.

Limitations

The Web Agent is not a general-purpose browser automation tool. It is limited to:

• One browser instance at a time per agent.
• One tab open at a time per flow (multiple tabs are supported but not the default).
• Pages that the user can reach in a regular browser.
• Pages that do not require login credentials (the Coordinator does not pass credentials to the Web Agent).

Pages that require login credentials are reached through a separate flow that uses the user's session cookies. The Coordinator owns the session cookies; the Web Agent uses them.

Configuration

The Web Agent's configuration:

Failure modes

The full failure catalog is in Failure Catalog.

Privacy and security

The Web Agent does not log page content. It logs:

• The URL it opened.
• The actions it took.
• The errors it encountered.

The page content is returned to the Coordinator in the response. The Coordinator decides whether to log the content (typically not).

The Web Agent does not store cookies between invocations. Cookies are session-scoped. The Coordinator manages persistent cookies separately.

Future work

• Multi-tab flows. Support flows that span multiple tabs.
• Persistent sessions. Allow the user to log in once and reuse the session.
• Vision fallback. When the accessibility tree is insufficient, fall back to vision-based interaction.
• Recorded flows. Allow the user to record a flow and replay it.
• Mobile emulation. Emulate a mobile device for mobile-specific flows.

See also

• Agents and Roles
• Coordinator
• Media Agent
• Memory and Context
• Failure Catalog
• Runbook → Browser CDP target detached