Browser Control

agent-afk ships five native tools for driving a real Chromium browser. Each session gets its own isolated browser context. Subagents share the parent session's context.

Prerequisites

The Playwright Node package ships as a regular dependency and is installed with pnpm install. The ~300 MB Chromium binaries are not bundled — install them once before using a browser tool:

pnpm exec playwright install chromium

Quick start

# Install Chromium binaries (one-time)
pnpm exec playwright install chromium

# Start the REPL and ask the agent to browse
afk
> Open github.com and tell me what's on the homepage.

The agent issues browser_open → browser_observe → additional actions under the hood. In REPL mode, Chromium opens headed by default so you can watch the agent work.

The five tools

All five appear in afk --help once the package is built, alongside bash, read_file, and other tools.

Targeting elements in browser_act

browser_act accepts three target shapes:

// Preferred — by visible label and ARIA role.
{ "kind": "semantic", "text": "Sign in", "role": "button" }

// For unambiguous follow-up on an element from a recent observation.
{ "kind": "element_id", "element_id": "el_a1b2c3" }

// Escape hatch when the page has no accessible labels.
{ "kind": "selector", "selector": "button.primary[type=submit]" }

When a semantic target matches two or more elements, the tool returns an error with a disambiguation list of up to five candidates. The agent retries using element_id. Nothing silently picks among ambiguous matches.

Element IDs are stable within one observation only. Re-observe after any DOM mutation to get fresh IDs.

Configuration

All browser environment variables are optional.

Domain policy

export AFK_BROWSER_ALLOWED_DOMAINS="github.com,*.atlassian.net"
export AFK_BROWSER_BLOCKED_DOMAINS="*.ads.example.com"

Block wins over allow. The match uses simple * glob against the URL host — not a regex, not a path matcher. When the allowlist is non-empty, any host not on the list is refused. When the allowlist is empty (default), only the blocklist applies.

A blocked navigation returns an error the agent can read and act on — it tries a different URL instead of crashing.

JSON config file

For per-project overrides, create ~/.afk/config/browser.json (or point AFK_BROWSER_CONFIG at any absolute path):

{
  "headless": true,
  "allowedDomains": ["my-internal-tool.example.com"],
  "blockedDomains": [],
  "domSnapshots": false,
  "backend": "playwright"
}

File values override env-derived values. Arrays replace rather than append.

Trace and screenshots

Every browser tool emits records to ~/.afk/state/witness/<sessionId>/trace.jsonl:

• tool_call — the generic dispatcher record for every tool.
• browser_event — browser-domain semantics: URL transitions, action outcomes, screenshot paths, ambiguity verdicts, policy refusals.

Screenshots are written as sidecar files under ~/.afk/state/witness/<sessionId>/browser/screenshots/ and referenced by path in the trace — never embedded in the JSONL line.

The witness layer redacts known credential formats (AWS keys, OpenAI bearer tokens, GitHub PATs, Slack tokens, JWTs, password= form values) before persisting. The page receives the real value; only the trace file is redacted.

Driving your real Chrome (afk browser)

The native browser_* tools use Playwright against an isolated Chromium instance — they cannot attach to your real, logged-in Chrome profile. The afk browser command group wires Google's chrome-devtools-mcp into agent-afk's MCP client so the agent can drive the browser you're already signed into.

Why not Playwright?

Chrome 136+ refuses --remote-debugging-port on the default user-data-dir, and Chrome M144's sanctioned --autoConnect consent flow requires Puppeteer's handleDevToolsAsPage option, which Playwright lacks. chrome-devtools-mcp vendors Puppeteer and implements the --autoConnect flow, so wiring it in via MCP is the supported path.

Subcommands

afk browser connect

Writes a chrome-devtools entry to ~/.afk/config/mcp.json (idempotent — preserves any other servers you have configured) and prints one-time setup steps.

afk browser connect                    # target stable Chrome (default)
afk browser connect --channel canary   # target Chrome Canary's profile

Flag:

The MCP entry written looks like:

{
  "mcpServers": {
    "chrome-devtools": {
      "command": "npx",
      "args": ["chrome-devtools-mcp@latest", "--autoConnect"]
    }
  }
}

Non-stable channels append --channel <channel> to that args array.

afk browser disconnect

Removes the chrome-devtools entry from ~/.afk/config/mcp.json.

afk browser disconnect

No flags. Safe to run even if the entry is not present (prints a warning and exits cleanly).

One-time Chrome setup

1. Open chrome://inspect/#remote-debugging and enable remote debugging.
2. The first time the agent drives Chrome, click Allow on the prompt Chrome shows.
3. A "being controlled by automated test software" banner appears in Chrome while a session is active — that is expected.

Requirement: Chrome ≥ 144. The connect command detects your installed version and warns if it is too old.

How it relates to the native browser tools

Once connected, the agent gains mcp__chrome-devtools__* tools (navigate_page, click, fill, take_snapshot, take_screenshot, …) that operate on your real profile. The native browser_* tools (Playwright / isolated Chromium) remain available alongside them for throwaway automation that does not need your real logins. Run /mcp in the REPL to confirm the chrome-devtools server connected.

Security note

The consent flow is intentionally human-gated and cannot be bypassed: you enable it once at chrome://inspect, then approve each new session. While connected, the agent can read and interact with anything in your logged-in profile — only connect when you want it acting as you.

Known limitation

chrome-devtools-mcp's take_screenshot returns an image, but agent-afk's MCP bridge currently text-stubs image results, so those screenshots are not yet visible to the model. Use take_snapshot (an accessibility-tree snapshot) as the primary automation surface; it works today.

Troubleshooting

browserType.launch: Executable doesn't exist — run pnpm exec playwright install chromium.

Cannot find module 'playwright' — run pnpm install.

Navigation blocked by policy — the agent tried to reach a host outside AFK_BROWSER_ALLOWED_DOMAINS. Adjust the allowlist or unset it for permissive mode.

Zombie Chromium after a crash — SIGINT/SIGTERM/exit handlers normally close the browser provider. After an ungraceful kill:

ps aux | grep chromium | awk '{print $2}' | xargs kill