12# AutoBrowse — Self-Improving Browser Skill
13
14Build reliable browser automation skills through iterative experimentation. An inner agent browses the site (evaluate.ts). You — the outer agent — read what happened and improve the instructions (strategy.md). Repeat until it passes consistently.
15
16## Entry Points
17
18Invocation is flexible — both explicit flags and free-form natural language work:
19
20```
21/autobrowse --task google-flights
22/autobrowse --task google-flights --iterations 10 --env remote
23/autobrowse --task google-flights --browser-trace
24/autobrowse --tasks google-flights,amazon-add-to-cart
25/autobrowse --all
26
27# Also fine — parse freely:
28/autobrowse https://flights.google.com/
29/autobrowse book a flight on delta.com
30/autobrowse fix the existing google-flights skill
31```
32
33--browser-trace (default off, remote-only): pairs each iteration with the sibling browser-trace skill — wraps the inner agent in a CDP capture for per-page network/console/page-lifecycle evidence. Implies --env remote; errors if combined with --env local. Requires the sibling browser-trace skill present at ${CLAUDE_SKILL_DIR}/../browser-trace/, and the BROWSERBASE_API_KEY env var.
34
35When the user drops a URL or free-form instruction instead of --task <name>:
36- If an existing task in ${WORKSPACE}/tasks/ clearly matches the site/intent, use it.
37- Otherwise, pick a short kebab-case name, create ${WORKSPACE}/tasks/<name>/task.md from ${CLAUDE_SKILL_DIR}/references/example-task.md, fill in the URL/goal based on what the user said, and proceed. Tell the user the chosen name in one line.
38
39---
40
41## How to run
42
43### Step 1 — Parse arguments and orient
44
45Check what was passed:
46- --task <name> → single task mode
47- --tasks a,b,c or --all → multi-task mode (spawn sub-agents)
48- --iterations N → how many evaluate → improve cycles (default: 5)
49- --env local|remote → browser environment (default: local; use remote for bot-protected sites)
50- --browser-trace → opt in to the browser-trace integration (default off). Implies --env remote. If --env local --browser-trace are both passed explicitly, error with: browser-trace requires Browserbase; drop --env local or drop --browser-trace.
51
52If the user passed free-form text instead, map it to one of the above before continuing.
53
54### Step 2 — Set up the workspace
55
56All training artifacts (task definitions, strategy iterations, traces, reports) live in a workspace directory in the **current working directory** — NOT inside ~/.claude/skills/. This keeps the inner agent's file writes out of Claude's home dir and away from permission friction.
57
58Default workspace: ${CWD}/autobrowse/
59
60```bash
61mkdir -p ./autobrowse/tasks ./autobrowse/traces ./autobrowse/reports
62```
63
64If the task directory (./autobrowse/tasks/<task>/task.md) doesn't exist yet, scaffold it:
65
66```bash
67mkdir -p ./autobrowse/tasks/<task>
68cp ${CLAUDE_SKILL_DIR}/references/example-task.md ./autobrowse/tasks/<task>/task.md
69# Then edit task.md to describe the URL, inputs, steps, and expected JSON output
70```
71
72The skill source at ${CLAUDE_SKILL_DIR} stays read-only — only ./autobrowse/ in CWD gets written to during training. Graduation (final step) writes a single file to ~/.claude/skills/<task>/SKILL.md.
73
74List available tasks:
75```bash
76ls ./autobrowse/tasks/
77```
78
79### Step 3 — Multi-task: spawn parallel sub-agents
80
81If running multiple tasks, use the Agent tool to spawn one sub-agent per task simultaneously. Each sub-agent receives a self-contained prompt to run the full autobrowse loop for its task:
82
83> "You are running the autobrowse skill for task <name>. Workspace: <absolute-path-to-workspace> (e.g. /path/to/project/autobrowse). Run <N> iterations of: evaluate → read trace → improve strategy.md → repeat. Use --env <env>. Pass --workspace <workspace> to every evaluate.mjs invocation. If the parent invocation used --browser-trace, you MUST use the traced-path block of the SKILL.md loop for every iteration (pre-create session, attach bb-capture, pass --connect-url to evaluate.mjs, stop+bisect, release) — do not fall back to the default single-command path. Follow the autobrowse loop instructions exactly.
84>
85> When graduating, install the skill to ~/.claude/skills/<task-name>/SKILL.md with proper agentskills frontmatter (name + description). Do not just copy strategy.md — write a self-contained skill.
86>
87> At the end, output a structured summary with: task name, pass/fail on final run, total cumulative cost, iterations completed, per-iteration table (iter number, turns, cost, status, hypothesis tested), and 2-3 bullet key learnings."
88
89Spawn all sub-agents in parallel, wait for all to complete, then collect their summaries and write the session report.
90
91**For single task**, skip this step and run the loop directly below.
92
93---
94
95## The Loop (run this for each task)
96
97### Iteration start
98
99Check that ./autobrowse/tasks/<task>/task.md exists (scaffold it from the template if not — see Step 2). strategy.md is auto-created empty by the harness on first run.
100
101### Requirements
102
103- ANTHROPIC_API_KEY must be in the environment (or in a .env file in CWD — evaluate.mjs auto-loads it). If missing, the harness prints a clear error and exits; don't hunt for keys in other paths.
104
105### Run the inner agent
106
107**Default path (no --browser-trace)** — single command, no orchestration:
108
109```bash
110node ${CLAUDE_SKILL_DIR}/scripts/evaluate.mjs --task <task-name> --workspace ./autobrowse
111# or for bot-protected sites:
112node ${CLAUDE_SKILL_DIR}/scripts/evaluate.mjs --task <task-name> --workspace ./autobrowse --env remote
113```
114
115This runs the browser session and writes a full trace to ./autobrowse/traces/<task>/latest/.
116
117**Traced path (--browser-trace, remote only)** — the outer harness pre-creates a Browserbase session, attaches bb-capture as a passive observer, and passes the session's connectUrl to evaluate.mjs so every inner browse call uses --cdp $connectUrl --session autobrowse-main (the canonical browser-trace pattern that gives observers full Network/Console events). Run this block once per iteration with $N set to the 1-indexed iteration number:
118
119```bash
120# Preflight — fail fast if browser-trace isn't installed alongside autobrowse.
121BT_DIR="${CLAUDE_SKILL_DIR}/../browser-trace"
122if [ ! -f "$BT_DIR/scripts/bb-capture.mjs" ]; then
123 echo "ERROR: --browser-trace requires the browser-trace skill at $BT_DIR." >&2
124 echo "Install it by cloning github.com/browserbase/skills and copying skills/browser-trace/" >&2
125 echo "into the same parent directory as autobrowse (e.g. ~/.claude/skills/browser-trace/)." >&2
126 exit 1
127fi
128
129# a. SESSION SETUP — pre-create the keep-alive session and derive its connectUrl
130sid=$(browse cloud sessions create --keep-alive --verified --proxies \
131 | node -e "let s='';process.stdin.on('data',c=>s+=c).on('end',()=>process.stdout.write(JSON.parse(s).id))")
132connect_url=$(browse cloud sessions get "$sid" \
133 | node -e "let s='';process.stdin.on('data',c=>s+=c).on('end',()=>process.stdout.write(JSON.parse(s).connectUrl))")
134
135RUN_ID="run-$(printf '%03d' "$N")"
136TRACE_ROOT="./autobrowse/traces/<task-name>/$RUN_ID"
137mkdir -p "$TRACE_ROOT"
138export O11Y_ROOT="$TRACE_ROOT/.o11y" # park browser-trace output inside the autobrowse run dir
139export O11Y_RUN_ID="$RUN_ID" # tells the browse CLI which run dir to write descriptors.ndjson into
140
141# b. ATTACH BROWSER-TRACE — passive observer; runs in background
142node ${CLAUDE_SKILL_DIR}/../browser-trace/scripts/bb-capture.mjs "$sid" "$RUN_ID" &
143sleep 2
144
145# c. RUN AUTOBROWSE — connectUrl flag tells evaluate.mjs to inject --cdp/--session
146# into every inner browse call. The inner agent never sees --remote.
147node ${CLAUDE_SKILL_DIR}/scripts/evaluate.mjs \
148 --task <task-name> --workspace ./autobrowse --env remote \
149 --connect-url "$connect_url" --run-number "$N"
150
151# d. STOP + BISECT + UNIFY — order matters; bisect needs the session to still
152# exist, and unify-trace joins the bisect output with autobrowse's trace.json
153# into a single time-ordered NDJSON the outer agent reads first each iter.
154node ${CLAUDE_SKILL_DIR}/../browser-trace/scripts/stop-capture.mjs "$RUN_ID"
155node ${CLAUDE_SKILL_DIR}/../browser-trace/scripts/bisect-cdp.mjs "$RUN_ID"
156node ${CLAUDE_SKILL_DIR}/scripts/unify-trace.mjs \
157 --trace-dir "$TRACE_ROOT" \
158 --o11y-dir "$O11Y_ROOT/$RUN_ID"
159
160# e. RELEASE
161browse cloud sessions update "$sid" --status REQUEST_RELEASE
162```
163
164This writes the inner-agent trace to ./autobrowse/traces/<task-name>/latest/ and the CDP bisect to ./autobrowse/traces/<task-name>/latest/.o11y/<run-id>/. The traced browse CLI also emits per-command rich node descriptors to .o11y/<run-id>/cdp/descriptors.ndjson (one JSON object per page-driving call: target tag/id/role/accessibleName/attributes/xpath/bounding-rect). The descriptors file feeds downstream codegen; it is **not** required for hypothesis formation — skip it when reading the trace.
165
166### Read the trace
167
168```bash
169cat ./autobrowse/traces/<task-name>/latest/summary.md
170```
171
172The summary has duration, cost, turns, the decision log, and the final JSON output.
173
174If the agent failed or got stuck, look deeper:
175- Read ./autobrowse/traces/<task-name>/latest/trace.json — search for the failure turn
176- Read screenshots around the failure point with the Read tool
177
178**When --browser-trace was used — start with unified-events.jsonl.** The harness joins the agent's turn log and the browser's CDP firehose into one time-ordered NDJSON stream at the run root. One file, source-tagged (source: "agent" | "browser"), interleaved by wall-clock timestamp. Skim it top-to-bottom; the failure cause is usually one or two adjacent lines (the agent issued command X, the browser responded with Y).
179
180```bash
181cat ./autobrowse/traces/<task-name>/latest/unified-events.jsonl
182```
183
184The structured files (trace.json, .o11y/<run-id>/cdp/*) are **also agent-consumable as drill-downs** when the unified stream points at something you need more of:
185
186| Need | Drill-down file or command |
187|---|---|
188| Per-page totals + timing (events, network counts, errors by page) | .o11y/<run-id>/cdp/summary.json |
189| All failed network requests in one place | .o11y/<run-id>/cdp/network/failed.jsonl |
190| Full console exception payloads (stacktraces, etc.) | .o11y/<run-id>/cdp/console/exceptions.jsonl |
191| Per-page slice (only events on page N) | .o11y/<run-id>/cdp/pages/<pid>/ |
192| Full reasoning text / untruncated tool outputs for a specific turn | trace.json (filter by turn === N) |
193| Ad-hoc grouped query (e.g. top hosts, errors-by-page) | O11Y_ROOT=./autobrowse/traces/<task-name>/latest/.o11y node ${CLAUDE_SKILL_DIR}/../browser-trace/scripts/query.mjs <run-id> <cmd> |
194
195The unified stream is the default; drill into structured files only when you need a grouped query, a full-text payload, or filtering the stream can't give you.
196
197### Form one hypothesis
198
199Find the exact turn where things went wrong. What single heuristic would have prevented it?
200
201Under --browser-trace, the hypothesis must cite a **specific event from unified-events.jsonl** (line number or timestamp) — or name the drill-down file if you had to descend into one. This keeps updates evidence-grounded rather than vibes-driven. A hypothesis based only on the agent's commands might say "the click didn't work"; grounded in the unified stream, it can say "line 47 of unified-events.jsonl: browse open was followed by Network.responseReceived status 403 on /api/checkout — switch to --verified --proxies."
202
203Examples:
204- "After clicking the dropdown, wait 1s — options animate in before they're clickable"
205- "Navigate directly to /pay-invoice/ — skip the landing page entirely"
206- "Use browse fill #field_3 value not browse type — this field clears on focus"
207- "The page shows a spinner at turn 8 — add browse wait timeout 2000 before snapshot"
208- (with --browser-trace) "At line 47 of unified-events.jsonl, 3 consecutive Network.responseReceived events on /api/availability returned 403 right after browse open — the site is fingerprinting; the next iter needs --verified --proxies."
209
210### Update strategy.md
211
212Edit ./autobrowse/tasks/<task-name>/strategy.md. Keep everything that worked. Fix the specific failure. Add a concrete heuristic.
213
214Good strategies have:
215- **Fast path**: direct URL or shortcuts to skip exploration
216- **Step-by-step workflow**: exact sequence with timing notes
217- **Site-specific knowledge**: selector IDs, form field names, success indicators
218- **Failure recovery**: what to do when X goes wrong
219
220### Judge the result
221
222Read the new summary. Did it pass? Make clear progress?
223- **Pass or progress** → keep, next iteration
224- **No progress or regression** → revert strategy.md to the previous version and try a different hypothesis
225
226### Generate a runnable script (optional)
227
228Once the task has converged, you can produce a deterministic, runnable script
229in one or more frameworks via scripts/codegen.mjs. This is one shot of an
230LLM call per framework, cached by content hash, with optional verify-against-
231fresh-session and rewrite-on-failure.
232
233```bash
234node ${CLAUDE_SKILL_DIR}/scripts/codegen.mjs \
235 --task <name> \
236 --workspace ./autobrowse \
237 --frameworks playwright,stagehand \
238 --verify
239```
240
241Each framework gets its own subdirectory under tasks/<name>/<framework>/
242with the emitted script and a self-contained scaffold (package.json,
243tsconfig.json). The directory is runnable standalone with
244cd tasks/<name>/playwright && npm install && npx tsx <name>.ts — the only
245runtime requirement is BROWSERBASE_API_KEY (plus ANTHROPIC_API_KEY for
246the Stagehand target).
247
248Builtin frameworks: playwright, stagehand. Add a custom framework with
249--prompt-template <path> --frameworks custom (and provide your own runner
250or pass --no-verify).
251
252Common flags:
253
254| Flag | Purpose |
255|---|---|
256| --frameworks a,b,... | Comma-separated; default playwright |
257| --verify / --no-verify | Run the produced script against a fresh BB session; default --verify |
258| --max-retries N | Rewrite-on-verify-failure cap; default 2 |
259| --cache-only | Error if cache miss (CI-friendly) |
260| --force | Bust the cache |
261| --dry-run | Estimate prompt size + cost; don't call the LLM |
262| --run <id> | Force a specific run-NNN (default: latest passing) |
263
264Output is one JSON line per framework on stdout. Non-zero exit if any
265selected framework's final state is passed: false.
266
267See references/playwright-cdp-bridge.md for the canonical
268connectOverCDP patterns the emitted scripts follow.
269
270### After all iterations — publish if ready
271
272If the task passed on 2+ of the last 3 iterations **or has reached the max iteration limit**, install it as a Claude Code skill. **Do not just copy strategy.md** — the skill must be self-contained and useful to someone who has never seen this codebase. If graduating at max iterations without a clean pass, note the known failure point but still document everything learned.
273
274Install by writing to ~/.claude/skills/<task-name>/SKILL.md:
275
276```bash
277mkdir -p ~/.claude/skills/<task-name>
278```
279
280Use this structure for the SKILL.md:
281
282```markdown
283---
284name: <task-name>
285description: <1-2 sentences describing what this skill does and when to use it. Include trigger keywords.>
286---
287
288# <Task Title> — Browser Skill
289
290## Purpose
291<1-2 sentences: what this automates and why it exists.>
292
293## When to Use
294<When should someone reach for this skill.>
295
296## Browse CLI Reference
297The inner agent uses the browse CLI. Key commands for this task:
298- browse stop — kill existing session (always run before switching to remote)
299- browse open <url> --remote — start a fresh Browserbase cloud session and navigate
300- browse open <url> --local — start a clean local browser and navigate
301- browse tab new <url> — open URL in a new tab
302- browse wait load — wait for page to finish loading
303- browse wait timeout <ms> — wait a fixed amount of time for spinners or animations
304- browse wait selector "<selector>" — wait for an element to become visible
305- browse get title — verify you're on the right page
306- browse get text body — extract all visible text (preferred for content extraction)
307- browse snapshot — get accessibility tree; each node has a ref in [X-Y] format (e.g. [0-5], [2-147])
308- browse click [X-Y] — click element by ref from the latest snapshot (include the brackets)
309
310**Never use --session <name> flags in SKILL.md.** Named sessions are a parallel-run workaround — they contaminate skills with infrastructure concerns. Skills must work in isolation with the default session.
311
312## Workflow
313
314### Step 1 — Start session
315<exact browse commands in order>
316
317### Step 2 — Navigate
318<exact URL and verification steps>
319
320### Step 3 — Extract
321<exact extraction commands>
322
323### Step 4 — Output
324<what JSON to emit, referencing the schema below>
325
326## Site-Specific Gotchas
327<Bullet list of every hard-won heuristic from the iterations. This is the core value of the skill.>
328
329## Failure Recovery
330<What to do when navigation fails, session is contaminated, or extraction returns garbage>
331
332## Expected Output
333```json
334<paste the exact expected output schema from task.md>
335```
336```
337
338After writing the SKILL.md, confirm it's installed:
339```bash
340ls ~/.claude/skills/<task-name>/SKILL.md
341```
342
343The skill is now available as /<task-name> in Claude Code.
344
345---
346
347## Final report (multi-task mode)
348
349After all sub-agents complete, print a markdown table:
350
351| Task | Iterations | Final Status | Graduated | Cost |
352|------|-----------|--------------|-----------|------|
353| google-flights | 5 | ✅ pass | yes | $0.42 |
354| amazon-add-to-cart | 5 | ❌ fail | no | $1.20 |
355
356Then write a persistent session report to ./autobrowse/reports/ so there's a durable record of the run inside the workspace:
357
358```bash
359mkdir -p ./autobrowse/reports
360```
361
362Write the file ./autobrowse/reports/YYYY-MM-DD-HH-MM-<tasks>.md with:
363
364```markdown
365# AutoBrowse Session Report
366**Date:** <ISO date>
367**Tasks:** <comma-separated list>
368**Environment:** remote|local
369**Total cost:** $X.XX
370
371## Results
372
373| Task | Iterations | Pass Rate | Final Status | Graduated | Cost |
374|------|-----------|-----------|--------------|-----------|------|
375| ... | ... | X/5 | ✅/❌ | yes/no | $X.XX |
376
377## Per-Task Learnings
378
379### <task-name>
380- **Key insight 1:** <what the agent learned>
381- **Key insight 2:** <another heuristic>
382- **Failure mode fixed:** <what was failing and how it was resolved>
383
384## Iteration Log
385
386### <task-name>
387| Iter | Turns | Cost | Status | Hypothesis tested |
388|------|-------|------|--------|-------------------|
389| 1 | 79 | $18.75 | ❌ fail | baseline |
390| 2 | 9 | $0.26 | ✅ pass | session contamination fix |
391| ... | ... | ... | ... | ... |
392```
393
394---
395
396## Rules
397
398- **Only edit strategy.md** — never touch task.md (unless creating it from the template) or evaluate.mjs
399- **Stay in the workspace** — all training writes go to ./autobrowse/, never to ~/.claude/skills/autobrowse/. The skill source is read-only.
400- **One hypothesis per iteration** — test one change at a time
401- **Build on wins** — keep what worked, add to it
402- **Trust the trace** — the inner agent shows exactly what it saw and did
403- **Graduate to ~/.claude/skills/** — the only file you write there is the final graduated SKILL.md
404- **Don't release before bisecting** — under --browser-trace, the order at the end of each iteration is non-negotiable: stop-capture → bisect-cdp → browse cloud sessions update REQUEST_RELEASE. Bisect depends on the session still existing when the trace stops.
405