9# Browser Trace
10
11Attach a **second, read-only CDP client** to a browser session that is already being driven by your main automation. The trace records the full DevTools firehose to NDJSON, polls for screenshots and DOM dumps in parallel, and slices everything into a directory tree that bash tools can search.
12
13This skill does **not** drive pages — it only listens. Pair it with the browser skill, browse, Stagehand, Playwright, or anything else that speaks CDP.
14
15## When to use
16
17- The user wants to debug a browser-automation run (failing form, missing element, hung navigation, JS exception).
18- The user has a running automation and wants to attach a trace mid-flight without restarting it.
19- The user wants to split a CDP firehose into network / console / DOM / page buckets.
20- The user wants screenshots + DOM snapshots over time, joined to CDP events by timestamp.
21
22If the user just wants to **drive** the browser, use the browser skill instead.
23
24## Setup check
25
26```bash
27node --version # require Node 18+
28which browse || npm install -g browse
29which jq || true # optional — used only for ad-hoc querying
30```
31
32Verify browse cdp exists:
33
34```bash
35browse --help | grep -q "^\s*cdp " || echo "browse cdp not available — update browse"
36```
37
38## How it works
39
40Every Chrome DevTools target accepts **multiple concurrent CDP clients**. Your main automation is one client; this skill adds a second one that only enables observation domains (Network, Console, Runtime, Log, Page) and never sends action commands.
41
42The tracer has three pieces:
43
441. **Firehose**: browse cdp <target> streams every CDP event as one JSON object per line to cdp/raw.ndjson.
452. **Sampler**: a polling loop calls browse screenshot --cdp <target> --path <file> and browse get html body --cdp <target> on an interval (default 2s). The helper passes --cdp when it samples so it can attach to the traced target from its own process; once a browse daemon session is attached to a CDP target, follow-up commands in that session do not need to repeat --cdp.
463. **Bisector**: after the run, bisect-cdp.mjs walks raw.ndjson once, slices it into per-bucket JSONL files keyed by CDP method, and additionally bisects per page using top-level Page.frameNavigated events as boundaries.
47
48## Quickstart
49
50### Local Chrome
51
52```bash
53# 1. Launch Chrome with a debugger port (any user-data-dir keeps it isolated).
54"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
55 --remote-debugging-port=9222 \
56 --user-data-dir=/tmp/chrome-o11y \
57 about:blank &
58
59# 2. Start the tracer.
60node scripts/start-capture.mjs 9222 my-run
61
62# 3. Run your main automation against port 9222.
63browse open https://example.com --cdp 9222
64# ...whatever the run does...
65
66# 4. Stop and bisect.
67node scripts/stop-capture.mjs my-run
68node scripts/bisect-cdp.mjs my-run
69```
70
71### Browserbase remote
72
73Two helpers wrap the platform-side bookkeeping: bb-capture.mjs creates or attaches to a session and starts the tracer; bb-finalize.mjs pulls platform artifacts (final session metadata, server logs, downloads) into the run dir at the end.
74
75> Browserbase ends a session as soon as its last CDP client disconnects. **Create with --keep-alive, then attach automation to the session's connectUrl before or together with the tracer.** bb-capture.mjs --new handles the keep-alive session and tracer setup; your automation still needs to attach.
76
77```bash
78export BROWSERBASE_API_KEY=...
79
80# 1. Create a keep-alive session AND start the tracer in one step.
81# Prints the session id, connectUrl prefix, and a live debugger URL you
82# can open in a browser to watch the run interactively.
83node scripts/bb-capture.mjs --new my-run
84
85# 2. Drive automation. bb-capture stamped the session id into the manifest.
86SID=$(jq -r .browserbase.session_id .o11y/my-run/manifest.json)
87CONNECT_URL="$(browse cloud sessions get "$SID" | jq -r .connectUrl)"
88BROWSE_NAME=my-run-browser
89browse open https://example.com --cdp "$CONNECT_URL" --session "$BROWSE_NAME"
90browse open https://news.ycombinator.com --session "$BROWSE_NAME"
91
92# 3. Stop the tracer, bisect, then pull platform artifacts and release.
93node scripts/stop-capture.mjs my-run
94node scripts/bisect-cdp.mjs my-run
95node scripts/bb-finalize.mjs my-run --release
96```
97
98Attaching to a session that's *already running* (e.g. one your production worker created) — bb-capture.mjs accepts a session id instead of --new:
99
100```bash
101# Pick a running session (filter client-side; browse cloud sessions list has no --status flag)
102browse cloud sessions list | jq -r '.[] | select(.status == "RUNNING") | .id'
103
104node scripts/bb-capture.mjs <session-id> mid-flight-debug
105# ...tracer runs alongside the existing automation client; no disruption...
106node scripts/stop-capture.mjs mid-flight-debug
107node scripts/bisect-cdp.mjs mid-flight-debug
108node scripts/bb-finalize.mjs mid-flight-debug # without --release: leave the session running
109```
110
111#### What you get from the Browserbase platform
112
113bb-capture.mjs adds a browserbase block to manifest.json (session id, project, region, started_at, expires_at, debugger URL). bb-finalize.mjs writes:
114
115- <run>/browserbase/session.json — final browse cloud sessions get snapshot (proxyBytes, status, ended_at, viewport, …)
116- <run>/browserbase/logs.json — browse cloud sessions logs output. **Often empty.** The CDP firehose in cdp/raw.ndjson is the source of truth; this is a side channel.
117- <run>/browserbase/downloads.zip — files the session downloaded, if any (the script discards the empty 22-byte zip you get when there are none)
118
119Session replay artifact fetching is **deprecated** and isn't fetched. Use the screenshots + DOM dumps in screenshots/ and dom/ for visual ground truth.
120
121The live debugger_url in the manifest opens an interactive Chrome DevTools view served by Browserbase — handy for *watching* a long-running automation while the tracer captures the firehose to disk.
122
123## Filesystem layout
124
125```
126.o11y/<run-id>/
127 manifest.json run metadata: target, domains, started_at, stopped_at
128 index.jsonl one line per sample: {ts, screenshot, dom, url}
129 cdp/
130 raw.ndjson full CDP firehose (one JSON object per line)
131 summary.json {sessionId, duration, totalEvents, pages[]} — see shape below
132 network/{requests,responses,finished,failed,websocket}.jsonl session-wide buckets (always written)
133 console/{logs,exceptions}.jsonl
134 runtime/all.jsonl
135 log/entries.jsonl
136 page/{navigations,lifecycle,frames,dialogs,all}.jsonl
137 dom/all.jsonl (only if O11Y_DOMAINS includes DOM)
138 target/{attached,detached}.jsonl
139 pages/ per-page slices, indexed by top-level frameNavigated boundaries
140 000/ first concrete page
141 url.txt the URL for this page
142 summary.json this page's domains/network/timing block (same shape as a pages[] entry)
143 raw.jsonl firehose scoped to this page
144 network/, console/, page/, runtime/, log/, target/, dom/ same buckets, only non-empty files
145 screenshots/<iso-ts>.png one PNG per sample interval
146 dom/<iso-ts>.html one HTML dump per sample interval
147 browserbase/ added by bb-finalize.mjs (Browserbase runs only)
148 session.json final browse cloud sessions get snapshot (proxyBytes, status, ended_at, …)
149 logs.json browse cloud sessions logs output (often [])
150 downloads.zip browse cloud sessions downloads get output (only if the session downloaded files)
151```
152
153When a run was started via bb-capture.mjs, manifest.json also carries a top-level browserbase block: session_id, project_id, region, started_at, expires_at, keep_alive, debugger_url.
154
155### Summary shape
156
157cdp/summary.json is the entry point for any analysis: it has session-level totals and a pages[] array indexed by top-level Page.frameNavigated. Per-page entries are emitted in navigation order (page 0 = first concrete URL).
158
159```json
160{
161 "sessionId": "45f28023-…",
162 "duration": { "startMs": 1777312533000, "endMs": 1777312609000, "totalMs": 76000 },
163 "totalEvents": 420,
164 "pages": [
165 {
166 "pageId": 0,
167 "url": "https://example.com/",
168 "startMs": 1777312533000, "endMs": 1777312538886, "durationMs": 5886,
169 "eventCount": 60,
170 "domains": {
171 "Network": { "count": 18, "errors": 1 },
172 "Console": { "count": 2 },
173 "Page": { "count": 24 },
174 "Runtime": { "count": 13 }
175 },
176 "network": { "requests": 4, "failed": 1, "byType": { "Document": 2, "Script": 1, "Other": 1 } }
177 }
178 ]
179}
180```
181
182startMs / endMs / durationMs are wall-clock ms, derived from manifest.started_at plus the offset of each event's CDP monotonic timestamp. domains[*] only includes errors/warnings keys when non-zero.
183
184### Drilling in with query.mjs
185
186For interactive exploration, use scripts/query.mjs <run-id> <command> instead of remembering paths:
187
188```bash
189node scripts/query.mjs my-run list # one-line table of pages
190node scripts/query.mjs my-run page 1 # full summary for page 1
191node scripts/query.mjs my-run page 1 network/failed # cat failed.jsonl for page 1
192node scripts/query.mjs my-run errors # all errors across pages, attributed by pid
193node scripts/query.mjs my-run errors 2 # errors from page 2 only
194node scripts/query.mjs my-run hosts # top hosts by request count
195node scripts/query.mjs my-run host api.example.com # all requests/responses for a host
196node scripts/query.mjs my-run summary # full summary.json
197```
198
199Behind the scenes it just reads cdp/summary.json and the cdp/pages/<pid>/ tree — feel free to bypass it with raw jq/rg once you know the shape.
200
201## Top traversal recipes
202
203```bash
204# All failed network requests (use jq -c to keep it line-delimited)
205jq -c '.params' .o11y/<run>/cdp/network/failed.jsonl
206
207# Find requests to a specific host
208jq -c 'select(.params.request.url | test("api\\.example\\.com"))' \
209 .o11y/<run>/cdp/network/requests.jsonl
210
211# 4xx/5xx responses
212jq -c 'select(.params.response.status >= 400)
213 | {status: .params.response.status, url: .params.response.url}' \
214 .o11y/<run>/cdp/network/responses.jsonl
215
216# Console errors only
217jq -c 'select(.params.type == "error")' .o11y/<run>/cdp/console/logs.jsonl
218
219# Sequence of URLs visited
220jq -r '.params.frame.url' .o11y/<run>/cdp/page/navigations.jsonl
221
222# Find the screenshot taken closest to a timestamp (e.g., when an exception fired)
223ls .o11y/<run>/screenshots/ | sort | awk -v t=20260427T1714123NZ '
224 $0 >= t { print; exit }'
225```
226
227See **REFERENCE.md** for the full jq recipe library and a method-by-method bisect map. See **EXAMPLES.md** for end-to-end debug scenarios.
228
229## Best practices
230
2311. **Use bb-capture.mjs on Browserbase**: it enforces --keep-alive, fetches the connectUrl, captures the debugger URL, and stamps the manifest. Doing it manually invites mistakes.
2322. **Don't --release a session you don't own**: bb-finalize.mjs --release is for sessions *you* created with --new. When attaching to a production session via bb-capture.mjs <session-id>, run bb-finalize.mjs without --release so the original automation keeps running.
2333. **Order matters for remote**: on Browserbase, attach the main automation client before (or together with) the tracer, and create the session with --keep-alive. Otherwise the session ends as soon as the tracer's WS closes.
2344. **Don't poll faster than ~1s**: each sample runs browser CLI read commands and screenshots Chrome. 2s is a good default.
2355. **Pick domains deliberately**: defaults (Network Console Runtime Log Page) cover most debugging. Add DOM for DOM-tree mutations (very noisy) via O11Y_DOMAINS="$O11Y_DOMAINS DOM".
2366. **Reuse one Browserbase session for the automation client on remote** by attaching to that session's connectUrl with browse open ... --cdp "$CONNECT_URL" --session <name>. The --session flag names the local browse daemon; it is not a Browserbase session attach flag.
2377. **Always run stop-capture.mjs**, even after a crash, so background processes don't linger and the manifest gets stopped_at.
2388. **Bisect once per run**: bisect-cdp.mjs is idempotent — it overwrites the per-bucket files from raw.ndjson each time.
239
240## Troubleshooting
241
242- **browse cdp exited immediately**: usually means the target is unreachable (wrong port) or the Browserbase session has already ended. For remote, verify with browse cloud sessions get <id> — if status is COMPLETED, recreate with --keep-alive and attach automation first.
243- **Empty raw.ndjson even though processes are running**: confirm a CDP client is actually driving the page. The tracer only emits events that the browser generates, so an idle browser produces ~5 lines of attach/discover messages and nothing else.
244- **Screenshots all look identical**: check index.jsonl — if url doesn't change, the page hasn't navigated yet. The polling loop runs independently of the main automation's pace.
245- **Browserbase session ends mid-run**: it likely hit --timeout. Recreate with a higher timeout (BB_SESSION_TIMEOUT=1800 node scripts/bb-capture.mjs --new ...) or remove the timeout flag.
246- **bb-capture.mjs <id> says "not RUNNING"**: the session you tried to attach to ended. List candidates with browse cloud sessions list | jq '.[] | select(.status == "RUNNING")' and try again.
247- **browserbase/logs.json is empty []**: expected — browse cloud sessions logs is sparse in practice. The CDP firehose in cdp/raw.ndjson is the source of truth.
248- **Where's the session recording (rrweb)?**: session replay artifact fetching is deprecated; this skill doesn't fetch it. Use the screenshot stream in screenshots/ and DOM dumps in dom/.
249
250For full reference, see [REFERENCE.md](REFERENCE.md).
251For example debug runs, see [EXAMPLES.md](EXAMPLES.md).
252