Crawl4AI

Name: Crawl4AI
Author: brettdavies

Scrape JavaScript-heavy pages and SPAs, crawl many URLs concurrently, and extract structured data with reusable CSS/JSON schemas via the…

You say

Install this skill Read the source first Free Written by brettdavies · unverified publisher

Context cost: 96.3k tokensestimated from the bundle, loaded when it triggers
Bundle: 46 files · 385.0 kB12 scripts among them — read before you run
Licence: Apache-2.0 OR MITfree to use
Last change: —no release on file
Servers it uses: Noneruns standalone

What it does

Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser.

Installed, it changes the agent in these ways.

What this skill changes about the agent is not written down here yet. The listing was collected from its source, and the description is in its own SKILL.md.

Workflow

Runs a procedure end to end.

crawlingspaheadlessschema-extraction

Filed under

Browser Automation

The skill itself

This is the whole product. A skill is instructions the model reads, so there is nothing behind the listing you cannot see first — the front matter loads with every session, and the body below it loads when the skill triggers.

SKILL.md16.7 kB · 398 lines

--- name: crawl4ai description: Use when scraping JavaScript-heavy pages or SPAs, crawling multiple URLs concurrently, extracting structured data with reusable CSS/JSON schemas, or building automated web data pipelines. Wraps the Crawl4AI library (`crwl` CLI and Python SDK) with schema-generation patterns for LLM-free extraction. Triggers on crawl4ai, crwl, scrape JS-heavy site, scrape SPA, headless browser scrape, schema-based extraction, batch crawl, sitemap crawl, web data pipeline. SKIP when a static HTML page can be read with `defuddle` / `fetch-web` — those are faster cold-start and don't need a browser. argument-hint: "[url]" ---

7# Crawl4AI

9**Verified against crawl4ai** [VERSION](VERSION). PEP 723 pins in scripts/*.py and tests/*.py floor at that

10version.

12## Overview

14Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't

15reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable

16schema-based extraction.

18This skill exposes both interfaces of the underlying library:

20- **CLI** (crwl) — quick, scriptable commands: [CLI Guide](references/cli-guide.md)

21- **Python SDK** — full programmatic control: [SDK Guide](references/sdk-guide.md)

23## Invoked with a URL argument

25When the user runs /crawl4ai <url> with a single URL and no further qualifier, treat it as the JS-heavy fetch case and

26default to:

28```bash

29crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown

30```

32wait_until=networkidle waits for the network to be quiet for ~500ms post-load — the right default when the user hasn't

33named a specific element on a JS-rendered page. (Avoid wait_for=css:body: <body> exists at t=0 on every HTML

34response, so it's satisfied before JS renders content.) Then return the markdown to the agent context. Adjust to

35wait_for=css:<selector> if the user named a specific element. Skip the default and route to the relevant section below

36for any task that names extraction, batch / multi-URL, login / session, screenshot / PDF, or URL discovery — those each

37have their own pipeline. If the URL is clearly static (a docs page, a blog post), route the user to /fetch-web instead

38per the "When NOT to use" section below.

40## When NOT to use this skill

42- **Static HTML pages** (most documentation sites, blog posts, news articles, tweets) — use /fetch-web or defuddle

43 directly. Static extraction is ~0ms cold start; crawl4ai pays a ~2s browser startup tax.

44- **Local file conversion** (.pdf, .docx, .pptx, .epub) — use /markdown-convert.

45- **One-URL agent-context reads** (the agent just needs to read this page) — use /fetch-web and let it route to

46 defuddle.

47- **Mutating UI flows** (form fills, multi-step clicks, login + navigation) — /browse (gstack's persistent headless

48 Chromium) is built for that.

50## When stuck

52For unknown crwl/SDK flags, scrape failures, or extraction edge cases the references don't cover, see

53[references/escalation.md](references/escalation.md) for the lookup order (qmd solutions → upstream docs → GitHub issues

54→ ask the user) and worked examples.

56---

58## Quick Start

60### Installation

62```bash

63pip install crawl4ai

64crawl4ai-setup

66# Verify installation

67crawl4ai-doctor

68```

70### CLI (Recommended)

72```bash

73# Basic crawling - returns markdown

74crwl https://example.com

76# Get markdown output

77crwl https://example.com -o markdown

79# JSON output with cache bypass

80crwl https://example.com -o json -v --bypass-cache

82# See more examples

83crwl --example

84```

86### Python SDK

88```python

89import asyncio

90from crawl4ai import AsyncWebCrawler

92async def main():

93 async with AsyncWebCrawler() as crawler:

94 result = await crawler.arun("https://example.com")

95 print(result.markdown[:500])

97asyncio.run(main())

98```

100For SDK configuration details: [SDK Guide - Configuration](references/sdk-guide.md#configuration).

101

102---

103

104## Core Concepts

105

106### Configuration Layers

107

108Both CLI and SDK use the same underlying configuration:

109

110| Concept | CLI | SDK |

111| ---------------- | -------------------------------------- | ------------------------- |

112| Browser settings | -B browser.yml or -b "param=value" | BrowserConfig(...) |

113| Crawl settings | -C crawler.yml or -c "param=value" | CrawlerRunConfig(...) |

114| Extraction | -e extract.yml -s schema.json | extraction_strategy=... |

115| Content filter | -f filter.yml | markdown_generator=... |

116

117### Key Parameters

118

119**Browser Configuration:**

120

121- headless: Run with/without GUI

122- viewport_width/height: Browser dimensions

123- user_agent: Custom user agent

124- proxy_config: Proxy settings

125

126**Crawler Configuration:**

127

128- page_timeout: Max page load time (ms)

129- wait_for: CSS selector or JS condition to wait for

130- cache_mode: bypass, enabled, disabled

131- js_code: JavaScript to execute

132- css_selector: Focus on specific element

133

134For complete parameters: [CLI Config](references/cli-guide.md#configuration) |

135[SDK Config](references/sdk-guide.md#configuration)

136

137### Output Content

138

139Every crawl returns:

140

141- **markdown** - Clean, formatted markdown

142- **html** - Raw HTML

143- **links** - Internal and external links discovered

144- **media** - Images, videos, audio found

145- **extracted_content** - Structured data (if extraction configured)

146

147---

148

149## Markdown Generation (Primary Use Case)

150

151Crawl4AI excels at generating clean, well-formatted markdown.

152

153### CLI

154

155```bash

156crwl https://docs.example.com -o markdown # raw markdown

157crwl https://docs.example.com -o markdown-fit # filtered (noise removed)

158crwl https://docs.example.com -f templates/filter_bm25.yml -o markdown-fit # BM25-relevance filter

159crwl https://docs.example.com -f templates/filter_pruning.yml -o markdown-fit # quality-based filter

160```

161

162Filter templates: [templates/filter_bm25.yml](templates/filter_bm25.yml) (relevance-scored against a query),

163[templates/filter_pruning.yml](templates/filter_pruning.yml) (no query, prunes low-quality blocks).

164

165### Python SDK

166

167```python

168from crawl4ai.content_filter_strategy import BM25ContentFilter

169from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

170

171bm25_filter = BM25ContentFilter(user_query="machine learning", bm25_threshold=1.0)

172md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)

173

174config = CrawlerRunConfig(markdown_generator=md_generator)

175result = await crawler.arun(url, config=config)

176

177print(result.markdown.fit_markdown) # Filtered

178print(result.markdown.raw_markdown) # Original

179```

180

181For filter selection and config field reference, see [Content Filters](references/content-filters.md).

182

183---

184

185## Data Extraction

186

187### 1. Schema-Based CSS Extraction (Most Efficient)

188

189**No LLM required** at extract time — fast, deterministic, cost-free. One-time LLM cost to derive the schema, then reuse

190indefinitely. The bundled scripts split the pipeline by responsibility:

191

192```bash

193./scripts/generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json

194./scripts/extract_with_schema.py https://shop.example.com shop_schema.json products.json

195```

196

197Or via the CLI with the YAML strategy template + the saved schema:

198

199```bash

200crwl https://shop.example.com -e templates/extract_css.yml -s shop_schema.json -o json

201```

202

203Schema skeleton: [templates/css_schema.json](templates/css_schema.json). Strategy YAML:

204[templates/extract_css.yml](templates/extract_css.yml).

205

206### 2. LLM-Based Extraction

207

208For one-off / irregular content where a CSS schema is too brittle:

209

210```bash

211./scripts/extract_with_llm.py https://news.example.com "Extract headlines, dates, summaries" news.json

212```

213

214Or via the CLI with the strategy template:

215

216```bash

217crwl https://news.example.com -e templates/extract_llm.yml -o json

218```

219

220Strategy YAML: [templates/extract_llm.yml](templates/extract_llm.yml). Pays an LLM call per URL — for repeat

221extraction, prefer the schema pipeline above.

222

223For extraction strategy reference: [Extraction Strategies](references/complete-sdk-reference.md#extraction-strategies).

224

225---

226

227## Advanced Patterns

228

229### Dynamic Content (JavaScript-Heavy Sites)

230

231```bash

232crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000"

233crwl https://example.com -C templates/crawler.yml # all options in a YAML file

234```

235

236Crawler config template: [templates/crawler.yml](templates/crawler.yml).

237

238### Multi-URL Processing

239

240```bash

241./scripts/batch_crawl.py urls.txt --max-concurrent 5 --out batch_markdown/

242./scripts/batch_extract.py urls.txt shop_schema.json --max-concurrent 5 --out products.json

243```

244

245The two scripts split on responsibility: batch_crawl.py returns markdown per URL; batch_extract.py returns

246schema-extracted JSON per URL. Python equivalent uses arun_many():

247

248```python

249urls = ["https://site1.com", "https://site2.com", "https://site3.com"]

250results = await crawler.arun_many(urls, config=config)

251```

252

253For batch processing reference: [arun_many() Reference](references/complete-sdk-reference.md#arunmany-reference).

254

255### URL Discovery Before Crawl

256

257When the URL list comes from a sitemap / domain rather than a known list, do discovery first, then feed the result into

258batch_crawl.py / batch_extract.py. See [URL Discovery](references/url-discovery.md) for the full surface; quick

259shape:

260

261```python

262from crawl4ai import AsyncUrlSeeder, SeedingConfig

263seeds = await AsyncUrlSeeder().urls("example.com", SeedingConfig(

264 source="sitemap+cc", pattern="*/blog/*", query="machine learning", score_threshold=0.3, live_check=True,

265))

266urls = [s["url"] for s in seeds]

267```

268

269AsyncUrlSeeder is best when you want BM25-scored filtering against a query; DomainMapper is best when you want

270maximum coverage of one domain.

271

272### Session & Authentication

273

274Fill the login template, then reuse the session id on subsequent crawls:

275

276```bash

277crwl https://site.com/login -C templates/login_crawler.yml

278crwl https://site.com/protected -c "session_id=user_session"

279```

280

281Login template: [templates/login_crawler.yml](templates/login_crawler.yml) (fill in the field-id selectors and the

282post-login wait condition before use).

283

284For session management reference: [Advanced Features](references/complete-sdk-reference.md#advanced-features).

285

286### Anti-Detection & Proxies

287

288```bash

289crwl https://example.com -B templates/browser.yml

290```

291

292Browser config template: [templates/browser.yml](templates/browser.yml) (uncomment proxy_config and init_scripts

293as needed). For pre-page-load script injection (fingerprint patches that must fire **before** any site script), populate

294init_scripts: rather than js_code: (which fires after the page loads). proxy_config works with both the browser

295strategy and the non-browser HTTPCrawlerStrategy — the latter is the cheap path for static fetches behind a corporate

296proxy.

297

298Full surface (CDP attachment, undetected mode, init script patterns): [Anti-Detection](references/anti-detection.md).

299

300### Rendering Cached HTML (raw: / file://)

301

302If the agent already has HTML in hand (e.g., from defuddle or a previous crawl) and only needs a screenshot, PDF, or

303MHTML render, skip the network fetch and pass the HTML directly. base_url controls relative-link resolution:

304

305```python

306result = await crawler.arun(

307 url="raw:" + html_string,

308 config=CrawlerRunConfig(base_url="https://example.com", screenshot=True, pdf=True),

309)

310```

311

312```python

313result = await crawler.arun(

314 url="file:///path/to/page.html",

315 config=CrawlerRunConfig(screenshot=True),

316)

317```

318

319---

320

321## Common Use Cases

322

323Eight worked end-to-end flows (docs page, JS-heavy SPA, e-commerce product extraction, news aggregation, topic-bound

324domain crawl, login-required content, render existing HTML, Q&A) live in [Recipes](references/recipes.md). Pick the

325recipe closest to the task at hand and adapt.

326

327---

328

329## Resources

330

331### Provided Scripts

332

333| Script | Responsibility |

334| ---------------------------------------------------- | ---------------------------------------------------- |

335| scripts/basic_crawler.py <url> | One URL → markdown + screenshot |

336| scripts/batch_crawl.py <urls.txt> | Many URLs → markdown files |

337| scripts/batch_extract.py <urls.txt> <schema.json> | Many URLs + schema → JSON |

338| scripts/generate_schema.py <url> "<instruction>" | Derive a reusable CSS schema (one-time LLM call) |

339| scripts/extract_with_schema.py <url> <schema.json> | Apply a saved schema (no LLM) |

340| scripts/extract_with_llm.py <url> "<instruction>" | Per-request LLM extraction (expensive; one-off only) |

341

342### Templates

343

344YAML and JSON skeletons users copy and fill. All sit at the skill root under templates/:

345

346| Template | Used for |

347| ------------------------------ | ----------------------------------------------------------- |

348| templates/browser.yml | BrowserConfig (headless, proxy, user agent, init scripts) |

349| templates/crawler.yml | CrawlerRunConfig (cache, wait, timeout, JS) |

350| templates/extract_css.yml | JsonCssExtractionStrategy declaration |

351| templates/extract_llm.yml | LLMExtractionStrategy declaration |

352| templates/filter_bm25.yml | BM25 content filter (relevance-scored) |

353| templates/filter_pruning.yml | Pruning content filter (quality-based, no query) |

354| templates/login_crawler.yml | Session-establishing login flow |

355| templates/css_schema.json | CSS schema skeleton |

356

357### Reference Documentation

358

359| Document | Purpose |

360| -------------------------------------------------------------- | --------------------------------------------------------------- |

361| [CLI Guide](references/cli-guide.md) | Command-line interface reference |

362| [SDK Guide](references/sdk-guide.md) | Python SDK quick reference |

363| [Recipes](references/recipes.md) | Eight worked end-to-end flows |

364| [URL Discovery](references/url-discovery.md) | AsyncUrlSeeder, SeedingConfig, DomainMapper |

365| [Content Filters](references/content-filters.md) | BM25 vs Pruning vs LLMContentFilter — when to use which |

366| [Anti-Detection](references/anti-detection.md) | init_scripts, proxy_config, undetected mode, CDP attachment |

367| [Troubleshooting](references/troubleshooting.md) | Symptoms, causes, fixes; what to try before escalating |

368| [Complete SDK Reference](references/complete-sdk-reference.md) | Full API documentation (5900+ lines) |

369| [Escalation](references/escalation.md) | Lookup order, iron rule, halt-vs-continue, worked examples |

370

371---

372

373## Best Practices

374

3751. **Start with CLI** for quick tasks, SDK for automation

3762. **Use schema-based extraction** - 10-100x more efficient than LLM

3773. **Enable caching during development** - --bypass-cache only when needed

3784. **Set appropriate timeouts** - 30s normal, 60s+ for JS-heavy sites

3795. **Use content filters** for cleaner, focused markdown

3806. **Respect rate limits** - Add delays between requests

381

382---

383

384## Troubleshooting

385

386For symptom → cause → fix tables (JS not loading, bot detection, empty extracted content, session not persisting, slow

387crawl, schema generation nonsense, post-upgrade regressions), see [Troubleshooting](references/troubleshooting.md). For

388unknown surface the references don't cover, follow [Escalation](references/escalation.md).

389

390---

391

392For comprehensive API documentation, see [Complete SDK Reference](references/complete-sdk-reference.md).

393

394## License

395

396Dual-licensed under [MIT](LICENSE-MIT) OR [Apache-2.0](LICENSE-APACHE) at your option (SPDX: MIT OR Apache-2.0). See

397[LICENSE](LICENSE) for the explainer + the carve-out for the upstream-mirrored references/complete-sdk-reference.md.

398

In the file

SKILL.md1,809 words

Files46

LicenceApache-2.0 OR MIT

Why you can read it

Nothing in a skill executes. The client loads the text and the model follows it, so a skill can be audited the way a runbook is — by reading it.

What it costs in context

Skills are not billed by the call. They are paid for in context: every token the instructions occupy is a token your code, your diff and your conversation cannot use. Here is what this one takes and when it takes it.

≈160

always loaded

The name and description, so the model knows the skill exists and when to reach for it.

96,090

on trigger

The instruction body and 45 supporting files, read only when the skill fires.

48.1%

of a 200k window

Ten skills this size would take about 481% of the window before you open a file.

050k100k150k200k context window

96.3k tokens, estimated from the bundle at four bytes to the token, held for the rest of the session once it triggers. Heavy. Teams tend to install this one per project rather than globally, and load it only when the job comes up.

Servers bill, skills cost

A server charges by the month. A skill charges once per session, in context, and then keeps charging it for as long as the session lives.

Before and after

The same question, put to the same model twice: once as it comes, and once with these instructions loaded.

No worked example has been published for this skill yet.

Adoption

Installsnone yet

Ratingno reviews yet

The procedure it runs

The procedure has not been published here. It is in the skill’s own SKILL.md, which its author has not sent to the marketplace yet.

Prose, not code

These steps are written for a model to follow, not executed by a runtime. It can still be told to skip one, and it will say so when it does.

Servers it uses

None. This skill calls no MCP servers at all.

Everything it needs is in the instructions, so it works in a project with nothing connected — the model reads the file and changes how it works with what it can already reach.

It writes no files and reaches no network. All it changes is how the model reasons and writes.

What it asks for

Writes filesno

Network accessno

Read from the allowed-tools line of this skill’s own SKILL.md. A skill grants no permissions of its own — it can only ask for tools your client already has.

What it will not do

Every skill is narrow, and the useful ones say where they stop. These are the jobs this one is the wrong tool for.

What this skill is not for has not been published here. Nothing is implied by that: it is a section the author has not filled in.

If this is not it

8 other skills in Browser Automation

What is in the bundle

46 files, 385.0 kB on disk. Mostly text — the instructions the model reads — with 12 scripts in it that your client would run only if the instructions tell it to.

.markdownlint-cli2.yaml2.2 kB
AGENTS.md2.8 kB
CHANGELOG.md7.2 kB
CONTRIBUTING.md2.2 kB
README.md6.9 kB
SKILL.md16.7 kB
.github/pull_request_template.md4.5 kB
evals/README.md2.0 kB
evals/eval-01-spa-markdown.md2.8 kB
evals/eval-02-extract-products.md3.3 kB
evals/eval-03-topic-domain-crawl.md3.9 kB
evals/eval-04-render-cached-html.md5.1 kB
fixtures/README.md1.1 kB
fixtures/sample-products-expected.json0.2 kB
fixtures/sample-products-schema.json0.3 kB
fixtures/sample-products.html0.7 kB
references/anti-detection.md5.8 kB
references/cli-guide.md8.4 kB
references/complete-sdk-reference.md230.9 kB
references/content-filters.md5.0 kB
references/escalation.md7.1 kB
references/recipes.md5.7 kB
references/sdk-guide.md10.2 kB
references/troubleshooting.md5.9 kB
references/url-discovery.md5.3 kB
scripts/basic_crawler.py2.3 kB
scripts/batch_crawl.py3.7 kB
scripts/batch_extract.py3.5 kB
scripts/extract_with_llm.py2.7 kB
scripts/extract_with_schema.py2.2 kB
scripts/generate_schema.py3.1 kB
templates/browser.yml0.9 kB
templates/crawler.yml1.3 kB
templates/css_schema.json0.4 kB
templates/extract_css.yml0.3 kB
templates/extract_llm.yml0.6 kB
templates/filter_bm25.yml0.3 kB
templates/filter_pruning.yml0.4 kB
templates/login_crawler.yml0.6 kB
tests/README.md2.6 kB
tests/run_all_tests.py1.6 kB
tests/test_advanced_patterns.py2.4 kB
tests/test_basic_crawling.py1.8 kB
tests/test_data_extraction.py2.4 kB
tests/test_fixtures.py2.4 kB
tests/test_markdown_generation.py3.3 kB

What is not in it

A skill installs nothing and depends on nothing: it is a folder your client reads. This one carries 12 scripts beside the text, so the bundle is 46 files you can review in full before installing. The Apache-2.0 OR MIT licence covers the templates and examples as well as the instructions.

Install

Installing copies the bundle into your project. Nothing runs at install time — the files sit on disk until the model reads them.

# Crawl4AI · 96.3k tokens when loaded
npx mcprush@latest skill add brettdavies/crawl4ai

Writes to .claude/skills/crawl4ai/ in the current project. Add --global to put it in your home directory instead, for every project.

Which clients pick it up on their own

A skill is a folder of text. A client with a skills folder reads it without being told; everywhere else the same text works, it is just handed to the model rather than found.

Claude Code.claude/skills/

Claude Desktop

ChatGPT

Cursor.cursor/skills/

VS Code.github/skills/

Codex CLI.agents/skills/

Gemini CLI.gemini/skills/

Grok.grok/skills/

Zed.agents/skills/

Windsurf.windsurf/skills/

Agent SDK.claude/skills/

HTTP / API

This release

Versionnot versioned

Publishedno release date on file

PriceFree

Referencebrettdavies/crawl4ai

Versions

Its author publishes no version number, so there is nothing here to pin to: what you install is the folder as it stands today. Instructions change more often than APIs do — a skill can be rewritten entirely without anything it depends on moving.

No earlier releases have been published to the marketplace.

Pinning

Nothing to pin to: this skill carries no version number of its own. What you install is what the folder holds on the day you install it.

Reviews

—no reviews yet · no installs yet

Nobody has reviewed this skill yet. The rating is the mean of the reviews written here, so there is none until somebody writes the first.

Who can post

Only accounts that have had the skill installed for fourteen days, so a review is written after living with it rather than after reading it. Publishers may reply once.

Who wrote it

brettdavies

Publishes on mcprush.

0 servers listed1 skill listednot claimed

Profile

Publisher

Servers0

Claim this skill