AI agent tools for crawler
Short answer: Our top pick for crawler: Firecrawl, with the highest AgentPick Score (94/100) among tools tagged "crawler". Runners-up: Crawl4AI (91) and Scrapling (84). Compare what each does, its license and how recently it was updated below.
| # | Tool | What it does | Score | Stars | License | Updated |
|---|---|---|---|---|---|---|
| #1 | Firecrawl | Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥 | 94/100 | ★ 192k | AGPL-3.0 | 14 Oct 2026 |
| #2 | Crawl4AI | Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with on… | 91/100 | ★ 86k | Apache-2.0 | 14 Oct 2026 |
| #3 | Scrapling | 🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.g… | 84/100 | ★ 88k | BSD-3-Clause | 14 Oct 2026 |
| #4 | Douyin_TikTok_Download_API | 🚀 抖音、TikTok 数据采集与无水印视频下载 API，自托管，支持 MCP 调用与 Docker 一键部署。| Self-hosted TikTok & Douyin scraper and no-watermark video downloader — async REST API, MCP… | 75/100 | ★ 21k | Apache-2.0 | 2 Oct 2026 |
| #5 | crw | Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible … | 66/100 | ★ 1.1k | AGPL-3.0 | 14 Oct 2026 |
| #6 | fess | Open-source, self-hosted enterprise & site search server built on OpenSearch. Crawls web / file / DB / cloud sources, 20+ languages, REST API, and AI/… | 62/100 | ★ 1.1k | Apache-2.0 | 14 Oct 2026 |
| #7 | reader | Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents. | 58/100 | ★ 557 | Apache-2.0 | 6 Oct 2026 |
| #8 | spider | Foundational low latency web data collecting in Rust | 54/100 | ★ 2.8k | MIT | 1 Oct 2026 |
Source: each project's public GitHub repository (description, stars, license, last push). Listed tools aren't tested or ranked by AgentPick.
How to choose
- Grant the narrowest permissions the server allows.
- Check whether it runs locally or calls a remote service.
- Prefer servers listed in the official MCP registry.
Get the best new agent tools every week. Free, with a verdict on each. Get the weekly digest
Related
AI agent tools for local-first setups →
MCP servers
AI agent tools for knowledge graphs and bases →
MCP servers
AI agent tools for RAG (retrieval-augmented generation) →
MCP servers
AI agent tools for MCP client →
MCP servers
AI agent tools for cybersecurity →
MCP servers
AI agent tools for DevOps →
MCP servers
AgentPick Weekly
The week's new agent tools with a verdict on each, every week. Free. We'll email you a link to confirm, and you can unsubscribe any time. See our Privacy Policy.