GH Repository · spider-rs

spider

Get web data for AI agents and LLMs — fast, efficient, and reliable with Rust

Open on GitHub ↗created 2018-01-07updated 2026-09-16read 2026-09-17

stars: 2,718
30-day movement: +52/day
Related entries: 60
Connections: 2

nixweb-dataweb-crawlerautomationotherRustweb-scrapingscrapingheadless-chromespiderai-agentrustcrawler

Spider is a Rust-based web crawler and scraper for collecting web data, with headless Chrome support for rendering pages. It is positioned for feeding data to AI agents and LLMs.

You need a fast, Rust-native crawler to gather website content for AI or scraping pipelines.

Use it to

Crawl entire websites to collect page content
Scrape rendered pages via headless Chrome
Feed gathered web data into LLM or agent workflows
Automate recurring web data collection

For Developers building scrapers, crawlers, or AI data pipelines

Role: other
Language: Rust
Licence: MIT
Forks: 301
Last push: 2026-09-16
Latest release: 1.0.3 · 2018-02-06

topicsweb-crawlerscrapingrustheadless-chromeai-agentsweb-data