sk Skill · us
crw-map
Discover all URLs on a website without fetching content — fast, low-cost URL inventory via sitemap.xml + link extraction BFS. Use when you need to know which pages exist before deciding what to scrape or crawl: "list all pages", "find URLs on this site", "discover links", "what pages does this site have", "map the site". Step 3 of the crw workflow ladder.
Open on skills.sh ↗read 2026-09-21
- installs 8w
- 0
- 30-day movement
- starts with the next reading
- Related entries
- 1
- Connections
- 1
bashRust
- Host repository
- us/crw
- Version
- 0.3.0
- Allowed tools
- Bash(crw:*) Bash(curl:*) Read
- Licence
- AGPL-3.0
- Host stars
- 1,059
- Host language
- Rust