GH Repository · oxylabs
ai-crawler-py
Crawl a website starting from a URL, find relevant pages, and extract data – all guided by your natural language prompt.
- stars
- 3,264
- 30-day movement
- +10/day
- Related entries
- 55
- Connections
- 1
agent-appai-web-crawlerai-scrapingweb-crawlerai-crawlercrawl-agentai-agentsai-studioai
ai-crawler-py is a Python agent app from Oxylabs that crawls a website starting from a URL, finds relevant pages, and extracts data based on a natural language prompt you provide. The repository's recorded structure only names the ai-crawler-py package; further capabilities are not detailed in the material.
Use it when you want targeted web crawling and data extraction driven by a plain-language prompt instead of hand-written crawl rules.
Use it to
- Crawl a site from a starting URL
- Guide page discovery with a natural language prompt
- Extract data from relevant pages
For Developers building AI-guided web crawling and scraping workflows
- Role
- agent-app
- Forks
- 12
- Open issues
- 1
- Last push
- 2026-08-21
topicsai-agentsweb-crawlerai-scrapingcrawl-agentpython