#Web-crawler
Showing 37 of 37 repositories tagged #web-crawler, ranked by stars
The API to search, scrape, and interact with the web at scale. 🔥
Python scraper based on AI
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Ingest, parse, and optimize any data format ➡️ from documents to multimedia ➡️ for enhanced compatibility with GenAI frameworks
Web data
The headless Chrome/Chromium driver on top of Puppeteer. Take screenshots, generate PDFs, extract text and HTML with a production-ready API.
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
CLI tool for saving a faithful copy of a complete web page in a single HTML file (based on SingleFile)
Scalable Python web scraping scripts for +40 popular domains
扫描“微信读书”已购图书并下载本地PDF的爬虫
🔥 This repository contains complete application examples, including websites and other projects, developed using Firecrawl.
Free desktop SEO crawler - open source alternative to Screaming Frog and similar tools. Crawl websites, analyze links, extract SEO data, and export results without subscription fees. Fully customizable and extensible!
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
Open source web infrastructure for AI. Scrape, crawl, and automate the web, clean markdown, browser sessions, ready for your agents.
The simple, easy to use command line web crawler.
Fast, lightweight Firecrawl/Tavily alternative in Rust. Web scraper, crawler & search API with MCP server for AI agents. Drop-in Firecrawl-compatible API (/scrape, /crawl, /search). 2.3x faster than Tavily, 1.5x faster than Firecrawl in 1K-URL benchmarks. 6 MB RAM, single binary. Self-host or use managed cloud.
Data Analysis & Mining for lagou.com
Golang Crawling and scraping framework
High-performance web crawling engine with bindings for 11 languages
A Machine Learning Project implemented from scratch which involves web scraping, data engineering, exploratory data analysis and machine learning to predict housing prices in New York Tri-State Area.
A web crawler for Sina, search and retrieve microblogs that contain certain keywords 一个简单的python爬虫实践，爬取包含关键词的新浪微博
Fast, highly configurable, cloud native dark web crawler.
:dart: predict the price trend of individual stocks using deep learning and natural language processing
Powerful Google Maps Crawler / Scraper tool with REST API, Docker support & multi-format export
A tool to find redirection chains in multiple URLs
Displays all the 2019 CVPR Accepted Papers in a way that they are easy to parse.
Full AI context and content layer for coding agents over one MCP server — tree-sitter code-map, document RAG, shared memory, multi-agent comms, web crawl, git history + blame. 300+ languages, 10+ agent harnesses, pure Rust.
This is a project of build knowledge graph course. The project leverages historical stock price, and integrates social media listening from customers to predict market Trend On Dow Jones Industrial Average (DJIA).
This script will dump youtube video comments to a CSV from youtube video links. Video links can be placed inside a variable or list or CSV
"Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases" by Jiarui Li and Ye Yuan and Zehua Zhang
Web Scraping using Python Data mining , Data Analyzing & Data Visualization of the collected Data, The python script is written to fetch all the individual categories the website , The code is written for fetching the data from the first page and it iterates to each and every pages of website ( activities, categories, count of bought), and I used statistical techniques for mathematically analysis and presenting the data into visualization
RayScan 1.0 — 开箱即用的 Web 漏洞扫描器 | SQL注入 / XSS / 命令注入 / LFI / SSRF / XXE / RCE / API安全 | 11个检测模块 | CLI + Web UI 双模式 | Metasploitable 2 实战验证发现83漏洞
Convert the public web into AI-ready Markdown with a local Python CLI/SDK/MCP crawler.
Turn any website into clean, LLM-ready data. Open-source web crawler with stealth mode, distributed crawling, real-time WebSocket progress & Markdown output. Power your AI apps with GcrawlAI.
Self-hosted RAG engine for AI coding assistants. Ingests technical docs & code repositories locally with structure-aware chunking. Serves grounded context via MCP to prevent hallucinations in software development workflows.