#Web-crawling
Showing 13 of 13 repositories tagged #web-crawling, ranked by stars
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Declarative data automation language and Go runtime for structured extraction workflows.
The All in One Framework to Build Undefeatable Scrapers
A powerful Model Context Protocol (MCP) server that provides an all-in-one solution for public web access.
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
:zap: Ayakashi.io - The next generation web scraping framework
A tool for scraping emails, social media accounts, and much more information from websites using Google Search Results.
Apify SDK for Python—The official library for building Apify Actors: serverless cloud programs for web scraping, browser automation, data processing, and AI agents. Manages the Actor lifecycle, storages (datasets, key-value stores, request queues), events, proxies, and pay-per-event monetization. Built on top of the the Apify API Client.
Apify API client for Python—Programmatically run Actors, manage and stream data from storages (datasets, key-value stores, request queues), schedule and monitor runs, and access the full Apify platform API. Sync and async interfaces with automatic retries and pagination.
Compares price of the product entered by the user from e-commerce sites Amazon and Flipkart :moneybag: :bar_chart:
Another curated list of Python frameworks
Repository for the projects needed to complete the Data Analyst Nanodegree.