#Crawling

D4Vinci

Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

scrapy

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

gocolly

colly

Elegant Scraper and Crawler Framework for Golang

apify

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

getmaxun

maxun

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

codelucas

newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

apify

crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

go-rod

rod

A Chrome DevTools Protocol driver for web automation and scraping.

hakluke

hakrawler

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

hardkoded

puppeteer-sharp

Headless Chrome .NET API

edoardottt

cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

transitive-bullshit

awesome-puppeteer

A curated list of awesome puppeteer resources.

lorien

grab

Web Scraping Framework

rebrowser

rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps to avoid Cloudflare and DataDome CAPTCHA pages. Easy to patch/unpatch, can be enabled/disabled on demand.

needleworm

bhban_rpa

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

scrapfly

scrapfly-scrapers

Scalable Python web scraping scripts for +40 popular domains

clemfromspace

scrapy-selenium

Scrapy middleware to handle javascript pages using selenium

scrapinghub

scrapyrt

HTTP API for Scrapy spiders

iawia002

Lulu

[Unmaintained] A simple and clean video/music/image downloader 👾

janreges

siteone-crawler

SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).

StJudeWasHere

seonaut

Open source SEO audit tool.

slotix

dataflowkit

Extract structured data from web sites. Web sites scraping.

cxcscmu

Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

l4rm4nd

LinkedInDumper

Python 3 script to dump/scrape/extract company employees from LinkedIn API

lumpinif

deepcrawl

100% free and full open-source edge Firecrawl alternative with better links extraction for agents - that you can deploy to cloudflare or vercel by yourself.

Florents-Tselai

WarcDB

WarcDB: Web crawl data as SQLite databases.

mhmdiaa

second-order

Second-order subdomain takeover scanner

XORbit01

webpalm

🕸️ Crawl in the web network

rivermont

spidy

The simple, easy to use command line web crawler.

mustafadalga

Instagram-Bot

An Instagram bot developed using the Selenium Framework

18520339

facebook-data-extraction

Experience for effectively fetching Facebook data by Querying Graph API with Account-based Token and Operating undetectable scraping Bots to extract Client/Server-side Rendered content

carlosplanchon

spidercreator

Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. Ideal for large enterprises with recurring data extraction needs.

A3h1nt

Grawler

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

ArchiveBox

abx-dl

⬇️ A simple all-in-one CLI tool to download EVERYTHING from a URL (like youtube-dl/yt-dlp, forum-dl, gallery-dl, simpler ArchiveBox). 🎭 Uses headless Chrome to get HTML, JS, CSS, images/video/audio/subtitles, PDFs, screenshots, article text, git repos, and more...

kreuzberg-dev

kreuzcrawl

High-performance web crawling engine with bindings for 11 languages

DBeath

feedsearch-crawler

Crawl sites for RSS, Atom, and JSON feeds.

creekorful

bathyscaphe

Fast, highly configurable, cloud native dark web crawler.

apify

actor-rag-web-browser

RAG Web Browser is an Apify Actor to feed your LLM applications and RAG pipelines with up-to-date text content scraped from the web.

guptachetan1997

crawling-projects

Web scraping and automation using python

pzaino

thecrowler

A Content Discovery and Development Platform. Empowering Cybersecurity, AI, Marketing, and Finance professionals and researchers to discover, analyze, and interact with the web in all its dimensions.

supacrawler

supacrawler

Supacrawler's ultralight engine for scraping and crawling the web. Written in go for maximum performance and concurrency.

Conso1eCowb0y

Deepminer

Deep web crawler and search engine

Swader

diffbot-php-client

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

lunarwhite

covid-social-analysis

Apply ML on weibo sentiment. 疫情背景下微博文本情感分析与可视化

squirm-framework

squirm

This was the night of the crawling terror!

jy617lee

popular_restaurants_from_officials

서울시 공무원의 업무추진비를 분석하여 진짜 맛집 찾기 프로젝트

ivan-sincek

scrapy-scraper

Web crawler and scraper based on Scrapy and Playwright's headless browser.